Monday, October 29, 2007

Please help! Our Appraiser doesn't understand us!

Please help!

We are due to have our CMMI level 3 final appraisal soon & have run into a problem. We have had some gap analysis & employed a consultant to help us some time ago. They came up with a really good estimation process that helped our business no end over the past few months. At pre-appraisal the representative from the lead appraisers company tried to tell us that all CMMI estimations had to be done by size. We had already investiated this point with other CMMI consultants who had clearly told us that with CMMI v1.1 this was true, but with v1.2 that this was not the case; we could use size, but other methods (e.g. effort - as we do use) were perfectly acceptable for CMMI level 3, 4 & 5. With this knowledge we did not want to change a system that really worked for us. Knowing this we presented the evidence & discussed with the person from the lead appraisers company until he finally 'admitted' (off the record) that we were correct, but that he could not guarantee that another lead appraiser would be aware of this subtle change.

We took the attitude that if we were meeting the standard & that the method was giving a benefit to our company that it was fine to continue with it. Our lead appraiser is now saying that our ('non size') estimation method will mean that we will not pass CMMI level 3.

I have emailed the SEI who initially said that 'if the practice worked for us & did not adversely effect other areas' then it was acceptable. However they stopped short of saying that our appraisers information was out of date. We have emailed them again, but thay have said that they can not get into a dispute between an individual lead appraiser & a company. So basically:

1. Is it correct that estimation does not have to be based on 'size'
2. If this is the case then given we have a contract with the lead appraiser how can we get the SEI to get the lead appraiser educated correctly, as he & his company are too stubborn to admit that their information is out of date & that they are wrong on this point.

Boy, everybody has an opinion!

It sounds as if your LA is trying to be very a wee bit prescriptive about how you use “size” to conduct estimates. Sometimes SOME appraiser’s don’t know where their advice ends and their appraisal starts. If someone is telling you that anything in the CMMI “has to be done . . .” a certain way, I would view that as a reason to look for alternate appraiser. Who knows what else he is “interpreting” for you? The vast majority of LA's are reasonable, logical individuals - there are a few that are not.

There is a single goal in Project Planning for estimating: SG 1 “Establish Estimates.” This is the REQUIRED component. Now, in order to satisfy this goal, the model EXPECTS (read: doesn’t require) that you’ll determine what the scope is, what work products are going to be created and what their components are, and how much it’s going to cost (in time and money). If you don’t do those things, you must provide evidence of an alternative that gets you to a reliable estimate (and one that has “helped our business no end over the past few months” sounds pretty reliable). What you are describing could be an alternatve to "size" for estimating.

I assume your LA is referring to the second practice, SP1.2 “Establish Estimates of Work Product and Task Attributes.” If he were to read the informative material it goes on to say “Size is the primary input to many models . . . “ (notice it doesn’t say “all models.”). It goes on to say “ . . . can also be based on inputs such as connectivity, complexity, and structure.” This, as I read it, also means there are other methods, yet to be determined (or understood by the SEI), that one can come to an estimate. The options in the list are examples, not the only choices. Alternates are allowed if they satisfy the Goal.

You didn’t provide detail on HOW you were producing estimates, but, even if “size” were required, there are many elements of “size.” In the Agile world, time (manifested in Releases and Iterations) is a “size” attribute, as is number of features. Isn’t effort, if viewed as dollars, hours or amount of time, a calculation of size? I think it could be.

This type of “over-interpretation” of the model is a pet peeve of mine. I know of too many companies who have soured on the entire CMMI experience because there are a few appraiser’s out there with either no imagination, limited or no experience in software engineering, or have such a closed mind that they think you need to do it “their way” for it to be valid. Well I’m here to tell you . . . . there are many ways to establish estimates. This is too bad, because the CMMI can truly be a powerful and liberating model.

Bottom line on the LA is this: the SEI WON'T get in the middle (nor should they). LA's are "authorized" by the SEI, and therefore, we are permitted to deliver these appraisals on their behalf. This method works for the vast majority of appraisals . . .

So, what should you do? If you truly believe he won't/can't understand your business, it's time to face that fact and quickly disengage yourself from him - and find another one who will work WITH you to understand how you run your business.

Best of luck to you!

No comments: