Is our Appraisal Team over-reaching?

Dear Jeff,

We are a small organization; only about 20 people support the software projects. We recently completed a SCAMPI B in our quest to reach CMMI-SWML3.

I have a question about the numbers of direct artifacts required for each practice. When we look at Generic practices such as “Identify and Involve RelevantStakeholders,” I understand where it might require 2 direct artifacts to show that we do indeed first identify them, then involve them. But in other cases, how many examples do we need to show? For instance, SAM SP.1.1 wants a list of the acquisition types for each product orproduct component to be acquired. So, say we have 3 products we are acquiring, one CFE, one COTS, and one via subcontract. Now my question is, for all the rest of the SAM SP’s, do we need to show 3 differentdirect artifacts, one for each of the 3 products??

Our SCAMPI B observations revealed that some of the mini-teams repeatedly wanted to see “more such examples to show that the process is an ongoing activity.” I was surprised; I thought the SCAMPI way was to show one good direct and one indirect.

Finally, another thing surprised me from this appraisal team. I know the definition of an Indirect artifact: They are artifacts that are either a consequence of performing the practice, or that substantiate the practice. When we provided them a Direct artifact that they thought was strong, they wanted the Indirect artifact to be linked to that particular Direct artifact. So, if we showed, say, meeting minutes for the Direct artifact, they wanted the Indirect to be something associated with those particular meeting minutes, e.g., an action item that was generated from the meeting, relating to the practice. Is this normal? It presents a challenge for us, because, sometimes it was the case that the meeting minutes were discussing the issue related to the SP, but no action items were generated.

I would appreciate your opinions on any/all of my questions!!


Jeff Says: Whew! I'm tired just thinking about answering this question!

The minimum requirement for evidence in a SCAMPI A Appraisal is OneDirect + (one indirect OR one affirmation). More than the minimum could be required, if the artifacts were incomplete, but if that’s not the case it could easily create an appraisal that has an unreasonable scope. You may drive yourself nuts trying to map one artifact to one practice. In your example, ID and Involve Stakeholders, you might use a RASIC chart to ID the stakeholders, but your evidence of involving them could be partially in your workplan, partially in meeting minutes, meeting logs, emails, or calendar entries. Sometimes the evidence “lives”within another artifact that was meant to satisfy a totally different practice. This type of “synthesis” will greatly reduce the amount of artifacts you need to produce.

Ultimately it’s the Appraisal Team’s responsibility to ferret out that evidence, not your job to create a document for each and every practice. The evidence is out there . . .they just need to be creative in finding it. Of course, you can help them by creating a mapping to the model.

An appraisal is not a “QA Audit.” The team should not be auditing that“the correct stakeholders were identified for the project and that everyone of them was involved appropriately.” The CMMI isn’t magic – it doesn’t automatically create high performing clients that don’t show upf or meetings. They should be looking for the “infrastructure” you’ve put in place and some evidence that the process was indeed performed. But a functional audit may be too much.

You are correct in your definitions of Direct and Indirect, and your description of the Appraisal Team’s demands for more data seem to be overreaching to me (without knowing the details of course . . . ). There is no requirement that the indirect provided be linked to the direct in the way you’ve described, and there is no requirement that more than one direct be provided to show “ongoing” process performance. Of course, it depends what’s in the direct you’ve provided, but that’s another issue :-). For most cases, if the artifact is complete, one is enough.

That said, if the Appraisal Team (and LA) believe that an artifact or affirmation was fabricated for the Appraisal, or that is has significant weakness they are within their rights to ask for additional information, but that is not as common as you might think.

In the appraisals I lead I do not often see this – although I have heard of it happening. In your example for SAM you should be providing the direct artifact thatdescribes the types of products you would need to acquire, and one or more examples of how the acquisition was managed.

On the face of it, it sounds like your appraisal team may need some additional training.

www.broadswordsolutions.com