Are SCAMPI Appraisals really too expensive?

[Dear Readers, for the past several months, our good friend Pat O’Toole, CMMI expert and seasoned consultant, has been collaborating with us on a monthly series of CMMI-related posts, "Just the FAQs." Our goal with these posts is to provide answers to the most frequently asked questions about the CMMI, SCAMPI, engineering strategy and software process improvement. This month Jeff reveals whether SCAMPI appraisals are too expensive. Take it away, Jeff! ~ the CMMI Appraiser]

I love a good game of “bunchball.”

I mean, who doesn’t? You know, a dozen little Pele’s chasing a soccer ball down the field trying to score a goal and win one for the team.  Finally, one fast kid breaks out for the big kick, and ’’yippeeee!” the hero saves the day with that single goal of the game. Not bad, but hardly the stuff of league championships.

Meanwhile, far removed from the action, there is always one kid who decides not to chase glory that day but to stay back, just in case the ball were to make its way back down to their end of the field. Call it good coaching, training, or just pure talent, but that kid is going places. He plays his position, and he plays to win.

At the last few CMMI events I have attended there has been a lot of talk about how expensive appraisals have become, and that SOMETHING MUST BE DONE! Stories are told of the thousands of hours of work required to “prepare” for an appraisal, and that, in some cases, the cost far exceeds the benefits. If that’s true, then they’re right – we should do something.

But are some of these organizations just playing bunchball while attempting to win the league championship? Is their difficulty in achieving “goals” a signal that the game is too complex, or is it a signal of their level of capability?

I’m a visual thinker and anyone that has worked with me knows how much I love to draw on a whiteboard. Pictures help me think through an idea that I may not otherwise be able to convey using only words. My artwork won’t be fetching any top bids at Sothebeys, but my absolute favorite drawing is of a cliff with a set of (poorly drawn) stick figures.

One set of stick figures is clawing their way up the cliff, hanging off the edge by their fingernails while yelling “whooo hoooo, we MADE Level Three!”  The other set is standing ON TOP of the cliff, lifting barbells over their head, stretching, and quietly saying to themselves “we ARE Level Three.” Which appraisal do you think was “too expensive?”

The antidote to expensive appraisals is for organizations to actually be performing at the target level before they even start working on them! If a team is spending too much time and money locating evidence of process performance, working on PIIDs, and creating “artifacts” to “fill the gaps,” (the expensive part) then perhaps they’re not quite ready for the appraisal that the boss wants to have by Tuesday. That doesn’t mean they’re not doing great things, it just means they are not quite ready for the league championship.

If a bunchball coach were tasked by a school principal to “win the league championship before the end of fiscal year 2014,” what would he do? Well, he might: 

• bring in consultants to tell them how they won the last game and teach them that one technique they used
• hire ringers to kick the ball, QA the team, and serve in important roles (like goalie for instance)
• have the consultant follow each player around and question every move he makes, “writing him up” in red‐pen on a clipboard if he or she does something wrong
• lobby the league’s governing body to use referees that are known to be friendly to their team

You get the idea.

The team might actually win some games, but after it was over they would just be the same bunchball team.

On the other hand, a wiser (and braver) coach might:

• advise the principal that his request was not possible, but you COULD have a winning season this year if we:
• trained and practiced with the team regularly
• coached the players to play positions, thereby transforming the team from a bunchball team to a soccer team
• brought in some expert help to assist the team in improving their game, not just advice on winning the league championship
• evaluated each player for their skills and put them in the right positions
• made sure we were getting honest feedback from unbiased referees

In other words, we’ll win when we’re ready to win. And we’ll do it by being a great team.

And that’s the point. Appraisals, like league championships, should be challenging but they don’t have to be really expensive. The CMMI is an international benchmark for great performance and if we want the “stamp” to mean something, we should aspire keep them that way. However, an organization that is ML3 will have little trouble proving that they are, and one that isn’t will have tremendous difficulty (and have tremendous costs) doing the same.

“But what about PIIDs ("Process Implementation Indicator Documents") and document inventories?” asked a new Lead Appraiser at the conference.  “Don’t they take a lot of time and effort to complete?”

Hmmmm…. Do they?

PIIDs and document inventories are interesting indicators of appraisal readiness, and might even be useful sometimes. But an ML2‐worthy organization will demonstrate strong, positive control over their work products (“evidence”) through solid Configuration Management and Data Management behaviors. These behaviors make locating artifacts pretty easy, reducing (or eliminating) the overhead associated with an inventory altogether. And THAT makes appraisals a whole lot less costly.

As I’m fond of saying to prospective clients:  “it’s cheaper to be great than it is to fake it!”

©Copyright 2014: Process Assessment, Consulting & Training and Broadsword Solutions

“Just the FAQs” is written/edited by Pat O’Toole and Jeff Dalton. Please contact the authors at pact.otoole@att.net and jeff@broadswordsolutions.com to suggest enhancements to their answers, or to provide an alternative response to the question posed. New questions are also welcomed!