Just the FAQs: What's the value in tailoring and tailoring guidelines?

[Dear Readers, our good friend Pat O’Toole, CMMI expert and seasoned consultant, is collaborating with us on a new monthly series of CMMI-related posts, "Just the FAQs." Our goal with these posts is to provide answers to the most frequently asked questions about CMMI, SCAMPI, engineering strategy and software process improvement. This month Pat discusses the new "Action Plan Re-Appraisal." Take it away! ~ the CMMI Appraiser]

We’re struggling with “tailoring” and “tailoring guidelines.” We think we understand the concepts, but we can’t seem to find the value. What are we missing?

[Note: Jeff addressed much of this in “Just the FAQs #06: What can we learn about process architecture from Justin Bieber?” It’s a terrific article, and if you haven’t watched the YouTube video that he included at the end, stop reading and watch it – it’s a hoot!]

PAT: I actively participate in many of the CMMI‐related Yahoo and LinkedIn forums. I’ve noticed that in many of the exchanges, participants tend to talk past, rather than to, one another, as illustrated in the following fictitious exchange:

A: Our lead appraiser said that we don’t have to perform some of the CMMI practices and we can still get our maturity level.

B: Well THAT’s not right – unless, of course, you are performing alternative practices instead.

A: Nope, he said we are not expected to have alternative practices in place either.

B: Your lead appraiser is dead wrong and should have his certification revoked!

This seemingly meaningful conversation goes back and forth for some time until…

A: Yeah, well, we are going for ML2 and he says that none of the ML3, ML4, or ML5 practices have to be performed – in fact, he won’t even be looking at them during our appraisal.

B: Oh, uh, well, never mind…

The point is that context is important!

Keeping that in mind, consider that “tailoring” can be performed at various levels of process decomposition, and you really have to understand the context in which tailoring is being discussed in order to have a meaningful dialogue. As I see it, there are four layers of tailoring:

First, there’s MEGA‐“T” tailoring – essentially, selecting a life cycle. Whether you are consciously aware of it or not, once you’ve chosen a given development life cycle, a whole lot of rituals associated with all of the other life cycles approved for use have just been tailored out. One could think of this as “tailoring with an ax.”

Next, there is CAPITAL‐“T”, or phase‐level tailoring. In very small projects, for example, the tailoring guidelines may allow the Requirements and Design phases to be combined into a single “Desirements” phase. Similarly, verification and validation activities might also be conducted in an integrated manner.

One could think of this as “tailoring with a knife.” Then there is small‐“t” tailoring – selecting among various methods within a given phase ‐ combining High Level Design and Low Level Design specs, for example, or selecting either formal inspection, desk‐check review, or buddy review for verifying a given document. This is probably what most people think about when they hear the word, “tailoring.” One might think of this as “tailoring with scissors.”

Finally, there is micro‐“t” tailoring – adjustments that are made much closer to the bottom of the food chain. For example, your formal inspection process may indicate that the work product to be inspected must be pre‐published at least 3.17 days in advance of the inspection meeting, thereby giving the participants adequate time to prepare. However, a tailoring guideline may indicate that the item can be pre‐published with less lead time provided each participant explicitly agrees that the truncated review period still provides them adequate preparation time. One might think of this as “tailoring with a needle.”

Until you establish the proper context for a given tailoring discussion, people may be doing a lot of talking, but very little communicating.

. . . . .

OK, now that I’ve (successfully?) argued that context is important to conduct a meaningful conversation about tailoring, let me set the context for what I see as one of the primary stumbling blocks with respect to deriving maximum value from the “tailoring guidelines” implemented by many/most organizations – the source of which is found in the last bit of the CMMI Glossary’s definition: “Tailoring guidelines describe what can and cannot be modified and identify components that are candidates for modification.”

Having conducted a ton of appraisals targeting ML3 or higher, I’ve noticed a couple of recurring themes when it comes to tailoring guidelines – both of which appear to align much too closely to this portion of the Glossary definition.

One of the recurring themes I see is sets of tailoring guidelines that indicate which process elements are mandatory, which are strongly encouraged, and which can be tailored out. These are clearly delineated by CMMI process area to ensure full credit is given to all GP3.1 practices when conducting an appraisal. This aligns much too closely with that bit in the Glossary that indicates “tailoring guidelines describe what can and cannot be modified.”

The other recurring theme I see is use of an Excel spreadsheet that lists every conceivable arrow in the organization’s process quiver. It includes every policy, life cycle description, process document, methodology, workflow, procedure, work instruction, guideline, template, form, checklist, metric, bathroom stall number, etc. that can be selected. The project manager is expected to meticulously mark those elements that the she intends to use. This theme aligns much too closely with that bit in the Glossary that indicates, “Tailoring guidelines… identify components that are candidates for modification.” Well, it’s more “inclusion” than “modification,” but the point still stands – explicit decisions are made regarding the project’s means of conducting its business – its “defined process.”

(Note: organizations having trouble conjuring up uses of formal decision making are likely to use DAR to make the tailoring decisions on each project – thereby bagging an appraisal twofer!)

Having a list of mandatory, suggested, and optional process elements provides some modest value, as does using the 1000+ row Excel checklist to make explicit project decisions regarding process element inclusion. But, like you, I believe there is more value to be had from the concept of tailoring.

Let me try to make the point by relying on everyone’s my favorite analogy – marathon running! Most long distance runners are concerned about the “Five H’s” – hills, heat, humidity, height (altitude), and head winds. Two of the five, hills and height, are attributes of the course and can be considered well in advance. The other three, heat, humidity, and head winds, are race day attributes – and, weather forecasts notwithstanding, you really won’t know what you’re up against until the day of the race.

Magazines like “Runner’s World” have lots of articles dedicated to such topics:

• Heat and How to Handle It
• Don’t Sweat Running in Humidity
• Running Against the Wind

These articles, written by very experienced marathoners, share secrets, hints, and tips about adjustments you may want to consider due to race‐day conditions. The author’s objective is to arm you with information based on their tried‐and‐true experience, but the decision to follow their advice or not is entirely up to you. After all, what works for them might not work for you.

So why not apply this same value‐added concept to process tailoring? In addition to following the standard script for establishing bland and boring tailoring guidelines (the one you know will pass the appraisal), why not write your own “articles” filled with tailoring guidance. Organizations that adopt such a “tips and hints” approach typically find it much more valuable than those that only follow the more mechanistic approaches above. Your project retrospectives, lesson learned sessions, or, my favorite, project post mortems (yet another project has died; let’s examine the corpse and see what we can find) will evolve from generating a repeatable list of what did and didn’t go well on the project, to lively discussions that also explore the question, “If we knew then what we know now, what might we have done differently?”

Like the “Runner’s World” articles, tailoring guidance is intended to provide experiential insight for future projects’ consideration. Codifying the secrets, hints, and tips of how to tailor project activities to overcome the inevitable hills, heat, humidity, height, and head winds will arm the next project team with advice that should make it easier for them to go the distance, and hopefully that’s the value that you were looking for.

Like this post?  Please forward to your nearest engineering or software professional!

© Copyright 2015: Process Assessment, Consulting & Training and Broadsword Solutions

“Just the FAQs” is written/edited by Pat O’Toole and Jeff Dalton. Please contact the authors at PACT.otoole@att.net and jeff@broadswordsolutions.com to suggest enhancements to their answers, or to provide an alternative response to the question posed. New questions are also welcomed!

Just the FAQs: SPECIAL EDITION! CMMI Institute Announces "Action Plan Re-Appraisal!"

[Dear Readers, our good friend Pat O’Toole, CMMI expert and seasoned consultant, is collaborating with us on a new monthly series of CMMI-related posts, "Just the FAQs." Our goal with these posts is to provide answers to the most frequently asked questions about the CMMI, SCAMPI, engineering strategy and software process improvement. This month Pat discusses the new "Action Plan Re-Appraisal." Take it away! ~ the CMMI Appraiser]

I heard that the CMMI Institute just released this thing called an “Action Plan Reappraisal.” What is it and why should I care?

PAT: Imagine that your organization just completed a maturity level 4 (ML4) SCAMPI A appraisal, but because you failed a goal in Organizational Process Performance (OPP) at ML4 and had an “Not Rated” goal in Risk Management (RSKM) at ML3, the SCAMPI A appraisal resulted in an ML2 rating. As Maxwell Smart might say, “You missed it by that much!”

Let’s assume that the ML4 rating was important to your team as a formal recognition of accomplishment and/or because a strategically important client requires it. In either case, you would want to remediate the goal‐impacting weaknesses, institutionalize the associated behavior changes, and be re‐appraised as soon as possible. Prior to the Action Plan Reappraisal being released, your organization would have to undergo another soup‐to‐nuts ML4 SCAMPI A appraisal – a very expensive and disruptive undertaking.

Now, however, the Appraisal Sponsor can elect to conduct an Action Plan Reappraisal (APR) – essentially a “delta appraisal.” An action plan is generated and executed by the organization to remediate the weaknesses and institutionalize the behavior changes, after which the appraisal team re‐evaluates the failed or not rated goals. If any of these goals are now rated “Satisfied,” the maturity level is regenerated. In the happiest of scenarios, both the OPP and RSKM goals would now be rated “Satisfied,” and ML4 will have been achieved!

Any questions?

Is there a time limit in which the Action Plan Reappraisal (APR) must be conducted?
The APR must conclude within 4 months of the SCAMPI A Final Findings presentation.

Why 4 months?
4 months was selected as an appropriate time frame for remediating minor goal‐threatening weaknesses; the APR is not intended for use when remediating major or systemic weaknesses.

Is there a limit regarding how many APRs may be conducted?
Only one APR can be conducted for a given SCAMPI A appraisal.

What happens if our current maturity level expires during this 4‐month period?
An appraisal result that exceeds its 3‐year validity period is no longer published on PARS. If the current SCAMPI A was completed and the associated APR is concluded within the 4‐month time limit, the Appraisal Sponsor can authorize the current results to be published in PARS. Publication in PARS would start after the ARP has been submitted to the CMMI Institute via the SCAMPI Appraisal System (SAS) and the CMMI Institute has concluded its Quality Audit.

How is the 3‐year validity period established for a SCAMPI A that had an APR?
Such an appraisal would expire 3 years after the delivery of the SCAMPI A Final Finding presentation.

Is it now required to conduct an APR if the target maturity level/capability level profile was not achieved in the SCAMPI A appraisal? If not required, who makes the decision to conduct an APR or not?
No, the APR is not required; it is an option available to the Appraisal Sponsor. The Lead Appraiser provides a good‐faith recommendation on whether an APR might be successful if pursued, but the final decision rests with the Appraisal Sponsor. If proceeding with an APR, this decision must be communicated to the CMMI Institute prior to submitting the SCAMPI A appraisal via SAS.

Typically, a goal fails due to one or more practices being characterized as “partially implemented.” or “not implemented.” Does the APR evaluate just these practices? Goals? Process area? Maturity level?
The APR appraises all of the “Unsatisfied” and “Not Rated” goals in the APR appraisal scope in the same manner as in a SCAMPI A: for each practice that supports the goal, documents are reviewed and interviews conducted to determine if that practice is fully, largely, partially, or not implemented on that project. These project‐level characterizations are then used to characterize the practice overall for the organization, or what’s referred to as the “OU practice characterization.” These OU practice characterizations are then used to rate the goal as “Satisfied” or “Unsatisfied.”

Note that it may be determined that additional goals need to be put in scope for the APR to properly evaluate the remediation. For example, if a Project Planning goal was rated “Unsatisfied” in the SCAMPI A, it might be appropriate to evaluate one or more goals from the Project Monitoring and Control process area as well. This decision is made by the Lead Appraiser.

Do we have to evaluate the remediated goals on the same projects, or can we do so on others?
The expectation is that the same projects will be evaluated. However, there may be cases where this is not feasible – for example, the project terminated prior to the remediation being applied, or the project is beyond the point in the lifecycle where the remediated practices are executed. In such cases, replacement projects with the same sampling characteristics may be evaluated instead.

What happens if some of the goals are still rated “Unsatisfied” or “Not Rated” at the end of the ARP?
In the example above, if the previously failed OPP goal was now rated “Satisfied” but the previously unrated RSKM goal was still “Not Rated,” the maturity level would be regenerated but the outcome would be the same – ML2. On the other hand, if the RSKM goal was now rated “Satisfied” and the OPP goal was still rated “Unsatisfied,” the regenerated maturity level rating would be ML3.

Must the organization have all failed and unrated goals re‐evaluated as part of the APR?
No, the Appraisal Sponsor can elect to conduct the APR on a subset of these. In the example above, the sponsor may decide to include only the unrated RSKM goal in the APR scope and, if the goal is successfully remediated, institutionalized, and rerated as “Satisfied,” ML3 will have been achieved.

Can the organization include new process areas in the APR? That is, in the example above, could they expand the model scope to include some or all of the ML5 process areas?
No, the model scope of the APR cannot be greater than that of the original SCAMPI A appraisal.

Do we have to use the same Lead Appraiser for the APR, and do all of the original appraisal team members have to be involved?
Yes, but... The original Lead Appraiser must be involved in all evaluation‐related APR activities, and with the exception of documentation review, these activities must be conducted onsite. In addition, for each failed or unrated goal in the APR model scope, to achieve continuity at least one member of the SCAMPI A mini‐team that evaluated the detailed evidence associated with that goal must also be involved in all of these onsite appraisal activities. In other words, there will be at least two members of the original SCAMPI A appraisal team, and maybe more, involved throughout. The other SCAMPI A appraisal team members must also be involved in OU practice characterizations and goal/capability/maturity level ratings, but they may participate remotely (e.g., via teleconference).

So in our example, if separate mini‐teams were responsible for evaluating the evidence and characterizing the practices for OPP and RSKM, then the Lead Appraiser and at least one member of each of these mini‐teams must participate in all evaluation‐related APR activities. All of the original appraisal team members must be involved in generating the OU practice characterizations and rating the goals.
Special provisions must be made with the CMMI Institute if any of the original appraisal team members, including the Lead Appraiser, cannot participate in the APR.

If there a limit on the number of failed goals that may be evaluated in an APR?
No, however, the Lead Appraiser must make a recommendation to the Appraisal Sponsor regarding the feasibility of achieving the objective. This may include reminding the sponsor that, for a goal to be rated “Satisfied,” the weaknesses have to be rectified AND the behavior changes have to be institutionalized.

Will the appraisal posting on PARS indicate that an APR was conducted?
No, the SCAMPI A and APR are considered a single appraisal event. The posting on PARS only indicates the final outcome of the appraisal; there is no indication that an APR was, or was not, conducted.

What do YOU perceive to be “the good, the bad, and the ugly” regarding Action Plan Reappraisals?
[Author’s note: Prior to this question, all answers were based on the changes introduced in the SCAMPI MDD v1.3b to accommodate the APR. In response to this question, Pat is providing his own opinion, which may or may not align with that of the CMMI Institute or his fellow Lead Appraisers!]

The Good:

Some organizations insist on being “rock solid” going into a SCAMPI A appraisal, as the fear and cost of failure is very high – in both financial and political terms. These organizations may conduct a series of Class C and Class B “dress rehearsals” until they have achieved 99.9% confidence that they will “pass” the SCAMPI A. The Action Plan Reappraisal option will allow such organizations to lower the overall cost by embracing more rating risk. The organization may forego one or more “dry run” appraisals if they know they can address any last potential goal threatening weaknesses over the ensuing four months.

Currently, some Lead Appraisers may struggle with the ethical decision to “flunk” an organization based on one or two marginal practices. They are torn between staying true to the model and method, and forcing the organization to incur the significant cost and disruption of another full SCAMPI A appraisal. The APR option makes the right decision much more palatable (for all “relevant stakeholders!”)

The Bad and the Ugly:

Some perceive that the introduction of the APR “tarnishes the brand.” That is, the SCAMPI A appraisal method is a rigorous evaluation of current organizational outputs and behaviors to determine if a predetermined set of CMMI goals is being satisfied. Appraisals bring focus to organizations that might otherwise be easily distracted. And since the SCAMPI A is typically perceived as the “gold standard” of appraisal methods, anything less, like the APR, is, well, less.

Some organizations may decide to accept too much risk. That is, rather than simply foregoing their 16th pre‐SCAMPI A dry run appraisal, they may throw prudence and caution to the wind and do NO pre‐ appraisal preparation, trusting that they can address any and all issues during their four month “grace period.” Damn the torpedoes, full steam ahead!

Finally, some CMMI‐adopting organizations may fear that an unscrupulous Lead Appraiser may see the Action Plan Reappraisal as a means of extracting even more revenue from them. “Oh gee, you failed a goal, I guess you’ll just have to pay me for another 4 or 5 days of service to get your level!”

Concluding Note
The SCAMPI Method Definition Document v1.3b, Section 4 is the Action Plan Reappraisal’s “official rule book.” If you would like a copy of this section, a copy of the full MDD, or if you have any additional questions or “Good, Bad, and Ugly” insights, please feel free to contact Pat at PACT.otoole@att.net.
Over time, I suspect that what “Dear Abby” said about birth control may also apply to the APR – “It’s better to have and not need than to need and not have.”

© Copyright 2015: Broadsword Solutions and Process Assessment, Consulting & Training
“Just the FAQs” is written/edited by Pat O’Toole and Jeff Dalton. Please contact the authors at pact.otoole@att.net and jeff@broadswordsolutions.com to suggest enhancements to their answers, or to provide an alternative response to the question posed. New questions are also welcomed!