[Dear Readers, our good friend Pat O’Toole, CMMI expert and seasoned consultant, is collaborating with us on a new monthly series of CMMI-related posts, "Just the FAQs." Our goal with these posts is to provide answers to the most frequently asked questions about the CMMI, SCAMPI, engineering strategy and software process improvement. This month Pat discusses the new "Action Plan Re-Appraisal." Take it away! ~ the CMMI Appraiser]
I heard that the CMMI Institute just released this thing called an “Action Plan Reappraisal.” What is it and why should I care?
PAT: Imagine that your organization just completed a maturity level 4 (ML4) SCAMPI A appraisal, but because you failed a goal in Organizational Process Performance (OPP) at ML4 and had an “Not Rated” goal in Risk Management (RSKM) at ML3, the SCAMPI A appraisal resulted in an ML2 rating. As Maxwell Smart might say, “You missed it by that much!”
Let’s assume that the ML4 rating was important to your team as a formal recognition of accomplishment and/or because a strategically important client requires it. In either case, you would want to remediate the goal‐impacting weaknesses, institutionalize the associated behavior changes, and be re‐appraised as soon as possible. Prior to the Action Plan Reappraisal being released, your organization would have to undergo another soup‐to‐nuts ML4 SCAMPI A appraisal – a very expensive and disruptive undertaking.
Now, however, the Appraisal Sponsor can elect to conduct an Action Plan Reappraisal (APR) – essentially a “delta appraisal.” An action plan is generated and executed by the organization to remediate the weaknesses and institutionalize the behavior changes, after which the appraisal team re‐evaluates the failed or not rated goals. If any of these goals are now rated “Satisfied,” the maturity level is regenerated. In the happiest of scenarios, both the OPP and RSKM goals would now be rated “Satisfied,” and ML4 will have been achieved!
Any questions?
Is there a time limit in which the Action Plan Reappraisal (APR) must be conducted?
The APR must conclude within 4 months of the SCAMPI A Final Findings presentation.
Why 4 months?
4 months was selected as an appropriate time frame for remediating minor goal‐threatening weaknesses; the APR is not intended for use when remediating major or systemic weaknesses.
Is there a limit regarding how many APRs may be conducted?
Only one APR can be conducted for a given SCAMPI A appraisal.
What happens if our current maturity level expires during this 4‐month period?
An appraisal result that exceeds its 3‐year validity period is no longer published on PARS. If the current SCAMPI A was completed and the associated APR is concluded within the 4‐month time limit, the Appraisal Sponsor can authorize the current results to be published in PARS. Publication in PARS would start after the ARP has been submitted to the CMMI Institute via the SCAMPI Appraisal System (SAS) and the CMMI Institute has concluded its Quality Audit.
How is the 3‐year validity period established for a SCAMPI A that had an APR?
Such an appraisal would expire 3 years after the delivery of the SCAMPI A Final Finding presentation.
I heard that the CMMI Institute just released this thing called an “Action Plan Reappraisal.” What is it and why should I care?
PAT: Imagine that your organization just completed a maturity level 4 (ML4) SCAMPI A appraisal, but because you failed a goal in Organizational Process Performance (OPP) at ML4 and had an “Not Rated” goal in Risk Management (RSKM) at ML3, the SCAMPI A appraisal resulted in an ML2 rating. As Maxwell Smart might say, “You missed it by that much!”
Let’s assume that the ML4 rating was important to your team as a formal recognition of accomplishment and/or because a strategically important client requires it. In either case, you would want to remediate the goal‐impacting weaknesses, institutionalize the associated behavior changes, and be re‐appraised as soon as possible. Prior to the Action Plan Reappraisal being released, your organization would have to undergo another soup‐to‐nuts ML4 SCAMPI A appraisal – a very expensive and disruptive undertaking.
Now, however, the Appraisal Sponsor can elect to conduct an Action Plan Reappraisal (APR) – essentially a “delta appraisal.” An action plan is generated and executed by the organization to remediate the weaknesses and institutionalize the behavior changes, after which the appraisal team re‐evaluates the failed or not rated goals. If any of these goals are now rated “Satisfied,” the maturity level is regenerated. In the happiest of scenarios, both the OPP and RSKM goals would now be rated “Satisfied,” and ML4 will have been achieved!
Any questions?
Is there a time limit in which the Action Plan Reappraisal (APR) must be conducted?
The APR must conclude within 4 months of the SCAMPI A Final Findings presentation.
Why 4 months?
4 months was selected as an appropriate time frame for remediating minor goal‐threatening weaknesses; the APR is not intended for use when remediating major or systemic weaknesses.
Is there a limit regarding how many APRs may be conducted?
Only one APR can be conducted for a given SCAMPI A appraisal.
What happens if our current maturity level expires during this 4‐month period?
An appraisal result that exceeds its 3‐year validity period is no longer published on PARS. If the current SCAMPI A was completed and the associated APR is concluded within the 4‐month time limit, the Appraisal Sponsor can authorize the current results to be published in PARS. Publication in PARS would start after the ARP has been submitted to the CMMI Institute via the SCAMPI Appraisal System (SAS) and the CMMI Institute has concluded its Quality Audit.
How is the 3‐year validity period established for a SCAMPI A that had an APR?
Such an appraisal would expire 3 years after the delivery of the SCAMPI A Final Finding presentation.
Is it now required to conduct an APR if the target maturity level/capability level profile was not achieved in the SCAMPI A appraisal? If not required, who makes the decision to conduct an APR or not?
No, the APR is not required; it is an option available to the Appraisal Sponsor. The Lead Appraiser provides a good‐faith recommendation on whether an APR might be successful if pursued, but the final decision rests with the Appraisal Sponsor. If proceeding with an APR, this decision must be communicated to the CMMI Institute prior to submitting the SCAMPI A appraisal via SAS.
Typically, a goal fails due to one or more practices being characterized as “partially implemented.” or “not implemented.” Does the APR evaluate just these practices? Goals? Process area? Maturity level?
The APR appraises all of the “Unsatisfied” and “Not Rated” goals in the APR appraisal scope in the same manner as in a SCAMPI A: for each practice that supports the goal, documents are reviewed and interviews conducted to determine if that practice is fully, largely, partially, or not implemented on that project. These project‐level characterizations are then used to characterize the practice overall for the organization, or what’s referred to as the “OU practice characterization.” These OU practice characterizations are then used to rate the goal as “Satisfied” or “Unsatisfied.”
Note that it may be determined that additional goals need to be put in scope for the APR to properly evaluate the remediation. For example, if a Project Planning goal was rated “Unsatisfied” in the SCAMPI A, it might be appropriate to evaluate one or more goals from the Project Monitoring and Control process area as well. This decision is made by the Lead Appraiser.
Do we have to evaluate the remediated goals on the same projects, or can we do so on others?
The expectation is that the same projects will be evaluated. However, there may be cases where this is not feasible – for example, the project terminated prior to the remediation being applied, or the project is beyond the point in the lifecycle where the remediated practices are executed. In such cases, replacement projects with the same sampling characteristics may be evaluated instead.
What happens if some of the goals are still rated “Unsatisfied” or “Not Rated” at the end of the ARP?
In the example above, if the previously failed OPP goal was now rated “Satisfied” but the previously unrated RSKM goal was still “Not Rated,” the maturity level would be regenerated but the outcome would be the same – ML2. On the other hand, if the RSKM goal was now rated “Satisfied” and the OPP goal was still rated “Unsatisfied,” the regenerated maturity level rating would be ML3.
Must the organization have all failed and unrated goals re‐evaluated as part of the APR?
No, the Appraisal Sponsor can elect to conduct the APR on a subset of these. In the example above, the sponsor may decide to include only the unrated RSKM goal in the APR scope and, if the goal is successfully remediated, institutionalized, and rerated as “Satisfied,” ML3 will have been achieved.
Can the organization include new process areas in the APR? That is, in the example above, could they
expand the model scope to include some or all of the ML5 process areas?
No, the model scope of the APR cannot be greater than that of the original SCAMPI A appraisal.
Do we have to use the same Lead Appraiser for the APR, and do all of the original appraisal team members have to be involved?
Yes, but... The original Lead Appraiser must be involved in all evaluation‐related APR activities, and with the exception of documentation review, these activities must be conducted onsite. In addition, for each failed or unrated goal in the APR model scope, to achieve continuity at least one member of the SCAMPI A mini‐team that evaluated the detailed evidence associated with that goal must also be involved in all of these onsite appraisal activities. In other words, there will be at least two members of the original SCAMPI A appraisal team, and maybe more, involved throughout. The other SCAMPI A appraisal team members must also be involved in OU practice characterizations and goal/capability/maturity level ratings, but they may participate remotely (e.g., via teleconference).
So in our example, if separate mini‐teams were responsible for evaluating the evidence and characterizing the practices for OPP and RSKM, then the Lead Appraiser and at least one member of each of these mini‐teams must participate in all evaluation‐related APR activities. All of the original appraisal team members must be involved in generating the OU practice characterizations and rating the goals.
Special provisions must be made with the CMMI Institute if any of the original appraisal team members, including the Lead Appraiser, cannot participate in the APR.
If there a limit on the number of failed goals that may be evaluated in an APR?
No, however, the Lead Appraiser must make a recommendation to the Appraisal Sponsor regarding the feasibility of achieving the objective. This may include reminding the sponsor that, for a goal to be rated “Satisfied,” the weaknesses have to be rectified AND the behavior changes have to be institutionalized.
Will the appraisal posting on PARS indicate that an APR was conducted?
No, the SCAMPI A and APR are considered a single appraisal event. The posting on PARS only indicates the final outcome of the appraisal; there is no indication that an APR was, or was not, conducted.
What do YOU perceive to be “the good, the bad, and the ugly” regarding Action Plan Reappraisals?
[Author’s note: Prior to this question, all answers were based on the changes introduced in the SCAMPI MDD v1.3b to accommodate the APR. In response to this question, Pat is providing his own opinion, which may or may not align with that of the CMMI Institute or his fellow Lead Appraisers!]
No, the model scope of the APR cannot be greater than that of the original SCAMPI A appraisal.
Do we have to use the same Lead Appraiser for the APR, and do all of the original appraisal team members have to be involved?
Yes, but... The original Lead Appraiser must be involved in all evaluation‐related APR activities, and with the exception of documentation review, these activities must be conducted onsite. In addition, for each failed or unrated goal in the APR model scope, to achieve continuity at least one member of the SCAMPI A mini‐team that evaluated the detailed evidence associated with that goal must also be involved in all of these onsite appraisal activities. In other words, there will be at least two members of the original SCAMPI A appraisal team, and maybe more, involved throughout. The other SCAMPI A appraisal team members must also be involved in OU practice characterizations and goal/capability/maturity level ratings, but they may participate remotely (e.g., via teleconference).
So in our example, if separate mini‐teams were responsible for evaluating the evidence and characterizing the practices for OPP and RSKM, then the Lead Appraiser and at least one member of each of these mini‐teams must participate in all evaluation‐related APR activities. All of the original appraisal team members must be involved in generating the OU practice characterizations and rating the goals.
Special provisions must be made with the CMMI Institute if any of the original appraisal team members, including the Lead Appraiser, cannot participate in the APR.
If there a limit on the number of failed goals that may be evaluated in an APR?
No, however, the Lead Appraiser must make a recommendation to the Appraisal Sponsor regarding the feasibility of achieving the objective. This may include reminding the sponsor that, for a goal to be rated “Satisfied,” the weaknesses have to be rectified AND the behavior changes have to be institutionalized.
Will the appraisal posting on PARS indicate that an APR was conducted?
No, the SCAMPI A and APR are considered a single appraisal event. The posting on PARS only indicates the final outcome of the appraisal; there is no indication that an APR was, or was not, conducted.
What do YOU perceive to be “the good, the bad, and the ugly” regarding Action Plan Reappraisals?
[Author’s note: Prior to this question, all answers were based on the changes introduced in the SCAMPI MDD v1.3b to accommodate the APR. In response to this question, Pat is providing his own opinion, which may or may not align with that of the CMMI Institute or his fellow Lead Appraisers!]
The Good:
Some organizations insist on being “rock solid” going into a SCAMPI A appraisal, as the fear and cost of failure is very high – in both financial and political terms. These organizations may conduct a series of Class C and Class B “dress rehearsals” until they have achieved 99.9% confidence that they will “pass” the SCAMPI A. The Action Plan Reappraisal option will allow such organizations to lower the overall cost by embracing more rating risk. The organization may forego one or more “dry run” appraisals if they know they can address any last potential goal threatening weaknesses over the ensuing four months.
Currently, some Lead Appraisers may struggle with the ethical decision to “flunk” an organization based on one or two marginal practices. They are torn between staying true to the model and method, and forcing the organization to incur the significant cost and disruption of another full SCAMPI A appraisal. The APR option makes the right decision much more palatable (for all “relevant stakeholders!”)
The Bad and the Ugly:
Some perceive that the introduction of the APR “tarnishes the brand.” That is, the SCAMPI A appraisal method is a rigorous evaluation of current organizational outputs and behaviors to determine if a predetermined set of CMMI goals is being satisfied. Appraisals bring focus to organizations that might otherwise be easily distracted. And since the SCAMPI A is typically perceived as the “gold standard” of appraisal methods, anything less, like the APR, is, well, less.
Some organizations may decide to accept too much risk. That is, rather than simply foregoing their 16th pre‐SCAMPI A dry run appraisal, they may throw prudence and caution to the wind and do NO pre‐ appraisal preparation, trusting that they can address any and all issues during their four month “grace period.” Damn the torpedoes, full steam ahead!
Finally, some CMMI‐adopting organizations may fear that an unscrupulous Lead Appraiser may see the Action Plan Reappraisal as a means of extracting even more revenue from them. “Oh gee, you failed a goal, I guess you’ll just have to pay me for another 4 or 5 days of service to get your level!”
Concluding Note
The SCAMPI Method Definition Document v1.3b, Section 4 is the Action Plan Reappraisal’s “official rule book.” If you would like a copy of this section, a copy of the full MDD, or if you have any additional questions or “Good, Bad, and Ugly” insights, please feel free to contact Pat at PACT.otoole@att.net.
Over time, I suspect that what “Dear Abby” said about birth control may also apply to the APR – “It’s better to have and not need than to need and not have.”
© Copyright 2015: Broadsword Solutions and Process Assessment, Consulting & Training
“Just the FAQs” is written/edited by Pat O’Toole and Jeff Dalton. Please contact the authors at pact.otoole@att.net and jeff@broadswordsolutions.com to suggest enhancements to their answers, or to provide an alternative response to the question posed. New questions are also welcomed!
No comments:
Post a Comment