Hey, Pat, We’ve just started implementing the CMMI and are really struggling with PPQA; any tips to point us in the right direction?
Pat: In some ways, implementing PPQA is like learning to play golf. If you don’t go to a pro early on, you’ll develop a lot of bad habits that will stay with you for a long, long time. So let me see if I can’t give you those requested tips before the bad habits take hold...
Let’s start by addressing one of the more common mistakes organizations make when they first embark on their
CMMI journey. Having been given an unreasonable time frame in which to achieve maturity level 2 (ML2), the newly formed EPG spawns a series of working groups – typically one for each of the seven ML2 process areas
*. For their assigned process area, each working group is charged with establishing the process infrastructure that supports organizational achievement of the coveted ML2 rating.
When the relatively clueless PPQA working group meets for the first time, they look up the PPQA process area to figure out what it is and how they might proceed. They collectively read through the one‐sentence PPQA Purpose Statement and figure, “OK, I kinda sorta get that...”
But when they start reading the Introductory Notes, they don’t get past the first bullet:
The Process and Product Quality Assurance process area involves the following activities:
• Objectively evaluating performed processes and work products against applicable process descriptions, standards, and procedures
They think to themselves, “WHAT process descriptions, standards, and procedures??” – we don’t have any of those bad boys yet!”
In this regard, some contend that PPQA should really be staged at maturity level 2.5. That is, until the other six working groups have established the process infrastructure for planning and monitoring projects, for measuring project activities, and for managing suppliers, requirements, and configuration items, there really isn’t a whole lot for the PPQA working group to do!
OK, so the PPQA working group hibernates until the other working groups have generated much of their process stuff. Now what?
Well, according to the CMMI, PPQA objectively evaluates two types of things – processes and work products. To that end, most PPQA groups generate a series of checklists, typically one checklist for each CMMI process area (a less CMMI‐oriented approach is provided below). Such checklists are intended to do triple duty – not only do they cover GP2.9 in their respective process areas, but collectively they are also intended to cover both process compliance (PPQA SP1.1) and work product compliance (PPQA SP1.2).
Left to their own devices, most PPQA groups wind up focusing much more intently on work product compliance than process compliance. That’s understandable as it is much easier to wrap your head around work products that you can touch and feel (and sometimes smell) than it is process stuff which is a tad more amorphous. After all, gravitational pull is based on mass (work products) rather than energy (processes).
But since I am trying to protect you from “your own devices,” let me suggest a way to establish a better balance:
1. For each documented process, include “Appendix A” which is the PPQA checklist for that process. To populate this process‐based checklist, ask yourself, “What are the most important steps in the process, and how would we know that they were executed properly?”
2. For each template, include “Appendix A” which is the PPQA checklist for the resulting work product. To populate this work product‐based checklist, ask yourself,” What are the most important elements in the work product generated based on this template?”
Building your checklists in this manner will provide a proper balance between process compliance and work product compliance, AND it will focus PPQA reviews on what you have deemed to be your most important process steps and work product elements. Remember that PPQA ensures that the work is being performed according to your process, NOT according to the CMMI – that will be handled by appraisals found in Organizational Process Focus (OPF).
Before I leave the subject of PPQA checklists, let me provide one more tip...
When conducting a SCAMPI A appraisal, each specific and generic practice in scope will ultimately be characterized as “Fully Implemented,” “Largely Implemented,” “Partially Implemented,” or “Not Implemented” or FI/LI/PI/NI. This four‐point scale enables the appraisal team to focus organizational attention on areas that are a bit weak (LI), very weak (PI), or abundantly absent (NI).
Unfortunately, when generating PPQA checklists, most organizations write questions of the “Yes/No” variety. Such an approach typically leads to project folks doing the absolute minimum amount of work necessary to rationalize a “Yes” in the review. Using a more robust scale, such as that employed by the SCAMPI A, allows the PPQA group to treat each review more as a consulting opportunity than a compliance check. I encourage you to give strong consideration to adopting a scale such “Fully Compliant,” “Largely Compliant,” “Partially Compliant,” and “Not Compliant.” Rather than debating “Yes” vs. “No,” the more robust scale leads to discussions that start with, “I gave you ‘Largely Compliant’ rather than ‘Fully Compliant’ because...”
In addition, using this approach enables you to generate a numeric score for your PPQA reviews by averaging the characterization of each checklist item using a scale such as: FC = 100; LC = 80; PC = 30; and NC = 0. Some organizations use an approach like this to drive their sampling selection. If you score 90 or above, your project won’t be reviewed on this process for the next 6 months; between 75 and 90, we’ll see you in 3 months; less than 75, we’ll be reviewing 100% of the implementations until you score high enough to earn a reprieve. You’ll also be placed on the list for earlier intervention (e.g., coaching) on your next project to make sure you “get it.”
All right, the other working groups are churning out their final deliverables and the PPQA working group is partnering with them to establish Appendix A for each element. Life is good!
However, just because the working groups “build it” does not mean that “they will come!” It’s going to take a while for (1) the rough edges to be sanded off; and (2) the value of performing the work in this more process‐disciplined way to be recognized. The premature introduction of PPQA reviews is likely to be counterproductive as “objectively verifying compliance” to processes that have not yet earned the trust and respect of those being encouraged to use them changes “encouraged” to “forced” – and NOBODY likes to be forced to do something.
Here’s how to introduce PPQA services for your new or significantly changed process/work product elements in a manner that is much less likely to encounter significant resistance:
1. Work with the EPG to find somebody willing to pilot the new or modified process/work product and convince them to agree to subject themselves to a “free” PPQA review.
2. When the process has been executed or the work product completed, generate a hard copy of the corresponding PPQA review checklist (Appendix A).
3. Sitting with the person who agreed to the pilot, fill out the PPQA checklist together, discussing the “proper” disposition of each item (FC/LC/PC/NC).
a. Note that the results of this review should not be shared with anyone else.
b. Through this exercise, you are simply “validating” the checklist items and
developing the norms for “scoring” the level of compliance for each item.
4. Based on the pilot, refine the checklist as necessary (add, change, delete items).
5. Recruit a willing participant to conduct a second “free” pilot.
6. At the end of the second process execution/work product population, generate multiple
copies of the PPQA checklist.
7. Recruit the person that conducted the first pilot, plus a number of others who will also
be using the new/modified process or template in the future.
8. Give them each a copy of the checklist and have them INDEPENDENTLY conduct the review.
9. After everyone is done, compare each item’s compliance scores and discuss the
inevitable differences.
a. Hopefully, you and the person who conducted the first pilot are reasonably well
aligned.
b. Through this exercise, you are trying to:
i. Ensure that the checklist items are focused on those process/work product elements that are the most relevant
ii. Establish norms for the proper scoring of each checklist item
iii. Demystify the way PPQA will be applied to this new/modified thingy.
c. Once again, these results should not get communicated to anyone else.
d. Engage the group by eliciting suggestions for tweaking the PPQA checklist.
i. Incorporate as many changes as you can, even if they are inconsequential. It is part of your marketing strategy – changing the view from “your” PPQA checklists to “our” PPQA checklists.
10. OK, now you’re ready to “go live” with the new process element and the associated PPQA checklist.
a. Think about establishing a rule that the first time a person is reviewed for a specific process or template, the results are not reported (i.e., it’s a freebie)''
b. You should try to establish the role of PPQA as that of “process coach” rather than that of “process proctologist” (by, among other things, using words like “PPQA review” rather than “PPQA audit”).
OK, we’re into the lightning round – quick tips with little explanation...
1. Some organizations perform their PPQA reviews after a life cycle phase has ended or, worse yet, after the project has completed. I would STRONGLY encourage you to conduct “in progress” reviews as most people find that information from a health check provides more timely feedback than that from an autopsy!
2. “Objectivity” is the key to successful PPQA, not necessarily “independence.” You may find that some PPQA reviews provide more value if they are led by subject matter experts rather than a quality professional. Or it may be better to use a round‐robin approach: Project A provides PPQA reviews for Project B; Project B reviews Project C; and Project C reviews Project A. Just remember to equip the PPQA reviewers with the skills and knowledge to do a professional job.
3. Some organizations rely on peer reviews for the “work product compliance” bits of PPQA. This can be effective, especially if enabled by: (1) the Appendix A approach to PPQA checklists; and (2) the explicit designation of the PPQA‐like role to a specific person. I know that “Quality is everybody’s business!” but PPQA is much more likely to be done properly if it is entrusted to a single qualified person.
4. Don’t have the EPG members also serve as the PPQA staff. PPQA is the “protector of the status quo,” while EPG members are forever looking for ways to evolve the status quo. Having the same people do both is likely to result in head explosions.
5. Don’t forget about PPQA GP2.9, typically referred to as “PPQA of PPQA.” Consider using the round‐robin approach suggested in #2 above to achieve objectivity. When PPQA non‐compliance issues are found, be sure to model “good reviewee behavior.” Don’t whine and pout; simply address the issues – and then tell everyone you purposely introduced those non‐compliances just so you could model good reviewee behavior!
6. Suggest that some management processes also enjoy PPQA services. After all, if it’s good for the project teams, then it should be good for management as well! (And it might just reduce the number of “golf course commitments” made by management).
7. Establish indicators of PPQA value. For example, count the number of incoming calls to PPQA where their value‐added services are being requested, and subtract the number of outgoing calls made by PPQA where they are informing the unsuspecting of an upcoming review. If this metric is positive, then congratulations – your value is being recognized! If it’s negative, then change how you’re operating to provide more value to those that you are reviewing (or bribe people to call in to fulfill O’Toole’s Law on Measurement: “What gets measured gets manipulated!”)
