Using Agile on a Waterfall project – What are the challenges?

What are the main challenges of using an Agile methodology on a Waterfall project? ~ Quora User

Dear Readers – Below is my response to a question posted on Quora.com by an Agile Leader who is thinking about using Agile methods in a "waterfall" environment - which can mean different things to different organizations, which I detail below. Regular users of Quora.com know this social media platform as a place for high-level discourse about engineering strategy and software process improvement, and the conversation has been especially thought-provoking recently. Enjoy! ~ the CMMI Appraiser

Dear Quora User,

The agile community has done a great job at defining and branding agile frameworks like Scrum and XP, although technically, they’re not methodologies. But they do provide great guidance for how to do things, what the roles are, and in what sequence actions are to take place.

“Waterfall,” is more ambiguous. There really isn’t a “waterfall community,” or a methodology that is called waterfall - although it is generally thought of as phased, task oriented, and planned for the project duration. Because of its use of project managers, strong oversight, metrics, and reporting, it’s often thought of (by agilistas anyway) as “low trust,” where “agile” is thought of (by agilistas) as “high trust,” due the the focus on self-organization and relying on the people who are doing the work to make important decisions.

A lot of people think they are implementing “agile” by adopting some techniques (ceremonies) such as the daily standup, or using a structure like a backlog for requirements. Some implement a tool like Jira (Atlassian), or TFS (Microsoft) and then call themselves “agile.” In the real world, “Agile” is a philosophical approach to running an organization based on a set of core values that include things such as “Transparency,” “Collaboration,” “Fail-fast” and others. Once an organization establishes their values (and people subscribe to them), then a set of methods, tools, and frameworks are established that align or trace to them.

For instance, “daily standup” traces to transparency and collaboration. “Sprints” trace to “fail-fast,” and so on. Scrum and XP are good examples of frameworks that have done a good job of aligning with agile values. Opinions vary, but there a some people who say SAFe (Scaled Agile) doesn’t do as good a job - one reason it is resonating so well with the government!

So, if by “waterfall” you mean “low trust” and “command-and-control,” you can use all the agile ceremonies you want, but you’ll struggle and not get a lot of value from them. If you mean that your work is “phased” with project mangers, and you have a strong culture based on agile values, you “are agile,” although you are not using Scrum or XP (for example). It’s possible to use waterfall techniques and still be agile, although it’s not very common in the industry. It’s also important to realize that Scrum does not equal agile, although it is a manifestation of agile values, if implemented properly.

If you choose to run a project using “waterfall,” but you want to encourage teams running their sub-projects to use Scrum, you are free to do that (in fact, I recommend it), but it all starts with leadership, who need to demonstrate a culture of agility and ensure the values are adopted by the organization - regardless of the techniques you use.

Good luck!

Like this blog? Forward to your nearest engineering or software leader!

Jeff Dalton is a Certified SCAMPI Lead Appraiser, Certified CMMI Instructor, author and consultant with years of real-world experience with the CMMI in all types of organizations.

Visit www.broadswordsolutions.com for more information about engineering strategy, performance innovation, software process improvement and running a successful CMMI program.

Where can I get CMMI V2.0-related content for studying?

Where could I get CMMI V2.0-related content for studying? I can’t get it because it is paid. I am working as a CMMI implementer in an IT company. ~ Quora User 

Dear Readers – recently I've been answering questions about performance improvement, CMMI, and Agile from engineering and software professionals and leaders, on Quora.com. Below is my response to a CMMI adopter who wants to be understand the new Model upgrade, CMMI V2.0. Enjoy! ~ the CMMI Appraiser

Dear Quora User, the only place to get the new version of CMMI V2.0 in online format. There is no longer a book that you can purchase or download. Neither are there CMMI-DEV or CMMI-SVC "constellations," just one model with different “views." These are some of the many changes you'll find in the new Model upgrade.

The reason for this is because the CMMI Institute has moved to an online subscription model, where you can only see the detailed content if you have an annual license. The price on this has been fluctuating as they experiment, but it’s somewhere between $400-1500US per user, depending on what you are buying.

The new CMMI is separated into two sections - protected content (anything that describes the meaning of the practices, including examples) and un-protected content (names of the Practice Areas (new terminology) and Practices. So you can see the list - just not what THEY think the meaning of each item is! For that you must buy a license.

Here are a couple of new things to get you started:

• Process Areas are now Practice Areas
• The sub-Practices have been eliminated
• The Generic Practices have been eliminated
• There are new Practice Areas for Governance and Implementation Infrastructure to replace and enhance the content from the Generic Practices
• New CMMI appraisals will see the CMMI Institute determining the sample, not the Lead Appraiser or Sponsor
• Each Practice Area has multiple levels within it (Practice Groups)

I’ve written pretty extensively on CMMI V2.0 here my blog, and have also done a number of videos (including interviews with the CMMI Institute’s Chief Architect) at CMMI-TV.com.

For more in-depth learning about CMMI V2.0, check out our upcoming CMMI V2.0 webinar, "CMMI V2.0 Is Here! Why It's the Best CMMI Yet.

Also consider signing up for a CMMI V2.0 Training class (where you will get the license as part of the training).

Good luck!

Like this blog? Forward to your nearest engineering or software leader!

Jeff Dalton is a Certified SCAMPI Lead Appraiser, Certified CMMI Instructor, author and consultant with years of real-world experience with the CMMI in all types of organizations.

Visit www.broadswordsolutions.com for more information about engineering strategy, performance innovation, software process improvement and running a successful CMMI program.

Large organizations adopting agile: How well is it going?

Jeff, in your exploration of over 300 organizations, a lot of them are the very big ones. When we look at the adoption profile, it's the late majority, the laggards, that are getting into Agile now. How well is that going? ~ Shane H.

[Editor's Note: During the coming weeks, this CMMI Appraiser will share excerpts from a recent conversation with Shane Hastie, Lead Editor for Culture & Methods on the “Engineering Culture by InfoQ” podcast about leadership, and the kind of leadership that is needed in today’s Agile world. Today’s blog post is the first installment. Listen to the full interview at http://bit.ly/infoqpodcast]

Well, Shane, this isn't a popular opinion, but based on the empirical data we've collected, it's not going well. The marketing of agility is doing far better than the reality of agility, for a lot of different reasons. I always say it's the early adopters who kind of set the tone for adoption.

Of course, the early adopters of Agile tended to be smaller, more compact organizations, and subsections of companies or organizations that were trying Agile. They had great success with Scrum and XP and some of those things. But as Agile has scaled, and as more and more big companies have adopted Agile, it hasn't been as successful.

The reason? Start with the culture. It’s the culture of the company that drives the behaviors of the people. Small startups, small subsections of teams, tend to have very collaborative, transparent cultures. But look at large organizations like General Motors, the Department of Defense, Lockheed Martin, and Nationwide Insurance. All of these organizations have 300 or 400 teams working together, and the culture of those teams is the culture of their organization.

I’ll use General Motors as kind of a metaphor because there are a lot of big companies like them. Why are General Motors’ software teams very document-heavy and very process-heavy with lots of low trust oversight? Because that's how the company operates. It has nothing to do with their software teams. It has everything to do with the culture of the company.

The same has been true with the CMMI. The CMMI has a reputation of being kind of a heavyweight process-burden model. But the only reason people think that is because the early adopters were General Electric, General Motors, Lockheed Martin and the DoD. These are organizations were already heavy, overburdened, over-processed companies, so when they adopted CMMI, they made it a process-heavy model. And when they adopted Agile, what do you think they did? They made it heavy, and over-burdened.

In my work assessing the agile performance of large organizations like these, I’ve found that they all have Project Managers that do tasking. They use Microsoft Project. They do a lot of things that you would think were Agile anti-patterns, or antithetical to agile values. They all do them!

It's only the smallest companies that are running Scrum projects using the Scrum roles as defined in the Scrum Guide. Most larger companies have Project Managers, Architects, Directors, Process Quality, and audits. They have all the things that you would say agile teams would never have. I observed this early on, and said, “Hey, there's a culture clash.”

I'm sure your audience knows what a “type mismatch” is in software, Shane. We call this phenomenon an “organizational type mismatch,” when the values and philosophy of the company are at odds with the values and philosophies of Agility.

When you look at the core agile values – collaboration, transparency, fail-fast, and so on – you see they are directly antithetical to the corporate philosophies of a company like General Electric, for example, and other large organizations that are very much command-and-control, low-trust, document-focused, audit-focused, etc. We noticed right away that senior management -- CIOs and CTOs especially -- were keen on becoming more Agile. But they weren't so keen on changing the corporate culture. That itself was an impediment to their success, and continues to be today. That's why I say it's not going well.

# # #

I hope my readers have enjoyed this segment of my interview with Shane Hastie on the InfoQ podcast. We'll be talking more about leadership, and whether leadership is more or less important in today’s Agile world, in the next segment. Please check back soon.

http://bit.ly/infoqpodcast

For those interested in a deeper dive into learning about Agile Leadership, please visit agilecxo.org for white papers, blog posts, podcasts and performance models to help software and engineering executives guide their organizations to be more agile, from top to bottom.

Like this blog? Forward to your nearest engineering or software leader!

Jeff Dalton is a Certified SCAMPI Lead Appraiser, Certified CMMI Instructor, author and consultant with years of real-world experience with the CMMI in all types of organizations.

Visit www.broadswordsolutions.com for more information about engineering strategy, performance innovation, software process improvement and running a successful CMMI program.

What's the best way to provide guidance for Scrum teams?

What's the best way of making sure that teams working on different projects follow the same Scrum process? Would you use a rigid approach or allow a degree of flexibility for each team? ~ Quora User

Dear Readers – Engineering strategy and software process improvement are popular topics on Quora.com, and I try to go in and answer questions as frequently as I can. Below is my response to an Agile Leader who wants to be more successful with Scrum. Enjoy! ~ the CMMI Appraiser

Dear Quora User,

Scrum, one of several popular frameworks that fall under the “agile” umbrella, is an “empirical process model.” This means, for this context, that teams learn and improve as they go, and may end up with different approaches over time, as compared to other teams.

Providing Scrum teams with guiderails 

The very nature of empirical models means that a “rigid” approach is not acceptable, and would corrupt the very architecture you’ve adopted - probably leading to negative results.

That doesn’t mean teams shouldn’t be provided with guidance (sometimes called "guiderails" in the community) on how the company expects them to perform.

Like a great musician, artist, or writer, the best performing scrum teams have MASTERED the art of scrum before they go off an make a lot of changes, versus the worst garage band, that just turns up the volume and celebrates their rebelliousness.

One performance model that is gaining in popularity is the Agile Performance Holarchy (APH) from AgileCxO.org. This set of guiderails is intended to influence leadership to set expectations of performance using a very disciplined approach, but then encourages teams to improve, adjust, learn, and modify based on their needs - but only AFTER mastering the craft. Makes sense.

Good luck!

Like this blog? Forward to your nearest engineering or software leader!

Jeff Dalton is a Certified SCAMPI Lead Appraiser, Certified CMMI Instructor, author and consultant with years of real-world experience with the CMMI in all types of organizations.

Visit www.broadswordsolutions.com for more information about engineering strategy, performance innovation, software process improvement and running a successful CMMI program.

Why is there such a disconnect in how Agile is marketed and how Agile is adopted?

Dear CMMI Appraiser,

For defense industry CMMI organizations like ours, why does there seem to be such a disconnect between the way Agile is marketed and the way it is actually adopted by teams? ~ Quora User

Dear Readers – Because there is often a high level of discourse at Quora.com about engineering strategy and software process improvement, I try to go in and answer questions as frequently as I can. Below is my response to an engineering professional in the defense industry who wants to be more successful with agile. Enjoy!

Dear Quora User,

Because reality is a harsh teacher.

Agile: why is there a disconnect?

When you consider that most marketing is about the “happy path,” and by its very nature doesn’t advertise the complexity of the product or service, or its potential points of failure, it makes more sense. Agile is complex, hard, and fraught with risk. It’s also rewarding.

Agile is popular because it espouses self-organization, collaboration, transparency, optimism, trust, rapid delivery of value, and celebrating early failure (among other things). These are all things that, in theory, produce more value, more quickly than what we sometimes call “waterfall,” or “traditional” project management techniques.

Then comes the reality. Companies, customers, teams, and humans are often NOT collaborative, optimistic, trusting, or willing to celebrate early failure. This is especially true in the government, health care, and aerospace industries. If they naturally were all those things, it would be simple to adopt and embrace agile values, ceremonies, and techniques, and everyone would happily march down the path to a more agile future. But they’re not - almost never.

For instance, as a reaction to top-down, command-and-control approaches to managing projects, Scrum doesn’t identify “project manager” as a role. After all, a Scrum team is self-organizing. Why would they need a manager? That sounds great - except there are MANY things a project manager needs to do related to product development that have nothing to do with tasking and oversight, and of the over 200 agile organizations I’ve assessed, 100% have project managers. Some are good at it - they limit the PM's role to ensure it aligns with agile values. Some are terrible at it, and have PMs that can’t resist being dictators.

The other problem is that leaders often don’t even know what agile “looks like,” and they go about the process of “agile transformation” without any attempt on their own part to change and become agile themselves. This creates immense friction in the organization and makes “real” agile adoption (as opposed to just adopting some techniques) almost impossible. AgileCxO’s model, the “Agile Performance Holarchy,” is the defacto standard for agile leaders who are serious about this, and AgileCxO’s research has shown that leaders, not teams, or the largest impediment to agile adoption.

So, the bottom line is that Agile is hard and requires very strong leadership - although not the kind we’re used to. Try marketing that!

Like this blog? Forward to your nearest engineering or software leader!

Jeff Dalton is a Certified SCAMPI Lead Appraiser, Certified CMMI Instructor, author and consultant with years of real-world experience with the CMMI in all types of organizations.

Visit www.broadswordsolutions.com for more information about engineering strategy, performance innovation, software process improvement and running a successful CMMI program.