Brittle Pile of Assumptions

Recently, my firm has been engaged in similar conversations with several large, global enterprises about the feasibility of applying Agile techniques and methodologies in their work. I suppose this is good news, since those enterprises are finally getting serious about embracing a new way of doing business. But for an avowed Agile advocate like me, with a deep well of experience in both iterative and waterfall programs, the level of pushback still comes as a surprise. There remain deeply-entrenched constituencies that truly feel that any deviation from the waterfall structures they have utilized for many years would be disastrous for the firm — or at least their own place within it.

I’ve been a PMP for 12+ years. I recognize the history and benefits of waterfall, and continue to utilize it in those instances where it makes the most sense. However, increasingly where massive enterprise data management programs are concerned, it just doesn’t.

Waterfall is Disingenuous

Waterfall project planning, especially for large enterprise initiatives in volatile environments, is disingenuous. If represents fiction as fact by claiming it can create line-of-sight visibility from the beginning of the project to its successful conclusion. Those of us who engage in product development for a living know that, even with known techniques, materials, and technologies, solid visibility beyond six weeks becomes a dodgy proposition. Even if all the attributes of production internally are stable, the world outside continues to turn! To claim that we can accurately plan a project over time frames of six or 12 months - or even longer, — becomes silly. That 400-row Gantt chart might make everyone feel comfortable [I’ve authored many of those myself, to be sure], but its a lie to suggest it represents anything more than a best guess.

Pile of Assumptions

In truth, that holistic plan is piled with layer-upon-layer of assumptions. A tower of brittle conjecture. What other choice do we have when asked to plan 6+ months of work in advance?

We’re not talking about Production, here, where all the variables are known, control is internal, and there is a long track-record of success. We’re talking about product development, wherein by its very nature the activity involves constructing something that has never been built before (at least by our team and within our environment). Humans are wildly creative, and able to envision highly complex solutions for anticipated challenges. That’s awesome, and even better when the vision turns out to be accurate. But to be fair and transparent, we must openly admit that these constructs — both the challenges and our associated solutions — are assumptions only.

Just as important is to recognize that when we begin to build one assumption atop another, our risk increases in a non-linear fashion. Even if this specific assumption turns out to be correct, it still may shatter if one or more of the assumptions it sits upon turn out to be invalid.

Traffic Jam

Because the plan has been accepted as fact, all players optimize for efficiency and feel empowered to plow-forward with their own tasks and agendas. Like speeding into a busy intersection with the expectation of a green light.

As soon as those assumptions inevitably begin breaking down, the intersection clogs with traffic. Each driver has accepted the plan as fact, and resourced the project as efficiently as possible, so there isn’t slack available for long delays or updated expectations. Therefore, each driver leans on the horn and tries to squeeze through the jam to meet her/his own commitments.

As the Gordian knot tightens, pressure increases, tension mounts, and collaboration grinds to a halt

How many times have you lived through this project?

Agile Philosophy

“[Every time you step into the river, it is a different river.]” — Heraclitus

The same work needs to get done regardless of the chosen product development methodology, so how is Agile different?

Agile recognizes that line-of-sight visibility of the entire project is fiction. The knowledge and experience we bring will help us tremendously, but flexibility remains our greatest asset. Agile optimizes for nimbleness rather than efficiency.

Flexibility is built into the very core of Agile process, not as some ‘integrated change control’ technique grafted-on to a static plan. The entire workflow is structured around the ability to minimize risk, learn as we go, pivot priorities and generate tangible value along the way.

Make no mistake, this philosophical shift represents a massive change for many organizations, particularly those that are large and complex. “What do you mean the team gets to decide what to build on its own???” “What do you mean you can’t tell me exactly how many fields will be migrated by X date??” “What do you mean I can’t throw a bunch of additional resources at the problem and expect a linear increase in speed???” Within a waterfall perspective, all of these are questions that have been asked and answered for decades. Agile shines a bright enough light on the situation, however, to expose that not only were those answers mostly bullshit, but these aren’t even the right questions to be asking.

As a prospect stated so articulately to me just this week, “No, I get it. You’re building a capability within our organization. And then we are all going to utilize that capability as hard as we can to move through the scope. And then you are going to leave, but we will still have the capability and keep working over time.”

EXACTLY.

Clearing the Intersection

To extend the traffic jam metaphor utilized above, no substantive progress can be made while the intersection is jammed with traffic. Regardless of methodology, once that thoroughfare is impassable, nobody is going anywhere.

Sometimes, my firm isn’t brought in until the situation reaches this level of dysfunction. When we encounter a scenario like this, we try our best to help client leadership understand that while we can absolutely help with forward progress — we have to just clear the intersection, first. A few attributes of this process:

• Suspend all the deadlines against which the team is working. By this point they’re all blown anyway. Calling a halt to the deadlines helps get people to take their foot off the gas.
• Understand the dependency terrain. Who is dependent upon whom, and for what? How divergent are the agendas, constraints, and timelines of these individual groups? Is there any way to lower the number of collaborative linkages necessary to deliver the product?
• Convert “them” into “us”. Until all those jammed in the intersection come to peace with the concept that they cannot get where they are going until they make space for others to pass as well, we’re all still stuck. Calm the tempers, Generate compassion across the players, re-cast the balance of allegiance between Team and Guild.
• Refactor the plan, together. Everyone gets to be included. Regardless of which way to tough calls go, everyone feels that they had an opportunity to articulate their position. Administer these sessions so as to publicly box-in the bullies and draw sharp contrast around the rule of law.

Agile Anchors

Below, find a few anchor concepts that apply broadly across the methodology regardless of initiative size or complexity. While everything may be negotiable in Agile, these are a few solid places to hammer-in stakes.

VISIBILITY

Accept that visibility will be limited. Focus on what is just ahead, and carry it as far through the process as possible.

ANIMOSITY

Animosity is a collaboration killer. Drain animosity out the project wherever possible:

• Drain Client Animosity: The client is directly involved in the Agile project every step of the way. Evolve from “we are building it for you” to “We are building it, together”. It’s much harder to point the finger when you are part of the solution team.
• Drain Governance Animosity: Governance doesn’t get to live in an ivory tower. It toils right along with the project team, beholden to the same dates, constraints, and iterative cadence. Governance ceases being about perfection, and becomes about best feasible holistic outcomes based upon existing constraints. Governance is an iterative discipline, just like everybody else.

SCOPE

We start with the easy stuff, and carry it all the way from System of Record through to user-facing display

• Recognize that accrued knowledge makes us smarter along the way
• Devise mechanisms for lightning-fast and transparent escalation of unexpected challenges discovered enroute
• Demand brutal prioritization from stakeholders and sponsors
• Accept that there is some threshold below which things we “ought to do” simply can’t gather enough political support to become the things we “will do”. This isn’t Agile being weak. This is Agile being honest.

TEAM

Create and strengthen tribal bonds. The team succeeds or fails together.

• Focus maniacally the intra-team activity, ignoring static from outside the crucible
• Activist project and product management liaises with external partners and stakeholders in order to protect the team from all unnecessary distractions
• Avoid resource churn. Small, stable, long-lived teams. This is a self-sustaining cycle.
• Provide teams with clear and public jurisdictional ownership over a specific sub-set of the product. “We own all BI for the Corporate Finance unit” is wildly more inspirational than “we own the tickets that fall into our queue”

REFACTORING

Recognize that understanding sharpens with progress, thus some prior work will need to be refactored along the way. Leave hooks in work product to ease the burden of refactoring when it inevitably occurs. Refactoring isn’t “re-doing work that should have been done right the first time”. It’s hardening the foundation of existing products so that they can be further scaled.

ITERATIVE DELIVERY

Don’t just tell stakeholders how good it will be when complete — let them experiment with the work in process

• Solicit feedback about delivery vs. expectation, fold that into the continual planning process
• Formally engage clients in backlog prioritization
• Elevate and promote business champions who recognize the value of the deliverables. Internal marketing of victories along the way
• Test the code as soon as it is written. Deliver Production-ready products along the way

Conclusion

Given proper buy-in from senior leadership, we have had tremendous success applying Agile techniques to massive data management programs at large and complex corporate entities. But adoption of this operating mode requires a deep willingness to question the “truths” that have been espoused by waterfall for decades. Agile is not equipped to provide some of the planning diagnostics that waterfall does, and it takes a keen observer to question the value of the associated telemetry flowing out of waterfall in the first place. You might think that waterfall’s marginal track record would automatically trigger this line of inquiry from its adherents, but old habits die hard.

Regardless of which methodology you might be a fan of, we can all agree about our shared responsibility to maximize value delivered to the firm. By that line of thinking, any technique with a decent likelihood to increase value over time should be worthy of research and consideration.