The Iterative Inchworm

11 min readDec 13, 2017

In scalable product development, slow and steady wins the race

Among the core tenets of the Agile product development methodology is the concept of Iterative delivery. Instead of building and deploying a product in one holistic run, we treat the initiative as multiple, smaller, deployable pieces.

While we equate Iterative delivery with software development, the benefits of operating in an Iterative mode extend far beyond that discipline, in some cases forming the core of an entire go-to-market philosophy. While an Iterative approach may appear plodding and incremental on paper, the reality is that slow and steady beats frenzied and dramatic — across all but the shortest lifecycles — every time.

We could fill entire books with details about how and why to execute an Agile product development strategy, and others have. But this is just a lowly blog post. Therefore, I will follow the advice of Ray Stasieczko , here, and focus first on the WHY behind a decision to do so. If you are interested in HOW to create a structure primed for Iterative delivery, keep your eyes on this blog, or see the Appendix at the bottom of this post.

One final up-front note: Iterative delivery is usually discussed as one component of a larger Agile methodology. In this post, I focus on the benefits/drawbacks around Iterative, alone, leaving the rest of Agile aside for future discussion.

Understanding Waterfall

The best way to understand Iterative delivery may be to describe the imperfect situation that led to its creation… Historically, projects were approached through what was called a “Waterfall” methodology. Like its namesake, a Waterfall project moves in one direction — from the top of the theoretical falls to the bottom — through several discrete phases. With each subsequent phase, the project advances further on its journey from concept to delivery.

By way of example, I present the core process that I utilize as the base of all the Waterfall projects that I manage. I call it the “Five Big D’s”:

DEFINE: This first phase is all about answering the core questions that undergird the initiative: Who are we? What is our underlying goal? What assets have been promised to our efforts? Under what constraints are we operating [time, cash, corporate, market, etc]? The Define phase often culminates in a Project Charter or similar document.

DESIGN: This phase establishes the detailed design for what is to be built. This doesn’t mean visual design, though that may be one component. This is the detailed solution strategy analysis that all specialists apply to determine how to bring their skills and tools to bear in order to address the opportunity defined above. For database analysts, this may consist of prototyping data structures, running some prototype migrations, and researching competitive solutions. For a charter school this might consist of modeling the cost of facility build-out against likely per-student revenue over time. For an aerospace engineer, detailed design might consist of choosing an established airfoil and testing to find a material that hits the strength/weight/cost sweet spot for this application. My father, a journeyman machinist, likes to say,

“It’s all in the prep.”

He’s exactly right, and that’s the focus of the Design phase.

DEVELOP: This is the actual build. From the outside perspective, this always looks like the hardest part. And while it often includes a fair number of challenges, my experience has been that if you hammer hard enough on the Define and Design phases, above, Development can progress in an orderly and controlled fashion. Transparency and communication are clutch here, in order to discern and address issues along the way.

DETECT: Most applications of product development include some amount of testing. Few products go to market without some vetting to confirm that they operate as expected. Different industries demand different investments around testing. When I worked in the Pharmaceutical business, I learned that testing, in the form of clinical trials, makes up the lion’s share of operational cost and activity. Because the stakes are just that high. Compare to the Digital technology business, where roll-back is accomplished with a keystroke and associated implications might be limited to a brief interruption in someone’s cat video binge. Different business constraints demand different approaches to testing.

DEPLOY: Ta-Dah! This is the big reveal. All the above work culminates in deployment of the product into its marketplace. With an internal tool or middleware component, that market consists of the colleagues and stakeholders whose work the solution benefits. With external tools, this normally means getting your product into the hands of customers. By way of example, in the newspaper business, the deployment process is complex and tightly coupled, involving folding and inserting machines, fleets of trucks, and little Jimmy’s ability to nail the toss right on your front porch while pedaling down the sidewalk at 6am.

Waterfall is a natural, logical, and organic methodology for getting something done, whether that be building an Ikea bookshelf for your boyfriend or building a consumer database for the global garment industry. Pure form Waterfall is holistic in its scope and moves relentlessly forward. Its rallying cry is,

Plan the work. Work the plan.

Waterfall Drawbacks

Unfortunately, though, the waterfall methodology has some baked-in weaknesses that undercut its utility — especially in an environment that changes quickly. Let’s take a look at those in order to understand some of the problems that Iterative delivery was designed to solve.

Design Definition Sequencing

Think for a moment about your level of expertise with the subject matter of a project on its first day — as opposed to its last. When do you understand the project best? At the end, right? With all the work required to get the project done comes a tremendous body of accrued expertise. Regardless of the process utilized for a waterfall project, its first, wobbly steps consist of detailed planning and architecture: the DESIGN phase from my sample process presented above. We need a blueprint before we can start hammering nails.

The “Cone of Uncertainty” is a model that attempts to describe this interplay of expertise across the run of a project. What that diagram makes quickly evident is that the waterfall methodology has us engaging in all of the detailed planning about the project at the worst possible time! That’s certainly not an optimal way to function. But Waterfall demands that we complete all of the planning up-front, so that associated estimates and expectations can be established, socialized, and tracked. That’s just how Waterfall works.

Accrued Risk

Every Waterfall project culminates in a formal event, at project’s end, during which the complete product would be released in a single, massive reveal. This approach certainly maximizes drama. Unfortunately, it also maximizes risk. Whether the project duration is six hours or six months, with each incremental advancement, the price tag associated with the project climbs higher, and with it the impact if the associated product doesn’t perform in the market to the level anticipated. We can control our build, but we can’t control the environment in which it operates once released to the wild. Surprises at the end are the most expensive surprises of all.

Plan Rigidity

Another problem with Waterfall is its rigidity where planning is concerned. As discussed earlier in context with the Cone of Uncertainty, Waterfall would have us perfectly understand all requirements and forge a resulting plan for execution. With all the elaborate analysis and estimation wrapped into the resulting plan, it becomes highly resistant to change. Waterfall offers techniques available for dealing with these changes, but they are clunky at best. This rigidity is a difficult pill to swallow in a business environment that evolves as quickly and dramatically as ours does, today.

Further, Waterfall product development has an “inevitability” built into it. Once a project is underway, and especially as the up-front cost accumulates, there is a default assumption that the project should continue moving forward. But not every project should. Sometimes, based on the complexity discovered during the build, or unexpected changes in the marketplace, or competing priorities within the enterprise, the right decision is to shut a project down where it stands and reallocate precious resources elsewhere. Waterfall lacks any organic review structure through which this thought process might occur. We can certainly establish one — and it’s a really good idea to do so — but out of the box Waterfall struggles in this regard. It might sound ridiculous, but I have worked on many initiatives that experienced dramatic relevance degradation over their lifecycles, yet continued trudging forward because of simple inertia. …I bet you have, as well.

Iterative Benefits

The holistic answer to these Waterfall issues is an alternative methodology, known as Agile. Reviewing all of the attributes of Agile product development is beyond the scope of this single post, but one key attribute is Agile’s adherence to the philosophy of Iterative delivery.

Iterative delivery consists of a different way of thinking about the analysis, planning, execution, and delivery of project assets over time. There’s no free lunch — all the work still needs to be done in order to bring the fully-envisioned product to market. But instead of treating the build as a monolithic body of work, Iterative construction organizes it into separate, more manageable chunks.

Let’s take a quick look at some of the specific attributes that make the difference.

Deployment early and often

Among the Iterative approach’s most powerful benefits, and steepest cost-of-entry, is its demand that we deploy our work-in-progress as early and often as possible. This is also usually the largest stumbling block for the team and leadership, alike. Professionals accustomed to deploying products in final, finished state, at the end of a complete build, can struggle to wrap their imaginations around the demands of deploying products that do not yet represent fully-formed concepts.

But the opportunities of doing so are massive. Not only can we begin learning from what the marketplace tells us about its interest in our products, but in some business models we may even begin to earn revenue from these less-than-fully-formed products in the near-term! Both of these attributes drain risk right out the bottom of our project, and provide invaluable opportunities to recalibrate our planning based on real-world feedback.

Successfully deploying incomplete products to the market is an art in-and-of itself. Utilize concepts like “Minimum Viable Product” to manage risk through this endeavor.

Plan flexibility

With an Iterative approach, the plan is almost infinitely flexible. Outside of the current sprint, our process can turn on a dime. Changes might be driven by real-world feedback [thanks to iterative deployment], some internal stimulus, or the need to track against a moving target in the market. The Iterative process happily admits that things change over time, embracing this change by organizing its entire workflow around it.

Progressive Detailed Design

Another tremendous benefit of the Iterative approach is its willingness to string-out our definition of the detailed design thought process over the life of the project. As mentioned within the context of the Cone of Uncertainty, above, it’s pretty crazy that in a Waterfall environment we believe we can accurately articulate ALL of the details of an entire project during its first, embryonic days/weeks. In attempting to do so, we are forced to pile assumptions on top of each other, creating a brittle tower of speculation. If one key assumption near the bottom of the pile turns out to be in error, the entire plan can come tumbling down. By dividing up the work into individual, manageable chunks, an Iterative approach allows the team to learn along the way, focusing energies on the attributes of the build in linear fashion. We learn as we go.

Iterative Challenges

Hooray for Iterative! But hold up just a sec. An Iterative approach brings with it a few key challenges that must be overcome in order for the team to be successful. And some are a pretty steep climb.

Refactoring

Refactoring is the process of rebuilding existing products based upon new requirements. Theoretically, this should be a rare occurrence in a Waterfall environment [the reality is somewhat less sanguine], since all analysis and planning is tackled up-front in a holistic fashion. With an Iterative approach, alternatively, design of product attributes are spread over the entire project lifecycle, and open for dramatic change. This approach inevitably leads to a situation wherein we learn information that invalidates some of the tools, techniques, or solutions previously implemented. When that happens, it’s time to pull out the tool box, crack the lug nuts, and refactor the previous work.

Developer: “If you had told me it would need to work this way two months ago when I built it, I would have architected it differently.”
Me: “Sorry, dude. That’s the price we pay for being Agile. How about I run downstairs and buy you a Starbucks while you start tearing into the code?…”

While the need to refactor can be distressing to the uninitiated, the reality is that Agile teams grow adept at building hooks into their work in order to assist with likely future refactoring. Coding to robust standards is one technique to mitigate the risks associated with refactoring in a software development environment, for example.

Embracing MVP

As articulated above, operating in an iterative fashion requires a fundamental shift in the expectations and operating mode of many enterprises. Even when the potential benefits are clearly understood on an intellectual level, discomfort around releasing incomplete products to the marketplace can be difficult or impossible to surmount. Some leadership teams just can’t get comfortable with this in their hearts and guts. In those cases, Iterative delivery might just not be the right fit.

Pushing authority down

Another philosophical stumbling block can be pushing necessary authority to non-executive resources in order to make decisions about the fate of the product. In a Waterfall environment, the executive leaders can participate in the project during the planning stages, only, and feel confident that their vision will be translated into the resulting products. When design analysis is spread across the entire project lifecycle, it becomes much less realistic that those leaders will be available to participate for the duration. And that leaves the authority to make these decisions in the hands of the team members dedicated to the project throughout its lifecycle. I can make plenty of arguments about why this is ultimately a superior approach, and recommend status/tollgate structures that help ensure senior leadership remains abreast of project progression, but many command-and-control focused organizations simply can’t get over the concept of ceding so much authority to non-executive resources.

Conclusion

I’m not here to bash Waterfall. It remains a perfect fit for lots of initiatives, and many of its weaknesses can be mitigated through the grafting of Agile techniques directly on to the Waterfall project, itself.

You could never build a nuclear submarine via an iterative approach, because the bulkheads would leak and the sailors would all glow in the dark.

For large, tightly integrated infrastructure projects like that, Waterfall might be just the ticket.

But an Iterative approach to product delivery brings with it a robust suite of benefits that actively foil the most prevalent drawbacks of traditional Waterfall. For projects and environments where it can be successful, an Iterative approach may represent a dramatic boost to enterprise product development delivery.

APPENDIX

This is a nice Agile primer: https://dzone.com/articles/short-guide-how-to-start-with-agile-project-manage

I don’t agree with everything here, but its an encyclopedic listing of Agile concepts and issues: https://glenngillen.com/thoughts/quickstart-guide-to-agile

Also really like this deck by Uri Nativ: https://www.slideshare.net/urinativ/agile-what-why-how

And, of course, the foundation of it all: https://www.scrumalliance.org/