Generative AI and Product Knowledge Accrual
In the race to integrate Generative AI tools into our product development processes, what do we risk losing?
Poking at AI
Like everybody else in the product development space, I have been tentatively poking at the portfolio of publicly-available GenAI tools over the past several months. My perspectives have been, at turns:
- Scoffing at all the vaporware
- Impressed by the insights and data analysis capabilities
- Amused by the silliness of some of the output
- Excited about the potential opportunities for application in our work
- In awe at what seems to be just around the bend
- Concern about the risks to my own employment, and the employment of many of the other humans I know
- Genuine fear about the potential dangers of these tools to our broader culture
Basically, the same gyrations represented in every general interest article and presentation about this technology in the media, today.
Tentatively Hands-On
A prominent client is currently experimenting enthusiastically with the application of these tools against basic business administration tasks. I have been really impressed at not just the breadth of information they can gather, but also the depth of detail they can convey.
By way of example, we are now using Microsoft Copilot for transcription and analysis of all departmental meetings. The debriefing notes generated by the tool are impressive.
As a long-time PM, I have generated scores of meeting notes during my career. It is a necessary task, but low-value as the information presented is locked in the context of the meeting instead of the context of the larger project.
As such, I always end up generating the meeting notes first, and then nearly duplicating that effort as I migrate key details from the meeting into other documentation that represents the holistic project scope. Cutting that initial ‘meeting notes’ step out is a huge benefit for a person like me. And given savvy prompt engineering, I’m betting most of the current GenAI tools could handle the second task effectively, as well.
The Value of Specs
Which brings me to the core subject of this post: the generation of knowledge. Much of the work we do in product development consists to experimentation, prototyping, and analysis related to the description of an envisioned product [or enhancement] that does not yet exist. For brevity, I’ll just lump all of this under the moniker “specification documentation”, or simply, “specs”.
The leaders who envision products and fund their development have long understood that the value of specs lies in:
- Describing the envisioned product
- Establishing shared terminology, syntax, and context for these product descriptions
- Representing a model from which specialists can derive a solution strategy for construction
- Serving as a single source of truth when adjudicating whether or not the resulting product has been properly constructed
Whether you operate in a mostly Waterfall [BRD/FSD/TSD] or Agile [epics/stories] mode, the importance of these specs cannot be overstated. (Note that even when the specs are truly pristine, we will still struggle around their collision with reality while construction is underway. Because product development is really hard. SEE ALSO: https://drewharteveld.medium.com/adequately-groomed-db9a8f28bb1b )
The (Other) Value of Specs
Like a painting in the pointillist style, from across the room it looks beautiful, logical, and obvious. Those individual blobs of color mix in the atmosphere between the canvas and the viewer to create a clearly understandable and pleasing narrative. Similarly, when described at a high level, most product concepts also seem beautiful and obvious.
Unfortunately, we can’t reliably build something that has been described at only a high level. To construct successfully, we must describe the envisioned product — via specs — at the most granular level. To extend the pointillism analogy, we must drag the user from across the room until their nose is right on top of the canvas. From this vantage point, the painting is a riot of color devoid of context. That beauty and logic has been replaced with tedious minutiae — and a likely headache.
This is the hard work of product development, shifting between ‘across the room’, where the product has pleasing context, and ‘nose on the canvas’ where the dirty details must be worked out. All of which must be documented in the specs.
This is the hard work of product development, shifting between ‘across the room’, where the product has pleasing context, and ‘nose on the canvas’ where the dirty details must be worked out. All of which must be documented in the specs
It is through this hard work that the product development team, especially those members tasked with authoring the specs, gain understanding about the product. For this special case, I will leapfrog beyond the word “understanding” and utilize the term “grok,” coined from Robert Heinlein’s 1961 novel Stranger in a Strange Land. Per Merriam Webster, to “grok” something is “to understand profoundly and intuitively.”
This is truly the perfect term, because the knowledge gained by the product development team while engaged in the hard work of crawling through the documentation of specs is certainly ‘profound and intuitive’. The difference between understanding the pointillist painting from across the room and understanding it on a dot-by-dot basis is all the difference in the world.
Further, this level of expertise becomes pivotal as the product build progresses, with those who have grokked the specs not only providing on-demand reference to that content, but also interpreting incoming questions with the benefit of the full body of context that knowledge provides.
This is really how the product development process works at the ground level on a daily basis.
Grokking for the Future
Extending this narrative just one step further, this grokked knowledge pays dividends not only within individual product development builds, but across the entire product lifecycle. Those who grok the product are best positioned to hydrate that knowledge with information about enterprise strategy and the competitive marketplace in order to chart the product’s roadmap into the future [SEE ALSO: Product Managers].
If we’ve learned anything about enterprise valuation over the past 50 years of market analysis it is that the marketplace value of a company is less about the impact of products and services deployed in the past, and even those being deployed to today, than it is about the impact of the products expected to be deployed in the future. Nobody buys Apple stock today because of the invention of the iPod in 2001, and few are even focused on the current uptake of iPhone 15’s last quarter. Most are are truly interested in what whizbang gizmo Apple will introduce next year, thus further jacking share value beyond what they themselves paid today.
As we have established, the best positioning to maximize future product impact is to grok every feature included in its current state. You have to understand where your product is today in order to effectively enhance it into tomorrow.
What Are We At Risk of Losing?
Here is where we wrap the story back to GenAI. We’ve all seen the power of this technology to interpret instructions against the content of of the entire Internet — plus proprietary content that may be provided in a prompt — to generate impressive output. It is no small leap to imagine utilizing these tools to generate specs in support of product development. I’m quite sure that some companies are doing so now to the realization of dramatic efficiency gains.
My concern is that since no human is putting in the sweat equity to generate those specs, no human is earning the level of grokked knowledge that can only come through that activity. This means, potentially, that they are generating lightning fast specs today, but not building the underlying knowledge base to effectively enhance their products tomorrow. These tools might allow us to advance very quickly, but how long until we don’t understand our products adequately to know where to take them in the future?
My concern is that since no human is putting in the sweat equity to generate those specs, no human is earning the level of grokked knowledge that can only come through that activity
Doomsday Scenario
One potential answer is — I guess — “we don’t need to know”. AI knows. It did the work, it has that grokked knowledge [plus all the information on the whole Internet], so AI will tell us where best to go in the future.
And this is the implication that most scares me. Once we surrender the strategic parts of the product development process to the AI, we open ourselves to all sorts of prejudices baked into that black box that we cannot discern. By its very nature, the output of AI can’t be pragmatically validated by existing quality assurance techniques. How are you going to parallel longhand a logic cascade that takes into account all the information on the planet? Can’t do it. Once we cede this function to AI our ability to recognize intrinsically unacceptable output, and control it, will begin to erode. Perhaps swiftly.
Practical Concerns for Product Development
That’s a whole lot of doom and gloom, and well beyond the scope of risk to be found in the kinds of digital product development most of us engage in on a regular basis. And I’m no ontologist — so not the right guy to make these arguments.
For many years I have been aware of the obvious value provided by specifications, as well as the quieter yet deeper knowledge created through their generation. It’s not the kind of thing we talk about often with our clients, as they start to worry that we are treating their critical product development build as more of a science project than a commercial endeavor. But at the level of the team, we deeply understand the importance of the knowledge created, as well as the pivotal role played in the project by those who have grokked it.
A quick, final vignette: I was recently having breakfast with a colleague who runs the PMO for a mid-size technology provider. One of their PMs had recently asked to engage in some more strategic work. In close proximity, this leader receiving a request to generate a go-to-market plan for a product with an impending launch. Attempting to tie demand to the supply, my contact provided the PM with relevant inputs and asked her to take a crack at drafting the plan.
A scant two hours later, the PM delivered a passable marketing plan that smelled distinctly of GenAI. My contact thanked the PM and dropped the report into the garbage. The point of that exercise wasn’t the plan itself, but the knowledge to be gained by the PM through the act of its generation. And the PMO lead needed a PM with that knowledge fully grokked in order to successfully execute the plan in reality. The application of GenAI in this case completely missed the point.
Walking the Path
My concern is that in a world wherein no human is forced to do the hard work of research, analysis, and description to generate product development specs, no human will be grokking the knowledge about those products that comes only from walking that path.
I certainly don’t want to stand in the way of progress in the adoption of these powerful tools into our workflows. Nor do I want to miss out on the speed and efficiency to be gained by their embrace. But I also don’t want us to lose that skillset without careful consideration about the larger implications, today and into the future.
Originally published at https://www.linkedin.com.
