Overspecification: When improvisation defies definition

Why good engineers build overly constraining systems

Software that enables people to do their work well requires a fundamental design choice. This choice precedes every technical decision: to what extent do you define how people act, and to what extent do you leave it open? Software only succeeds when it frees people to improvise where it matters and takes care of them in other areas. Therefore, build to empower people.

A practical example

In 2020, I was involved in knowledge elicitation with a sales representative. The company we worked for employed staff who carried out assignments for other companies. Different customers had different ways the assignment was paid for.

The sales representative shared: There are roughly four types of assignments. We have open-ended assignments, where we invoice the hours worked, and we also have fixed-term assignments. For the latter, it might be that a fixed end date has been agreed upon, or that there is a limit to how many hours a customer is willing to purchase. These types of assignments are often extended a few times. Finally, we have projects where the employee does not go to the customer but addressses their needs in a team within our own company. We usually charge a fixed amount per month for those collaborations.

This information led to a model consisting of four different assignment types, where the user chose the form of the assignment beforehand. Each assignment type had its own validation rules to ensure the data was protected as well as possible. We thought we were making it easier for the user to prevent errors.

What we missed: the ‘roughly’ right at the beginning of the story. The assignment payment structure took shape through negotiation with the customer. The four forms were more of a guideline than a fact.

The result was that, two years later, our sales department only used the implemented form for open-ended assignments and kept track of the agreements with the customer in a notes field, because the validation rules for the other assignment types were too restrictive. The process was actually much more flexible and fluid than we thought, and now the system we built to help was working against the needs of its user.

Standardization does not solve every problem

Many processes are stable, repeatable, and immutable. In such processes, a software system can help those involved by removing leeway. Less discipline is required if the software protects the process: Everyone works in the same way, variance is reduced, and the process becomes auditable and manageable.

In knowledge work, however, the efficiency gain is rarely found in uniformity but rather in the ability of individuals to switch gears quickly, read situations, and act based on a human measure. A call center employee who steps outside the script and solves a problem, or a team that improvises on a situation that never occurred before, is difficult from a governance perspective, but that is where an organization’s adaptive capacity lives. Few organizations recognize that this is where their value-proposition comes from.

Standardization is the organizational default answer to complexity, but it only optimizes for the visible costs of variance: unpredictability, errors, dependency on specific people or knowledge. What this approach overlooks are the costs of uniformity: loss of adaptability, growth in organizational inertia, and the necessary knowledge of workarounds where the system does not align with reality.

Two types of work require two types of systems

Snowden’s Cynefin framework offers means to determine which approach works. This conceptual model divides processes into two dimensions. The first dimension revolves around how well cause and effect can be derived from each other (determinism), and the second dimension revolves around how strongly the processes are driven by intuition or knowledge.

In processes that fall within the ordered domain (deterministic and knowledge-driven), we look at a problem and then choose an appropriate procedure from a fixed set of options. That is a ‘sense-categorize-respond’ approach. We are only dealing with known unknowns.

In (non-deterministic) complex and chaotic domains, that approach does not work. Our procedures must follow from discovery and arise from the interplay between context and the knowledge-holder, a ‘probe-sense-respond’ approach. Here we are dealing with unknown unknowns (or even unknowable unknowns), and thus, prior specification of the process is not possible. The process arises through emergence, is discovered while we study the problem, and follows organically from improvisation and creativity.

Snowden distinguishes between ‘enabling constraints’ and ‘fixed constraints’. The former create space for action without prescribing behavior. They work in complex and chaotic domains. The latter pin and fix behavior. They work in ordered domains.

Rigid software applies ‘fixed constraints’. Support for business processes in complex or chaotic domains requires ‘enabling constraints’. Building an ordered system in these domains forces a model onto a user, while that model must arise organically from their creative and innovative expertise.

Workarounds as a diagnostic tool

Karl Weick described that ad-hoc workflows serve as cognitive tools: ways people absorb uncertainty that are not formally recognized by the organization. We rationalize our work as a fixed process, even if this is not the case, to maintain the illusion of simplicity. Lucy Suchman observed that what people describe as their process and what they actually do can radically diverge. Structured procedures are essentially post-hoc reconstructions. If we eliminate ad-hoc working methods by imposing a software system, the uncertainty that was hidden by the ad hoc rationalization of an approach does not disappear.

Donella Meadows describes this as ‘policy resistance’: interventions in complex business processes produce unintended consequences because the actors in that system begin to adapt. Introducing a prescriptive system is essentially no different from enforcing a fixed procedure. The people who work with it adapt by improvising outside the system, which is seen as a risk by an organization and is combated with increasingly strict regulations. Tension escalates.

As software engineers, it is therefore our job to design software as infrastructure for human action, instead of replacing it. Not only do we then reduce the friction experienced by those involved, but we also offer the organization transparency in improvisation, thereby reducing risk.

Thorough analysis is part of the problem

The tendency toward overspecification stems from the qualities that make engineers valuable: the ability to bring structure to complex situations. They logically follow from how analytically skilled people deal with uncertainty. These qualities are a problem when applied to domains that do not allow certainty.

Rozenblit and Keil showed that people systematically overestimate how well they understand complex systems. Gary Klein shows that under time pressure, experts fall back on pattern recognition and see what they expect instead of what is there.

As engineers invest more time in a theory, the psychological attachment to it grows. We begin to see contradictory information as a variation on the structured process, while that model is a rationalization based on accidental sequentiality. The structure is the edge case: The real world is rarely as tidy as the platonic ideal.

Four ways to break the pattern

At an individual level, breaking this tendency begins with a hypothesis-driven attitude and a willingness to be surprised, as this also works for refining assumptions in specification. Donald Schön calls this the ‘reflective practitioner’ who keeps analysis going, even when implementation has already begun.

At the team level, collaborative modeling is a central instrument. Making mental models explicit and testing them together reveals the gaps and conflicts that remain invisible in an individual design. The choice of a representation determines which aspects of the mental model can even be discussed. We cannot test with others what we do not explicitly model.

At the team level, we must also question whether our models follow from a process structure driven by the desire to control, or by the desire to unburden. In the former case, we must be skeptical, as there is a high chance that we are introducing false certainty and harming human action (and thus the adaptive capacity of the organization). Cynefin helps us with this analysis.

At the organizational level, the most direct intervention is a different reading of workarounds. A workaround is a signal: the system does not allow what the situation requires. Organizations that standardize without understanding whether they are dealing with ad-hoc or ordered processes eliminate much-needed flexibility. Organizations that read workarounds as informative, as evidence of what the system does not yet support, are building this capacity.

Learning teams build learning systems

Jessica Kerr describes software teams as a learning system whose components also learn. The implementation learns from friction with reality, the people continuously learn from their product, and the team refines its analyses and collaboration. A prescriptive system breaks that learning loop. A system that allows room for improvisation keeps it going. A team that knows how to apply this not only builds a system that empowers those involved but also builds itself.

In short

Overspecification is the result of treating a process that depends on human intelligence and adaptability as mechanical and fully analyzable. The tendency to introduce structure, even where structure must dynamically emerge again and again, and a blind spot for our own preference for clarity and structure, makes engineers susceptible to this.

An organizational preference for uniformity also contributes to this problem. The questions every team must ask themselves are: Is the structure revealed by our analysis one that describes the real process, or is it an ideal that oversimplifies the human dimension to appear understandable and manageable? And, what does the friction between our model and the real world (the workarounds our users find) tell us about what we don’t understand?