Complexity, the great enemy

Complexity is the biggest challenge in software engineering. Complex systems are difficult to maintain, difficult to comprehend, difficult to manage, and difficult to extend. Hence, a software engineer must be an expert at managing complexity. This means we must reduce complexity where possible, limit it when we cannot avoid it, and always question it when it is introduced.

All in all, it means we need to develop the wisdom to distinguish between complexity resulting from our actions and complexity arising from the problem itself.

Before we can discuss how to tackle this, we must consider what complexity entails. To do this, we must make a distinction between essential and accidental complexity.

Essential complexity

Business processes arise from creativity: a client has a certain challenge, and suppliers devise a service that addresses the challenge, preferably with as little cost and effort as possible. This is how the supplier makes itself valuable to the client.

The client’s challenge requires a solution that takes into account all facets that influence the effectiveness of the approach. This includes the client’s limitations, the unchangeable steps associated with their process, and other matters that a solution provider cannot avoid (i.e., the ‘features’ the solution must support).

In short, essential complexity is the complexity that is intrinsically part of the problem and which we therefore cannot avoid. This complexity always returns in our solutions, be they software or human actions.

The best we can do with this complexity is to ensure that we deal with it in a responsible manner, explicitly and intuitively, so that our solutions are clear about the complexity they solve.

In software engineering, this is reflected in principles like clean code, information hiding, and segregation of responsibilities.

Accidental complexity

On top of this, accidental complexity arises. This encompasses the complexity that can be avoided: the choice of specific technologies that align to a greater or lesser extent with the problem domain, the interpretations and translations to make solutions generic for multiple problems, and potential misunderstanding of the client’s challenge.

That is, accidental complexity stems from our choices for the solution, the technology, and the implementation. Since we have control over these choices (after all, we devise the solution ourselves), it is our duty to introduce as little accidental complexity as possible, even if it is not easy.

Reducing Complexity

With the difference between essential and accidental complexity in mind, there are two ways to reduce complexity.

On the accidental side, we can reduce complexity by experimenting with different technologies and validating how they align with the problem, by avoiding interpretation errors and communicating effectively with our clients, and by implementing our solutions as simply and comprehensibly as possible.

On the essential side, we can only do one thing: complexity arises from the problem, so we must make the problem we are solving as simple as possible. This means we must search for the core of the problem and avoid confounding factors.

Small reductions in software functionality can lead to large reductions in complexity. By focusing on the most valuable functionalities, we can offer the most value with the least complexity.

Precisely this view of complexity is why the ‘minimal viable product’ approach is so effective, why technical debt arises, and why refactoring code is so important. This is also why collaborative modeling is crucial: by exploring the problem together, we arrive at the essence and find the least complex solution.