Essence discovery with domain experts

Valuable software solves more problems than it introduces. Given that maintaining modern software and the necessary infrastructure to support it is a serious challenge, distinguishing between solvable problems and foundational problems (essential and accidental complexity) is of the utmost importance for a software engineer. Fortunately, we are not alone in this.

Every company that uses software does so for one of two reasons: either to deliver an automated service or to make the work of its own employees more efficient. Secretly, both reasons come down to the same thing: Whether the user is internal or external to our company, we develop software to support a process.

In every process, there are people with broad knowledge of that process and people with deep knowledge of a part of it. These are domain experts, and they can help us understand how we can achieve our objectives. They can help us get to the essence of the problem and discern the most effective solution.

However, transferring that knowledge is not easy, especially since building software requires processes to be understood in a structured and concrete way. Therefore, domain experts must be helped to transfer knowledge effectively. We do this with elicitation techniques.

By applying these techniques, software engineers can discover the essence together with domain experts. I use the word ‘discover’ here for a reason: often, domain experts are just as oblivious of the essence of their work as we are, until they encounter conflicting insights when making ideas explicit and concrete. The act of making their ideas explicit helps them discover them in the first place.

In the process of knowledge elicitation, two feedback loops are applicable: a short cycle and a long cycle. The short cycle revolves around dialogue, and the long cycle revolves around documentation.

Knowledge elicitation begins with identifying sources. That is, we first look for domain experts with insights into the problem domains we are focusing on. This is followed by an elicitation phase in which we choose an elicitation technique that suits the domain expert and in which we encourage the domain expert to articulate their knowledge. The knowledge we gain and discover together must then be articulated and made explicit by us, so that the domain expert sees the conclusions and assumptions we are making.

In the short feedback (dialogue-oriented) loop, we conclude the cycle with verification, either by holding our new mental model against another domain expert or by testing our understanding against reality.

Sometimes building knowledge is not enough. Especially if the new knowledge is valuable, or susceptible to loss, documenting it is important. We then document our insights in visualizations, models, or documents, so that they are not lost.

An additional advantage of documentation is that we can perform analyses based on the documentation from which we can extract evidence for our assumptions and interpretations but also estimate the completeness of available information. This helps to search more targeted for missing knowledge.

In short, software engineering is as much about knowledge of implementation, technology, and architecture as it is about knowledge of elicitation processes, analytical skills, and making social contacts with carriers of domain knowledge. Software engineering is human work.