Software projects take long to deliver code that satisfies users. Many, if not most, software engineering teams, struggle with this. It causes distrust, disappointment and frustration. So much so that the software industry even invented a name for tackling it: Expectation Management. It moderates (or sometimes restrains) the user.
A time-tested tool to manage user expectations is the categorization of requests in must-haves, should-haves, could-haves and won't-haves. But the truth is that these categories are rarely a four-way arrangement, most projects deal with only two groups: the must haves, and the won't haves. Anything that is not a must have, is a won't have. The other categories exist to soften the blow, for engineers to be able to say: 'We will do that if there is time', instead of an outright but honest 'Nope, is unlikely to happen'. Because time is a resource few projects have to spare. Most of it is spent on learning what the user really wants.
Sidebar: many management guru's will now claim that you can fix all these issues with insert-cherished-project-management-method-here. But software project management methods can only manage uncertainty. They cannot fix it. Agile-based methods do this by working in small and short iterations that are, if necessary, easily rolled back if the user seems unhappy. Waterfall-methods handle it through long pages of "Risk and Indemnity Analysis" that basically boil down to figuring out who is at fault when the user is not satisfied (and pays the bill).
The user does not expect code
Few academic scholars, who hire a software developer in a research project, know what to expect precisely. That has nothing to do with the academic scholar. I have never seen any user in any industry who knew precisely what to expect from a software system at the beginning of a project. One thing is clear though: a user never wants the source code, a bunch of configuration files or a database. What users care about is achieving an effect in the real world. They want a tool that improves handwritten text recognition, a data visualisation that helps identify patterns in a large dataset, or an endless number of other real effects. Users expect engineers to deliver something, anything, that will accomplish that. If the solution happens to require code, configuration files and databases, then that is the engineer's concern not theirs.
The user is not a software engineer
When users start to try and clarify the desired effects they quickly move into the realm of the vague, ambiguous, and mystical: 'I want something that shows me all mentions of furniture inheritance in these 17th century notary acts' versus 'in this set of computer readable texts and based on this language model of 17th century Dutch, French and Latin, we need a minimum positive recognition score of XX% of these entities in this knowledge graph, in order for us to be able to draw representative conclusions.' To translate the vague into actionable requirements takes considerable skill and time. User expectations may or may not be technically feasible or within budget. They frequently are nuanced, contradictory, or even incorrect. And they often include a lot of jargon that comes with domain and processes.
There is an information gap between users and software engineers. We cannot expect users to think like software engineers. They are not. If they were, they wouldn't need us to write code. The user is an expert in the domain and has a problem. The developer is an expert in building a solution.
Domain engineers
Software developers put a lot of effort in closing the gap by going to trainings to learn to communicate better, participate in brainstorm sessions and acquiring other soft skills. From the user's side people are trying to build basic programming skills. In short: we are doing a lot in order to understand each other better. But training, brainstorming, education and communication is not enough. We need a more structural effort from both sides. The missing link are domain engineers who form the middle section of the bridge. I am excited to see specific disciplines being established to train such engineers. There are several masters in Bio-Informatics and more and more Digital Heritage/Humanities/Publishing/Law... courses. Domain Engineers are natural bridge builders. To guarantee the delivery of quality in any software project, they should become a standard member of any project on equal standing with the domain experts, UI/UX-experts, software engineers, and data engineers.
In my next blog Code that does the Right Thing I show the reality in projects without Domain Engineers.