Do I need a technical design?

Part 1 of a 4 part series (originally posted 20 Aug 2018)

In agile software development, there is architecture (decisions that are hard to change) and incremental design. Architecture, in this sense, is a pretty small number of things—programming language and probably application frameworks and data storage. Incremental design is the norm: we add classes, endpoints, and database tables as we identify a need for them, or remove them as they are unneeded or replaced.

But what about decisions in between these two extremes? For example, it used to be that users all signed up for the website as individuals, and now there is a need for some kind of organization which can manage the users under it. Or we used to have a bunch of separate products with their own logins, apps, and management and now there is a need to do some or all of those things in ways which apply to all products. Or our application used to assume that all users needed to be connected to the internet at all times, and now we want to build in offline operation.

I won’t completely rule out handling larger changes via the usual communication of incremental development—pair programming, discussion of individual stories, pull request review, and the like. But it can be hard to maintain a clear idea of the larger design that way, and I have usually been happier with a discussion which happens at a higher level and whose goal is to get a direction into which we can fit in the smaller decisions that we will make as we go.

I’ll write more later about who should drive this process, how to develop such a design, and what is worth writing down and communicating. But I’ll conclude this introductory post by asking when we should be doing this design.

It is tempting to say that the high level design of a system must happen before we can start breaking down the work or implementing pieces of it. Which sounds good, and is nice when it works out, but I have yet to see a design of this sort which does not get changed during implementation. There’s a lot of reality check (interactions with existing functionality, feedback which we only get when we have an early version to show, complications which we didn’t notice at first). Therefore I wouldn’t try to finalize the design before we start acting on it. And I wouldn’t go to the other extreme—of trying to make major changes in a fully incremental way and doing all the communication after the fact. My preference is to start with rough ideas and conversations about the design, and as those get refined and conversations continue, there is a point where the general contours start falling into place. That’s about when I start implementation. I want at least some of the coding to be happening (even if we know we might be revising it later), because otherwise I don’t really trust the design. In parallel, I’m stepping up the communication (documents, meetings, etc). As things fall into place (which may include allocating people’s time, agreeing on technical or business decisions, and getting a clearer picture of implementation choices), you’ll fall into the rhythm of building the thing, because the general contours of what you are building have been established by this point.