Distribute Design Authority with Architecture Decision Records

This experience report was originally published at the Agile 2018 Conference.

Introduction #

For over two years, the WIRE Team has recorded Architecture Decision Records (ADRs) as they worked to develop and maintain a suite of cloud-based microservices in support of the IBM Watson Discovery Service. An ADR is a lightweight text document, which describes a single design decision. To date, the WIRE Team has produced a decision log consisting of over 80 records, describing the birth, maturation, and death of microservices, and the evolution of a software system since its inception [1]. In this paper, you will hear from members of the WIRE team as they reflect on their design journey and the role ADRs played in helping them develop and maintain their part of the Watson Discovery Service.

About the WIRE Team #

This story is told primarily from the perspective of two members of the WIRE (Watson Information Retrieval, Enhanced) team, Michael and Joe, the authors of this paper. At the start of this story, Michael was serving as the team's tech lead and architect. Michael had worked professionally in software development for over 15 years and just started work on a book about software architecture design, since completed [2]. Meanwhile, Joe had just joined the team after graduating from university. Other than a few summer internships this was his first professional foray into software engineering. We'll hear from other WIRE teammates as the story unfolds.

The WIRE Team was a relatively young group of nine software developers with backgrounds in software engineering, artificial intelligence, natural language processing, and pure mathematics. The team had an average of about five years of experience, but the median experience was only two years. Some teammates had worked together previously on other software systems, but the WIRE project was a green field venture and only a small part of a significantly larger system involving over eight teams and upwards of 100 engineers. At the start of this story, the WIRE team had only been working together for a few months.

Filling a Gap: "Let's give this ADR thing a Try" #

The WIRE team's first mission was straightforward—build a set of APIs that would allow customers to train ranker models to improve search results, but communication during this period of exploration proved challenging.

Joe Runde: My first two months at IBM were spent sifting through requirements, running down people on every communication platform—slack, email, phone --- and trying to figure out what we could and couldn't build.

Michael Keeling: We had a major knowledge problem. We were moving fast and had a lot to learn, exploring new technologies and a new domain. Many architectural decisions were made in a vacuum.

Around the start of our work on what would become the Watson Discovery Service (we called it Pompipe back then) I read a tweet in Nat Pryce's Twitter stream (@natpryce) referencing this thing called "ADRs". A little investigation turned up a blog post by Michael Nygard, which described a lightweight documentation method for storing a decision log as markdown in the same repository as the code itself [3]. This immediately reminded me of an IEEE paper about recording architecture decision logs I had read as a grad student at Carnegie Mellon. ADRs seemed different; less stuffy and more accessible than the decision tables Tyree and Ackerman described [4]. I knew immediately this could help us with our knowledge sharing problems, but I was worried there might be some resistance from the team. The next day I talked it over with some people and suggested, "let's give this ADR thing a try and see how it works for us." By the end of the day I had created a template and shared our first decision record with some teammates.

Joe: I was like, "Oh man, not more documentation!" It seemed like documentation was all I was doing at the time. Here I was, staring at a page long exposition on why we were going to use a stateless microservice. Of course the service would be stateless, everybody on the team already knew that, we had been in requirements meetings for months!

Michael: In retrospect, our first ADRs were super basic. But they established the essential patterns and technology choices for our first services and how to record decisions. We stored the ADRs alongside the code, in the same Git repository as the service the decision affected. All decisions for a single component were in one place.

Joe: I was skeptical, but everybody else seemed to listen to this Michael guy, so I decided to humor him and play along for now. As long as this ADR business didn't take up all of my time I could live with a little more documentation. Two weeks later, everything began to unravel.

Eric Kaun: Our team's motto was "We eat change for breakfast!" It started as a joke but was true. Michael started using a picture of a bowl of coins with a spoon in it as our team logo.

Michael: Executive leadership asked us to change the whole technology stack. New platform, new databases, even new business goals. We had to completely pivot huge chunks of the architecture.

Joe: Within the span of a month I no longer knew which direction was up. That was when I finally realized what Michael was up to: creating a log that detailed exactly why we ended up where we were today.

Growing Pains: "Not Every Decision should go into an ADR!" #

Working alone initially, Michael captured several design decisions for a single microservice, which served as a pilot for both the technique and a template the team would soon use. He hoped to use ADRs to empower teammates to independently design and lead the construction of different microservices. In theory, decision making would be just as distributed as WIRE's microservices—but this dream proved more challenging to launch than expected.

Michael: I started asking teammates to review the ADRs I wrote by using GitHub's pull request process, the same workflow we used for code reviews. I thought this would be a good way to introduce ADRs to everyone and get more people involved with making design decisions. This also seemed like a great way to create awareness for architectural changes as they were happening.

Anastas Stoyanovsky: My sub-team was completely distributed. Day-to-day I had no idea what everyone was really doing or where they planned to go next. I liked that ADRs helped us formalize the design decisions that were already happening but not being shared.

Michael: The team was quite enthusiastic about trying ADRs. But the first pull requests I reviewed containing ADRs they wrote needed a lot of work.

Joe: I had no experience thinking through design decisions and their consequences, or how to share design decisions effectively with others. The feedback I received from pull requests encouraged me to think about the broader context and look beyond the obvious consequences of a decision.

Michael: Many of the same anti-patterns came up again and again. A single ADR would have multiple intertwined decisions bundled into a single record making it difficult to read and understand. Little thought went into the implications of a decision. I was worried we might miss risks or make accidental trade-offs. I wanted people to become more aware of the consequences their decisions had on the architecture and the broader Watson Discovery Service.

Eric: Michael and I paired with a lot of people and spent a lot of time writing comments on pull requests. Many ADRs would get stuck in a review and fix cycle.

Michael: A lot of the early ADRs were neither architectural nor design decisions. Many were sparse on analysis with only one consequence. Some were really down in the weeds. Others were more like guidelines documents. I would get pull requests for all sorts things --- general design guidelines, policies, API changes—crammed into our ADR template. It was as if the team was starved for documentation and ADRs just happened to be the first morsel they'd seen in weeks. I had to keep telling the team, "Not every decision we make should go into an ADR!"

Joe: One of my favorite "bad" ones was the "N is configurable" ADR (see Figure 3), which explains a simple implementation detail that could be changed without architectural implications.

Michael: We adjusted the template to include more guidance about what makes a design decision "architectural," emphasizing things like the influence on quality attributes and strategic technical debt. We also introduced a few other lightweight document templates for things like design guidelines and API specs.

Joe: Some people started recording an idea as an ADR without talking to anyone about it. This led to a lot of churn during reviews. Most of the churn involved brainstorming alternatives and discussing the context. We started a new rule: explore first at a whiteboard and only record a decision once a decision had actually been made.

Michael: I hoped people would take responsibility for the architecture, self-organize, make design decisions without a centralized design authority (for example, Eric or me) telling them to do it. In a way, ADRs perhaps facilitated decentralization too well.

Eric: Storing ADRs with the code worked well when the decision affected just that one component. Some architectural decisions influenced all our microservices, for example, when we decided to adopt gRPC over REST. We created a new repository for all cross-cutting documentation, including decision records about architectural governance, design guidelines, even recordings of architecture briefings, which summarized the overall design of a single microservice.

Joe: After a few months of training and practice nearly everyone could write a pretty darn good ADR. One of my favorites to write described the cost and benefits of a tech switch that was initially rejected in favor of a simpler solution. As our requirements changed this ADR was discussed again and accepted, and work began smoothly. Score one for good documentation!

Closer to a Distributed Design Authority: "Should we record an ADR for that?" #

After two years working together, the WIRE team had recorded over 80 ADRs across more than two dozen code repositories. The presence of ADRs seemed to be a leading indicator for microservices that will be easy to maintain and extend. Recording decision records became a recognized best practice on the team—and yet, only about half of the team recorded ADRs as part of their regular development practice.

Michael: Shortly after going live with our public APIs it was clear that a small number of microservices consistently accounted for over 90% of our problems day-to-day. Those two or three microservices were riddled with technical debt. Nobody liked working on them.

Joe: I was on call and there was an outage. A new feature was due at the same time. And then another feature was merged that caused our builds to run 50 minutes longer. I pulled three all-nighters to get the customers up and running and to implement everything else. I will never do that again.

Michael: Our most problematic service had only two ADRs. Other services that were the same age and experienced far fewer issues had nearly a dozen decisions logged. The architecture hadn't been thought through. It was a classic, "every system has an architecture, but nobody knows what it is" situation.

Joe: We talked with our product manager and convinced him that supporting these services was no longer sustainable. What little design existed could not support our requirements in the future, we needed to scrap and rewrite the service. It felt good to default that technical debt.

Michael: The team noticed that our worst services had no ADRs. I started hearing more people ask at the end of a whiteboard jam, "Should we record an ADR for that?" The team had greatly improved their architectural thinking. That said, it seems like only a few people think about recording ADRs without being reminded. In the worst case, I can delegate recording an ADR to someone participating in a design discussion or mention it at the daily stand-up.

Joe: When I did remind others to write ADRs, it would often spawn other design sessions that would have otherwise been missed. While working through one decision we would have trouble explaining the consequences in words, so we would pair on a lightweight view or sequence diagram. Usually this uncovered holes in the design or consequences we hadn't considered, which would lead to more decisions and ADRs.

Anastas: Sometimes it just took a little extra prodding. There was one time when Chuck (another member of the WIRE team) created a pull request with an ADR that went through a few rounds of reviews but just wasn't coming together. Michael walked up and asked Chuck, "Okay, what is really going on here?" Chuck became mildly indignant and spit out a few sentences that perfectly described the design he had chosen. Michael pointed to the keyboard and said, "That. Write that down."

Aiton Goldman: WIRE had two huge domains, training and search. Shortly after I joined the team, Michael told me about this ADR thing they were doing. I read the complete history of a few services and felt much better about this crazy ride I had signed up for.

Joe: Aiton submitted his first PR within a week of joining our team. We were already talking about past decisions and how that affected us today. He's a professional.

Michael: I'd see Joe, Chuck, Aiton and others just cranking away at a whiteboard, working on some bit of the architecture. It was great because I knew whatever they came up with I'd see a pull request with an ADR and sometimes more. It was extremely easy to keep the pulse of the team. I didn't have to stay on top of people and I didn't worry that I'd miss an important design decision.

Annie Latsko: The WIRE Team lost some people and I rotated in for a few months to help with changes to the training data preparation. There were tons of wacko assumptions going on, but Joe pointed me at the ADRs that explained them and they were pretty lit. I read some related to a component called the "Training Agent" after lunch and I was up and running, writing code by the next morning.

Mike Lipschultz: I oversaw the first major piece of new functionality on our team since our initial launch. The basic architecture involved a couple new microservices, a database, and some new frameworks from the platform. We needed to touch nearly every part of the system to deal with changes to our base assumptions. While exploring design options we noticed that some past ADRs from other microservices applied to this new chunk of the system.

Michael: We had created a small pattern catalog just for our team. That said, many microservices lacked decision logs, or there were only one or two decisions recorded to kick off the service then everyone forgot about it. But services with a decision log turned out to be tremendously useful for helping design new components and services, not only for giving people context about the past and how we arrived at where we are today.

Joe: I had no idea that part of our job as engineers was to link together the requirements, design, and code for a project. I would write out some benefits of a decision in an ADR, and Michael would respond, "Why are these benefits good to have?" I made the connection: Because they support the quality attributes we identified. I'm a better engineer for it today.

Michael: The team had grown tremendously in terms of craft, architectural skills, and design thinking. And we can prove it since we have a record of how we approached architecture design (see Figure 4). We started very top down with only me making decisions and really driving the architecture. Then we went through a period of delegation. Now we're at a point where everyone on the team has a say and feels ownership over the architecture. If you want to change the way we do something, then think it through, build some consensus, and make an ADR to let everyone know about the decision. Everyone is not only allowed to do this, but it's expected.

Expanding the Practice: "A Gateway to Better Design Thinking" #

Getting started with ADRs was relatively easy. Mastering the art of recording design decisions took some time. The WIRE team benefited greatly from ADRs, but there is always room for improvement. The future of ADRs is wide open. The more people who use this technique, the more we can uncover about how to be awesome at software design.

Michael: Writing an ADR is cheap. It takes maybe 10 minutes to write up a decision record once a decision had been made. The benefits are so huge, it just seems like a no brainer. Everyone should capture decision records for their software systems. Now that we've made it a habit, I can't imagine not recording ADRs.

Joe: We've been meaning to roll this out to the whole Discovery organization, not just the WIRE team for a while now. I see our team's old communication and design anti-patterns happening at an organization level. I'd really like to see if ADRs can help. We've been busy the past year, but we plan to find time to teach the rest of our organization how to use ADRs too.

Michael: There's been some turnover recently; we're IBM, it's a big organization and turnover happens. We haven't put this entirely to the test yet, but decision logs should help maintain design coherency over the long haul. That said, ADRs alone aren't a panacea for all architecture design ills. We've had a lot of success keeping the design aligned with the code and keeping the team aligned with the architecture by combining lightweight architecture design techniques.

Joe: I wrote a framework for one of our projects that tightly constrained the design to the decisions we had made in our ADRs [5]. Hoisting the quality requirements into the codebase using that framework helped us keep the designed architecture consistent with the actual architecture in our codebase. Architectural hoisting and ADRs go together like peanut butter and bacon.

Michael: Technical debt is always there. We could be using our ADRs or a something like it to get a better handle on strategic decision making. The consequences for a lot of our decisions imply we're taking on technical debt and even explicitly call it out, but that analysis is getting lost.

Joe: Nine months after launch, our training systems started to buckle from too much load. We knew there would be performance problems with our architecture once we hit a certain number of users. This was in the ADR but we forgot about it.

Michael: Any time we pass off a risk as "a problem we'd love to have," it should probably be captured as a consequence in an ADR somewhere.

Joe: There's this giant bank of decisions now in our Git repositories just begging for us to analyze them. We're a bunch of AI engineers, why can't we write a program to tell us when to write an ADR? Or when an ADR is good or bad?

Michael: I'm not going to say "Clippy for architecture decisions" but, sure why not? It seems like that would be super useful!

Joe: We've made many architectural decisions in the last two years that haven't been recorded. It seems like writing down these decisions after the fact would still be useful for sharing our design intentions with the team. It would be great if "Clippy for ADRs" told us when we churned a bunch of code and forgot to write a decision down.

Michael: It seems plausible that we could have an Alexa skill or Google Assistant app or even just a Slackbot to help walk people through creating a good ADR. How cool would it be if ADRs could be captured automatically during a whiteboard discussion and later pulled into an IDE's documentation just like Javadocs?

There are surely other "good practices" beyond technical debt management that could gain traction with the same lightweight treatment that decision records have received. I bet project and test planning could also benefit from lightweight, easy to maintain documentation that builds over time.

Joe: Michael and Eric were huge helps getting everyone on board with ADRs. I wonder though, if there hadn't been a "Michael" or an "Eric" on my team would we still be using ADRs? Not everybody has experienced architects on their team. Or tech leads who are willing to teach or try new things. That could be a barrier for some teams looking to adopt this technique. Can that barrier be removed?

Michael: There is a lot of interest in decision records at the moment. Why now? The idea of keeping a log of architecture design decisions is not new. Perhaps distributed version control systems like Git as well as the code and design renaissance we're in have something to do with it. ADRs seem to strike a nice balance between good documentation and lightweight methodology. They are successful because it's just markdown and it blends so well with the development workflow everyone already uses. Plus, getting started with ADRs is so simple. You just need a template, of which there are many [1] [2] [3] [4] [6], and someone willing to put up the first pull request. Even when you don't make perfect ADRs, going through the process and just practicing design is hugely beneficial.

We built microservices. ADRs were a very natural way to get more people involved in the design process. I hoped we'd communicate better with each other and I was pleasantly surprised to see how ADRs were a gateway to better design thinking throughout the team.

Joe: An important quality of a software engineer is to have strong convictions held loosely. When I first started this career I was never able to convince myself strongly enough that I was making plausible decisions. Writing ADRs taught me how to think through every implication of a decision, from the user experience to far reaching architectural details. The collaborative review and the ease of revising mistakes gives me the confidence to take on my share of design authority.

Acknowledgements #

This story was over two years in the making with contributions from many teammates. We thank all our WIRE teammates, past and present, who contributed to the architecture including Aiton, Anastas, Anish, Annie, Bill, Chuck, Eunyoung, Eric, Garrett, Kazi, Mike, Nisarga, Rishav, Ryan, Steve, Wenlong, Wentao, and Will.

Thank you to our shepherd, Rebecca Wirfs-Brock for supporting us with this experimental “oral history” style narrative for an experience report.

We thank United Airlines for their amazing loyalty program, which allowed Joe to attend the conference.

References #

1. M. Keeling and J. Runde, "Architecture Decision Records in Action," in SATURN, Denver, 2017.

2. M. Keeling, Design It! From Programmer to Software Architect, Raleigh, NC: Pragmatic Programmers, 2017.

3. M. Nygard, "Documenting Architecture Decisions," 15 November 2011. [Online]. Available: http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions. [Accessed 2 July 2018].

4. J. Tyree and A. Akerman, "Architecture Decisions: Demystifying Architecture," IEEE Software, vol. 22, no. 2, pp. 19-27, March/April 2005.

5. J. Runde, C. Gala and M. Keeling, "Architectural Hoisting: Or How I Learned to Stop Writing Breaking Code and Love the Architecture," in SATURN, Plano, 2018.

6. "Architecture Decision Record (ADR)," [Online]. Available: https://github.com/joelparkerhenderson/architecture_decision_record. [Accessed 2 July 2018].