ADR, or Architecture Decision Record, is a valuable tool in software development for several reasons. It provides a centralized location for documenting and tracking architectural decisions, aiding both current and future team members. ADRs enhance communication among team members by documenting the rationale behind architectural decisions, especially beneficial during onboarding of new team members or when revisiting decisions. They serve as a knowledge base, enabling teams to learn from past decisions and refine their decision-making process. Additionally, ADRs contribute to transparency by helping stakeholders understand the reasons behind specific architectural choices. As with any other tool or process, introducing them into an organization can face several obstacles, and overcoming these challenges is crucial for successful implementation. In this talk I go through some common problems and our way of solving them.
7. 1
2
3
4
5
6
7
8
9
10
~
~
~
~
~
~
~
~
~
presentation.md
master
VISUAL
# Some example ADs
- Using event-driven architecture
- Choosing a cloud provider
- Decomposing to microservices
- Replacing MariaDB with <add your favorite
NoSQL database here>
- Merging to a monorepo
- Using peer-to-peer instead of client-server
- Creating a data lake
18% 6:32
8. 1
2
3
4
5
6
7
8
9
10
~
~
~
~
~
~
~
~
~
presentation.md
master
VISUAL
# Title: "Redis as a persistent storage"
## Context: "We need a big key-value store and apparently
DynamoDB has a 400 KByte size limit while Redis has 512 MByte.
Also its blazingly fast."
## Decision: "We will use a Redis cluster to store everything."
## Status: "decided"
## Consequences:
### Good: "We don't need a caching layer anymore."
### Bad: "We have to provision a huge Redis cluster to fit
everything in memory but thats on operations team and they like
challenges anyways."
21% 7:32
17. 1
2
3
4
5
6
7
8
9
10
~
~
~
~
~
~
~
~
~
presentation.md
master
VISUAL
# McDonalds to the rescue
> It’s as if we’ve broken the ice with the worst
possible idea, and now that the discussion has
started, people suddenly get very creative. I
call it the McDonald’s Theory: people are
inspired to come up with good ideas to ward off
bad ones.
- https://jonbell.medium.com/mcdonalds-theory-9216e1c9da7d
48% 16:32
26. 1
2
3
4
5
6
7
8
9
10
~
~
~
~
~
~
~
~
~
presentation.md
master
VISUAL
# People who love writing documentation
- ADR that smells like a confluence page
- They are usually good at English
- Presenting their ADR on a review session
takes up the whole meeting
- Duplicating wiki pages as ADRs - sometimes
even linking the two together
60% 20:32
Alright, let's get started, welcome everybody and thank you for joining my session.
So today's talk will focus on the difficulties of introducing architecture decision records into an organization.
The title of this talk is kind of refurbished because I’ve used the same title when I talked about ADRs in general.
Later I figured out that this title can be used here because in this case it is not the decisions and the context around them that are being lost in the past … but our attempt to introduce ADRs to the company.
But lets see what we will cover during the presentation?
So first of all, we will talk about what is an ADR.
By the way, hands up in the air, who knows what is an ADR?
I can see quite some people, nice!
For the rest of you, no worries, as you can see we will also touch on the basics before moving on to the implementation part.
Then we are gonna talk about what this process looks like in practice and what are the typical issues when someone tries to introduce it with our ways of solving them.
And obviously at the end I hope we will have some awesome questions about the topic.
And what we won’t talk about?
We won’t talk about why having ADRs are a good thing. But let me give you a hint: it is not just a searchable UI where you can check previous decisions.
We won’t talk about good practices during the creation of ADRs. Required sections, how it should be formatted, how a good ADR looks like and so on.
Similarly we won’t talk about the anti-patterns. For example when you create an ADR but it is more like a sales pitch just to push the readers into a direction.
These are well documented and there are some great blog posts about them. I’ll link them at the end of this talk.
And most importantly, there won’t be a single process to take away.
Throughout this presentation i will talk about our problems mostly, but different companies have different problems.
But let's have a word about me:
My name is Krisztian Papp, I am from Hungary
I work as a Principal Software Engineer at Diligent where I am responsible for migrating a diverse set of products to the cloud.
I love neovim and if you do as well you might recognize the style of this slide deck.
I consider myself as an enemy of the mutable state which helped me survive Java 7.
I’m also a founder and co-host of a podcast and a developer community in Hungary.
But enough about me, let's move on to our topic.
Before jumping into what's an ADR:
Hands in the air, who used ADRs in the workplace?
Who made an attempt to introduce it in the organization?
and who failed miserably?
Let's hope this talk helps you next time!
ADR stands for architecture decision records but what is considered an architecture decision, the AD part?
Based on the relevant github page, an architecture decision is a justified software design choice that addresses a functional or non-functional requirement that is architecturally significant.
There's a debate about what is considered architecturally significant but for today's talk we are going to assume that those that are difficult to change.
Lets see some example decisions which can be considered architecturally significant.
Using event-driven architecture - its totally different than request-response way of working
Choosing a cloud provider - once you are there you are kinda vendor locked so it's hard to change
Decomposing to microservices - it sounds exciting, but it requires a lot of effort, and moves a bunch of complexity to the infrastructure
Replacing MariaDB with some NoSQL - Some might argue but even with a well layered architecture it can be painful especially migrating all the existing data
Merging to a monorepo - good luck migrating commits, rethinking processes, git flow and pipelines
Using peer-to-peer instead of client-server - well, lets hope you don’t have to make this decision in the near future
Creating a data lake - <confused> .. but how?
So an architecture decision record is an entry in a log which captures a single decision like these with the relevant context, decision drivers, considered options, links and so on.
When we talk about ADR we refer to a single entry and when we talk about ADRs usually we refer to the whole history of the records which can be called ADL as architecture decision log.
But lets see how an ADR might look like!
So first we have some title, that we are about to use redis as a persistent storage.
It is followed by some context to help us understanding this ADR.
Then we can see what is the decision
.. and what is the status of the ADR that it is decided, no way to go back.
.. but lets see some consequences of the decision:
hey, its good because we don’t need to deal with caching anymore!
on the other side, we need a huge cluster so everything can fit into memory but who cares? that's on operations team, right?
There are plenty of formats out there and this one is kinda basic as its missing other options, decision drivers and so on but as an example it should be enough.
This one is also markdown based, that's a common scenario as developers already know markdown but any other document type could work.
See that none of the details about how we plan to do that documented here?
That's on purpose because ADRs should be brief and if someone needs more context we can add link in the ADRs footer.
Okay, but how does it look like in practice?
How do these documents are born and how they end up in our log?
It depends on several factors, organizational hierarchy, the team and so on. Therefore we don’t have a single process which fits every situation.
But lets see some generic process which can help us to get a grasp on how it is like.
So the whole process starts with an architectural decision up ahead.
We have different options with pros and cons and we want to document the decision making along with the final decision somewhere.
Some of the options are better fit for our needs and some aren’t. We have to carefully investigate each of them to figure out what will be the best.
At Diligent we are storing the ADRs in git but the organizations wiki can be used for that purpose.
If you cast your vote on storing them in git like we do then there are a number of different tools out there to organize these logs.
One of them is log4brains, surprise: that's the same we are also using. Its markdown based, renders to HTML and it has nice github pages integration so you can have your architecture knowledge base website setup in no time.
We already checked the existing ADRs to see if there were similar decisions in the past but couldn’t find anything related so now everything is in the hands of the process.
So we made our decision and then we open a PR, documenting the context, other options, linking any research we made.
Once the ADR is created then it comes to the review. It can be async with review comments, or you can have a recurring or ad-hoc meeting or maybe the hybrid of both world.
We use a hybrid approach, the formal checks are done with review comments then it gets merged and since we have a nice github pages integration few minutes later it is visible with a proposed state.
Then during our recurring meeting we can go over these proposed ones and vote on them if they can be accepted/rejected or needs some work. Once voting is done, the PR gets updated and the ADR is now finalized.
Lets recap then:
first we have an idea which requires a decision
then we check if someone already made similar decision, was it accepted or rejected?
then we open a PR with a proposed state
then it comes to the formal checks on the PR
once it's done we’ll have a review - this should be proper meeting - during that review we decide about the ADR is it accepted, rejected or needs rework?
when we have the decision we update the state of the ADR and it is done
It seems pretty straightforward, isn’t it? It just a few simple steps, right?
Okay, but then why do I have this talk?
Because we are planning to introduce something new to an organization. Do you know what's easy to introduce in a company? 4-day weeks.
Just kidding, that also requires a lot of coordination.
When I talk with our developers about ADRs I usually hear that “ADRs are good!” but when it comes to participating in it.. well then people aren’t that eager to do so. And thats the root cause of a lot of problems.
Lets see some of the obstacles we faced and unfold them one by one.
First of all, you need to fight with managers to allocate some time to do this work. Teams are usually overloaded and there is not plenty of room there.
You need to fight the developers who are struggling with the process, sometimes feeling it is blocking the development.
You also need to fight the developers who hate documenting things and trust me there are a lot of them.
They might be a minority, but you’d be surprised that if someone loves writing docs… well it can also be a problem.
You need support from the top to kickstart things and at the same time you need support from the bottom to keep it going.
And last but not least every organization is different. So if you would expect a magic recipe for ADRs then I have some bad news for you.
Lets see those overloaded teams.
Writing ADRs might feel like a burden: Our engineers accepted the job to create the next AbstractAnnotationConfigDispatcherServletInitializer and not to waste their precious time writing these silly things.
Not to mention reviewing them. It similar to code reviews, who likes doing code reviews?
And sometimes they will just say, who cares? I don’t want to be involved in these decisions, I want to be close to the code itself.
Anyways, you are the architect, why don’t you do it? Its way above my paygrade.
Usually these are the people who will blame that architect when a decision is being made over their heads.
How can we solve this one?
Have you heard about the McDonalds theory? Jon Bell came up with this term.
It's about deciding where to have lunch with his co-workers and he recommends McDonalds.
His co-workers immediately say no and then out of thin air.. better alternatives emerge.
Why can’t we use something similar to tickle our teams brain?
So instead of McDonalds, let's use Kafka!
Let's say someone requires our help on deciding something and he or she approaches us:
Hey, you should figure out which database to use in our Content Service.
Let’s use Kafka!
You must be joking, right?
If you have a better idea, then why don’t you come up with an ADR?
Obviously we don’t want to fend off all the responsibilities from ourselves if we are responsible for the architecture but if we deliberately recommend something silly which could have awful consequences they must join the game. We can even spend some time documenting this silly decision somewhere.
Just make sure you we have a good poker face to do it. And also don’t forget to tell your manager about your plans or you might get into trouble.
Unfortunately, especially at the early phase of the introduction the whole process might seem like a bottleneck for the teams. Even though we are trying to empower the people to make decisions but actually they are stuck.
Let's say we try to enforce people to join the process to get some traction.
In this case we might see ADRs about stuff which can be toggled with an environment variable anytime which shouldn’t end up in this forum.
“Hey you wanted ADRs - here are you ADRs”
On the other spectrum if the created ADRs aren’t reviewed by others in time, then people might question the usefulness of the whole process and might move the bar of what is considered architecturally significant higher and higher so they can move on with their work.
This is especially true with tight deadlines when “we just don't have time for this”.
Also who is responsible to review the ADRs? What about the ones which aren’t reviewed/merged? Does those still valid?
What about the ADRs which requires some input from security/legal?
It is definitely something which blocks the adoption so we need to figure out something about it.
Managers love processes so how about we create one around ADRs which aligns with our existing ones?
We can also incorporate the creation of ADRs in certain cases which might help solving some of the problems here.
Lets say we already have a process for every major feature or change. For those we create some discovery document to gather all the details. There is even a template for this document.
How about we extend that document with an ADR link? By default every change requires an ADR but we can opt-out if our architect sign-off on that.
But this is just the first step.
If we have some example ADRs, good and bad as well then we can rule out the ones which shouldn’t end up in this forum. This is like our coding convention document but for ADRs. During the reviews we can always refer to these quickly shutting down debates on the format and drastically speeding up the review process of the ADRs.
Let's start with some low hanging fruits, like creating retrospective ADRs. These are the ones which you document after the decision is already made.
ADRs and unit tests have a lot in common. Both are better done first. But as with unit tests, a retrospective ADR is better than no ADR at all. Not to mention that this way we aren’t blocking the development until the team gets some practice on how to create them.
To speed up the reviews we should assign dedicated reviewers and keep rotating them. We can even fast track some of the ADRs if it is that urgent.
For the ADRs which left unattended in the repo, we can have some automation to close them if they are stale.
One example for this is when we had an AD about a certain requirement for one of the potential customers but at the end that deal has failed.
We should involve stakeholders early to avoid waiting on their input.
Most of our engineers hate writing documentation and I’m not judging them because there might be a couple of reasons to end up like this.
If we focus on the ADRs, then impostor syndrome might be a reason. They might also fear presenting these in front of a large crowd, scared that their ADR won’t be perfect at first.
Even though documentation is usually part of the definition of done, people still feel like its the last 20% of the task which takes up the second 80% so can’t we just skip it?
The same applies to ADRs:
- Hey, we already made our decision, why can’t we just start implementing it?
Not to mention that usually documenting something is just the first step; the real challenge lies in keeping them relevant and accurate over time.
So even if they were enthusiastic at the beginning, that's long gone because they've seen the fate of these wiki pages.
So how can we expect someone to spend time on something which becomes outdated and no one will keep it in a good shape?
So what can we do about this?
To further help with the impostor syndrome we can review their ADRs privately or even pair up with them while creating new ones.
About maintaining documentation. Luckily ADRs are immutable. Apart from being superseded by a new ADR they are not gonna change.
Also they are short so we aren’t expecting dozens of pages.
We should convince them that ADRs a platform for continuous learning by providing insights into various architectural patterns, trade-offs, and best practices, enabling them to expand their knowledge
On the other hand there are people who really love writing documentation. They aren’t necessarily good at it, but the sheer amount of words produced by them suggests that they love doing it.
If they happen to create ADRs then a single ADR will be 20 pages long and if you aren’t lucky enough, you should review it.
These people usually good at English and reading an ADR made by them will feel like reading an essay. Definitely a pleasant read but a long one.
Unfortunately the lengthier the ADR the more time consuming it to review - both synchronous and asynchronous.
Sometimes they will just copy and paste an existing wiki page and try ship it as an ADR. I’m not joking, I’ve seen three similar ADRs from the same person, each of them were about 4 page long.
But there are far worse examples out there. For example the dotnet reactive repository contains some ADR as well. This one was brought to my attention by a colleague. It even states in a section called “In brief” that this document is super long.
Guess how long this ADR is? And remember it is written in markdown so usually all these long sections are counted as a single line.
<animáció>
More than a thousand lines!
Remember our example earlier? That was about 10 lines.
Lets see what we can do to avoid these!
How about we contain or direct their creativity?
Time is money so we should keep ADRs brief to spare effort on both reviewing and writing it. It should be more like an executive summary of the decision making.
But if someone is interested in depth of the problem, for example wants to check performance tests or whatever can be found about a decision, the option is there because we can have links pointing to that 20 page document in our wiki.
To enforce this, we can even timebox these: if you can’t present your case in X minutes it means you need to rework it. We can even implement some sort of rule that if an ADR is longer than x lines then we reject it on sight.
We can also point them to an example ADR to get a sense of how detailed it should be.
But this is still about containing their creativity, how about we direct them in a direction?
If you read an ADR and it feels it was written in a hurry, because it is full of typos, that's not a good omen.
So why don’t we assign them to review other ADRs to keep them involved? They can help others by rephrasing them, fix typos early on. Hitting two birds with one stone.
But lets move on.
I have some homework for you:
- search your company wiki for the term “adr” and count the results. This way you can see how many of them failed before.
This is important because you don’t want to end up like them.
As with most initiatives you will need sponsors to start introducing something to your company.
Without that it can lose traction easily and you end up as the next item in that search result I have shown recently.
You will need buy-in because again - time is money -, and you need a pretty good case against the engineering managers who will try to focus all the efforts on shipping things.
Why is it good? Why should we spend our precious time on writing and reviewing something? Be prepared with an answer.
Also, who should be involved in the process? Everyone? Single team? Seniors or staff engineers only? The more senior engineers you need the more they will fight against it.
But ultimately people decide on how they spend their time and if you can’t convince them or their KPIs does not align with any of the benefits from the ADRs then they might as well ignore it.
If I should finish this epic and fix that bug in this quarter to get my bonus why should I spend any effort on cross-team knowledge sharing?
What can we do about it?
So my advice is to start small, in a single team. Based on shadow IT lets call this Shadow ADRs.
You should focus on finding the pain points:
The team keeps complaining on missing justification for certain technical decisions? Awesome!
People feel missing out of the decision making? Cool!
Why am I saying that? Because all these can be cured with ADRs. This will be your golden ticket against those managers we talked about in the previous slide.
At first you want to involve just a few selected people. Start with retrospective ADRs and then move on with the whole decision making process. You can showcase it to others once it started rolling.
This way it is easier to get approval from managers. They will be amazed that you convinced developers to document things in a structured way so you will get the greenlight to officially involve the team.
While it is small, you can figure out how to fit the process into the organization. When you start to extend it others won’t feel like guinea pigs in an experiment.
Once it is rolling in a single team you need to extend it to others.
But how do we go big? Convincing a single team is already challenging but multiple ones?
Lets say you announce the teams important decisions with the usage of ADRs during a town hall or any other forum.
People will start to ask questions about how to get their ideas out there? What is this tool? Where is the process documented? Luckily, you already made some progress on that front so a unified process can be used.
Once it is getting traction you can start using ADRs as a central forum.
If you already know about some previous technical decisions which should be changed then you can also announce the changes using ADRs.
Once you reached this point you can start extending to other teams in the organization.
But during extending the idea to other teams you might run into other issues.
Organization structure can be an obstacle as well. Maybe not exactly the structure itself but rather how Conway's law will take effect.
So what happens when your company has a bunch of different products from previous acquisitions? Ran by different teams and serving various personas. How to extend your process to them as well? How these gonna align with the companies architectural vision?
To further complicate things these products can run on different software stack. One might be go with react, others will be full stack javascript with angular, some might run on top of .NET core and so on.
From an average engineers standpoint it further increase the distance between them and the others.
If the teams are considered autonomous then why should they care about completely different products ADRs? Why do others care about what they do around here?
Even if those products are integrated somehow they only care about the contract between and that's about it. Who cares if others are replacing their database and especially why?
So there is an US and THEM mindset.
This will certainly introduce knowledge silos in the teams.
And you don’t even need different products to face this issue in the world of microservices. One of the selling point of this architectural style is to decentralize the decision making and providing higher ownership to the teams.
And now they have to discuss their architectural decisions with others?
What do we expect what will be their reaction? Its gonna be something like this. <next slide>
But again what can we do about this? How can we convince those teams that they are not far away islands but part of a bigger picture? That they share more than they think?
We should understand that usually there are some common problems.
It can be the infrastructure, how we deploy things, how we test or how we manage our third party libraries.
These decisions happen on an architectural level thats why it is called architectural decision.
If a containerized application is written in go or scala or javascript, what difference it gonna make on an architectural level? Usually it is going to be a similar box in a diagram. It will communicate through the same set of methods as the others with some rare exceptions. It will consume CPU and memory, same as the others, it will manage its dependencies with some package manager, same as the others, it uses I/O and so on, so why should they feel special?
I help you, they are not. But it is not enough. You need proof.
What if during these meetings they learn some new concept? They learn about how we decide on things, some typical decision drivers, trade-offs, not to mention architectural patterns.
There are a lot of information which they can benefit from and it can be considered as some learning material.
I don’t know about you all but I really love reading about tech. Usually I need to visit some external blog for that, like when I read about Amazon Prime Video was moving back from step functions and lambda to EC2 and ECS.
What if you can read things like that within your organization? What if your knowledge hub isn’t just a set of ADRs but some exciting read?
That is something you should work on. And remember those people who love writing documentation? You might want to involve them in this journey!
But first, we need to find these people…
Because you also need followers!
Why?
Because getting support from the management isn’t enough.
You need people who truly believe in ADRs.
Otherwise you’ll be the one who create most of the ADRs.
And maybe the only one who’ll review them.
If you have dedicated review meetings, you’ll be the one facilitating it.
And what happens if you go on vacation, get sick, leave or get fired?
You need others to take on these roles because the process must go on, otherwise it will be one of the many attempts to introduce it in the company.
But how can we find the right people in the company?
Let's start by hosting an internal meetup about the topic or it can be a brown bag session, lunch'n learn whatever is common in your company.
With this you might find the people who are really interested, with a bit of luck they can even be the early adopters.
It can be a self-organizing group regularly meeting, speaking about the issues, successes on their journey.
With them you can spread the idea in different teams, hear out their problems.
In summary, unfortunately there is no silver bullet here, every organization is different and the process should be tailored to it.
Introducing it is not a technical challenge but a management one.
Especially at the early phase you should make sure it is fun instead of focusing on doing it perfectly.
You can even reward the ones who participating in it if you have the power to do that .. otherwise you need sponsors, remember?
It is impossible to get the initial process right so you need to hold some retrospectives, collect ideas and iterate on the process.
It is a good idea to merge the earlier initiatives, lets leverage their previous attempts on documenting things. With this you can further grow the amount of information available through your initiative.
Introducing ADRs is not just having a searchable UI. It is the whole process, how you make decisions, how you spread the knowledge so you need to document this somewhere and make sure more and more people are aware of this.
Luckily the whole concept of ADR is trending nowadays so it's easier to find additional resources about the topic.
So there is a github page which talks about ADRs in general and it contains a lot of the relevant blog posts about the topic.
Olaf Zimmermann made some very useful articles about how to create ADRs, how to review them and so on … definitely worth checking his blog out.
And I’ve also added a link to the tool which we use for managing our ADRs as well.
