When Greenlight had 30 engineers and a single monolith, it was possible for everyone to understand the entire context of the product. Engineers could get together to discuss a potential change and be confident in their understanding of the consequences and potential blast radius if things went wrong. Now, at around 300 engineers and 30+ microservices in production, it’s challenging for any one engineer to feel confident in either of those things. Tracking down who has knowledge of what can quickly lead to a “design by committee” failure case. How do we make sure that the decisions we are making won’t negatively affect other systems while continuing moving forward fast?
At Greenlight, we’ve implemented a framework of Decision Logs and Technical Specs that, coupled with strong principles on ownership, increases cross-team awareness of impactful decisions that are being made across the engineering organization.
In this talk we will explain how the framework works, how it was first introduced by the Infrastructure and Operations group, its benefits, and some of the challenges we found while implementing it and getting it adopted in other parts of the engineering organization.
By the end of the talk attendees will know whether incorporating some of these patterns into their organizations would help with scaling their engineering teams and how to get started, learning from our wins and our mistakes.
Move Fast and Decide Things - Datadog Dash 2022.pdf
1. Move Fast and
Decide Things
Datadog Dash 2022
Matt Farmer (he/him), Principal Engineer - Infrastructure & Operations
matt.farmer@greenlight.me
10.13
2. Agenda: Move Fast and Decide Things
● Introduction
● Agenda
● Why are we talking about this?
● Decision Logs & Technical Specs
○ Overview of format and how it answers growing pains
○ How we implemented it
○ What went well & what didn’t
● Discussion / Q&A
All stock imagery in this talk is from unsplash. Citations on final slide.
7. Decision Logs & Technical Specs
Tech Specs
Decision Logs
Decision Logs are directional. They plot a
desired end state for a process, architecture,
or technology choice and explain why.
Tech Specs are documents that describe a
solution to a specific problem. They don’t
necessarily endorse the solution for other,
related problems but serve as a collaboration
point on the particular problem.
8. Common Elements
Insert Section Subtitle Here
The color green is the foundation of our brand. It
is in our name, it is part of our brand DNA and it is
the color the brand will lead with. The color green
is the foundation of our brand. It is in our name, it
is part of our brand DNA and it is the color the
brand will lead with.
Page Properties / Metadata
Searchable metadata about each document.
Things like:
● Author
● Date
● Scope of impact
● Current status (e.g. draft, review, accepted,
rejected, replaced, etc)
Objective & Problem Statement
What is the thing we’re actually trying to solve?
How bad is it affecting us?
For technical specs: list out goals, non-goals, and
anti-goals.
9. Insert Section Subtitle Here
The color green is the foundation of our brand. It
is in our name, it is part of our brand DNA and it is
the color the brand will lead with. The color green
is the foundation of our brand. It is in our name, it
is part of our brand DNA and it is the color the
brand will lead with.
Page Properties / Metadata
Searchable metadata about each document.
Things like:
● Author
● Date
● Scope of impact
● Current status (e.g. draft, review, accepted,
rejected, replaced, etc)
Objective & Problem Statement
10. Problem Statement
We need to assemble some cereal and need to decide how we’re going to do it.
Objective
Goals
● A delicious bowl of cereal.
Non-Goals
● Coffee to go with the cereal.
Anti-Goals
● Milk spilled all over the counter.
11. Common Elements
Insert Section Subtitle Here
The color green is the foundation of our brand. It
is in our name, it is part of our brand DNA and it is
the color the brand will lead with. The color green
is the foundation of our brand. It is in our name, it
is part of our brand DNA and it is the color the
brand will lead with.
Context
What technical and business context is relevant to
the decision making process. Depending on the
scope this document this could include
macroeconomic climate or minuta about code
formatting.
Assumptions & Constraints
What are we taking to be true as a given as we go
about proposing this decision or solution?
What are some restrictions on the future impact of
this decision or solution?
12. Context
We are hungry. Cereal is a delicious breakfast food that we would like to eat.
Assumptions and Constraints
Assumptions
● You already have a clean bowl, cereal, and milk.
Constraints
● This does not handle the case that your milk has gone bad.
13. Tech Spec Elements
Insert Section
Subtitle Here
Risks
Evaluation of how this
solution could impact us
from various perspectives.
E.g. We consider:
● Data
● Scale
● Security
● And others
Insert Section
Subtitle Here
Proposed Solution
The centerpiece of the tech
spec: the proposed
solution.
Use lots of diagrams and
charts!
Insert Section
Subtitle Here
Rejected Alternatives
What did we also evaluate
and then ultimately decide
not to pursue?
14. Proposed Solution
The proposed approach is to do the following:
1. Find a bowl.
2. Put cereal in the bowl.
3. Pour milk into the bowl with cereal.
15. Risks and Considerations
Security
This spec assumes you’ve locked the doors on your house and a bear will not enter your house and steal your cereal.
Scale
This approach should work for all quantities of cereal and milk provided that you have a big enough bowl.
16. Rejected Alternatives
Not Eating Cereal
We don’t have anything else to eat before lunch and we don’t want to be hangry for our morning meetings.
Pouring Milk into the Bowl Before the Cereal
We are skeptical that we could reliably achieve the correct cereal-to-milk ratio following this approach. If our priority was to
have a ton of milk we’d just drink a glass of milk.
17. Decision Log Elements
Insert Section
Subtitle Here
Options Considered
What other things did we
consider when making this
decision, if anything.
Insert Section
Subtitle Here
Decision
What are we going to do.
The centerpiece of the
document.
Insert Section
Subtitle Here
Consequences
What has to change as a
result of doing this?
18. Our Implementation
● Heavy emphasis on lazy consensus
● Confluence as the system-of-record
● Active communication about the introduction of these formats
● Published communication standards for communicating around these things
● Utilization of document formats wasn’t a requirement, but strong
recommendation
● Lead by example: technical leadership corps starts using these heavily
19. Wins
● Good adoption within Infrastructure, Architecture, and several other engineering groups.
● Fewer surprises from the discussion of a topic to its implementation
● Meetings that could be an email were now an email (or a series of Slack messages and Confluence
comments - tomato tomahto)
● Audit trail of point-in-time documents - automatic documentation about why we’re doing what we’re
doing
● Lazy consensus keeps us moving forward
20. Misses
● Communicated a lot up front; haven’t done a great job of ensuring continuing education on the topic as
folks join the organization.
● Some groups leaned a bit too far into the philosophy - sometimes creating documents when a JIRA
ticket would have sufficed.
● No “owner” - we didn’t really designate ownership for re-evaluation of these document formats and
ensuring that they remain useful to the entire organization.
