Lies Enterprise Architects Told Me - Gwen Shapira
•Download as PPTX, PDF•
1 like•399 views
Lets face it – we are all liars. We often lie unintentionally and most of all – we lie to ourselves. I’ve spent the last 10 years working with enterprise architects intent on modernizing their data infrastructure, and I’ve heard many “facts” that turned out to be… less than perfectly accurate. Lets face it – we are all liars. We often lie unintentionally and most of all – we lie to ourselves. I’ve spent the last 10 years working with enterprise architects intent on modernizing their data infrastructure, and I’ve heard many “facts” that turned out to be… less than perfectly accurate. Self-deception about state of the industry, our requirements and our capabilities can lead us to make bad choices, which leads us to build bad architectures and often leads to bad business outcomes. If you say or hear phrases like “we have big data”, “we don’t have big data”, “this business app must be real-time” and “hybrid-cloud doesn’t exist” – you may work for an organization that can use a bit of reality check. In this talk, Gwen Shapira, principal data architect at Confluent, will share common enterprise architecture myths that did not survive contact with reality and offer some advice on how to design good data architecture given our inherent capacity for self-deception.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Lies Enterprise Architects Told Me - Gwen Shapira
Similar to Lies Enterprise Architects Told Me - Gwen Shapira (20)
More from DevOpsDays Tel Aviv
More from DevOpsDays Tel Aviv (20)
Recently uploaded
Recently uploaded (20)
Lies Enterprise Architects Told Me - Gwen Shapira
- 1. Lies Enterprise Architects Tell Gwen Shapira @gwenshap Confluent Inc.
- 2. We have a real job Get software engineers to collaborate.
- 4. You can’t force software engineers to do anything.
- 5. We lie.
- 6. Lie #1: “We are building a REAL TIME system”
- 8. Lie #2: “We have big data”
- 9. The Truth: “We have requirements”
- 10. Actual Requirements • Throughput: X MB/sec • Latency: X ms from A to B • Storage: TB • Size of working set: GB • Availability • Durability • Data access patterns
- 11. Soft Requirements • Tolerance for failure • Tolerance for “cutting edge” • Operational maturity • Size of engineering team • Culture
- 12. Buzzwords are Suspicious • Cloud Native == Resilient & Elastic • Serverless == Less ops & Pay per use • Service mesh == Flexible infra & Central control
- 13. What I say: What They Hear: What I mean: “Oh, 15 minute produce-to-consume latency and 1 MB/sec latency is easy!” “You are working on a crappy system” “We can build something cheap, stable and easy to maintain. YAY!”
- 14. KEEP IT SIMPLE
- 15. Lie #3: Hybrid / multi cloud architectures don’t exist
- 16. The Truth: My architecture is copy-pasted from a vendor.
- 17. You can’t reason an architect out of a position that she didn’t reason herself into
- 19. Have an escape plan Once the data is in Kafka, it’s so much more portable… Portable data means you don’t need to worry about vendor lock-in.” — Chris Ricommini, WePay
- 20. Lie #4: “We have microservices architecture”
- 21. The Truth: “We are trying to migrate to microservices architecture”
- 22. Architectures exist in constant change
- 23. The truth: We are in hell.
- 25. Distributed Monolith? • How many services do you need to modify when making “a tiny change”? • One of your microservices crashed. How many microservices do you need to restart? • Can you deploy a new version of any microservice at any time and at any order?
- 28. Lie #5: We use “Best of Breed”
- 29. The Truth: “We use one of everything”
- 30. Cost of software maintenance Number of technologies integrated Super scientific graph! Totally not made up to make a point!
- 31. Anti-pattern: Only architects are allowed to Have fun Learn Have an updated CV Try new technologies
- 32. Pattern: Everyone experiments. You help pick and evangelize.
- 34. Lie #6: We can’t do X here. We are too Y.
- 35. The Truth: Other Y companies did X successfully.
- 36. The Truth: I don’t want to change.
- 37. — Orson Scott Card “…we all safely interpret dangerous things in ways that don’t require us to change our lives.”
- 38. Every organization can take some risk. Do it where it matters.
- 39. Bad Architect Good Architect Uses buzzwords to justify decisions Uses data to drive decisions Forces rapid adoption of trendy choices Looks at goals, costs and benefits Copy-pastes from vendors Does own research and POC Protects “play with tech” turf Help engineers adopt technologies Says “we’ve always done it this way” Is a change agent Avoids risk / ignores risk. Takes risks that make a difference.
- 40. — Rabbi Hanina bar Hama "I have learned much from my teachers, more from my colleagues, and the most from my students"
Editor's Notes
- Hey, I’m Gwen Shapira - I’m a committer on Apache Kafka project and I work for Confluent, which is the company building a streaming event platform out of Kafka. I left my title out, because if I included it, you wouldn’t believe a single word I’m about to say. Before I start, I want to find out what percentage of the audience I’ll accidentally insult this morning. So… how many of you have the title of “enterprise architect”? Those of you working as enterprise architects probably know what you do. So it may surprise you to discover that to a large extent the rest of the world doesn’t.
- Hey, I’m Gwen Shapira - I’m a committer on Apache Kafka project and I work for Confluent, which is the company building a streaming event platform out of Kafka. I left my title out, because if I included it, you wouldn’t believe a single word I’m about to say. Apache Kafka is often used for what some people call “fast data” systems. Which means that the first thing I often hear when I meet a new customer is…
- This is not a lie. Regardless of what you may think, we have real work to do. Architects are there to force software engineers across the company to collaborate.
- Software engineers want whatever they want. And their management is incentivized to let them do it, because hiring engineers is hard and keeping them productive is harder. You want to use Go? fantastic! Want to use Airflow? Go for it. Love Spark - you do you! Love REST APIs? We’ll do REST APIs. You prefer GRPC? Lets do GRPC! But we all need to work together, and this is where it gets challenging. So enterprise architects are supposed to pick “standards” and get the entire company to use them. Standard languages, frameworks, tools, methods, architectures, design styles, governance, etc.
- Engineers have tons of leverage. So you can’t force them to collaborate. You need to convince them to do things your way. Which is why no one knows what architects do: When we do a great job, it looks like we do nothing at all. Convincing smart and opinionated people ain’t easy. You need to build trust, you need to have good arguments, you may need to have proof which means that you need to build convincing proof of concepts, you need to do research, you need to educate, you need to win debates. This is difficult. So, sometimes we take shortcuts.
- Not always intentionally. Usually the first victim is ourselves. We want something to be true, so we convince our selves it is true. And then we started repeating it to other people. Our lies are a shortcut. A way to convince ourselves and everyone else that we made a good decision, instead of doing the hard work of research, proof and consensus building. Lets look at some examples of lies that I’ve heard again and again and again, in 10 years of advising enterprise architects, in companies large and small, on how to build their data infrastructure. Lets see look at the lies and the truth behind them, because knowing the truth will free you to do a better job.
- Since I work with Apache Kafka, which is loosely connected to something called “Fast data” or “speed layer”, I get this one a lot. If you naively imagine they are building something like…
- you are wrong. I learned to ask: Which part of the system need to be how fast? Because most of the time it is “15 minutes until the dashboard reflects changes to the DB”. Except when it is “15 ms until mobile app shows that the action was accepted”. Or even “15 microseconds until we trigger a trade”.
- That lie has an older cousin. I’ve heard this non-stop between 2012 to 2016. It was usually a reason for adopting an unusual, poorly understood and poorly tested data storage system. “Why are you re-building your system to use System-Z?” “Well, we have big data, you know”. Big data in that context was around 45GB. Not that it matters.
- Buzzwords like “Real time” and “big data” are nearly meaningless. All those buzzwords are a shortcut for specific requirements. The things you actually need your system to do. We take the shortcut because collecting requirements is difficult, and we can’t be really sure we nailed them. To make things worse, storage isn’t always as agile as other components - if we make a mistake, migration is difficult. But: Picking a system based on a buzzword is worse - because the probability it is an actual fit is pretty low.
- Talk to me in requirements. What’s your throughput? What is your latency tolerance. How long do you need to store your data? What’s the size of the working set? What are your data access patterns? What’s your reporting requirements?
- I’ve seen an email from an engineer who decided to adopt a relatively new technology, and use one of the most cutting edge features in it. The email basically said “I’ve now lost all my credibility and I need to switch tracks or I’ll lose my job too”. This is heartbreaking to me.
- Like lies, they are used as shortcuts. And often lose all meaning. You need to learn to translate them. Always look for the costs and benefits.
- I learned that I need to be very gentle when I give my opinion about those requirements. I definitely can’t laugh out loud. Architects can be sensitive. I say “Well, 15 minutes is trivial” and they hear “You are a crappy architect working on a crappy system”. What I mean is “Yay! We can build a system that is cheap, easy to maintain and low risk!”
- Lots of architects act like they are allergic to simplicity. Simple is good. Collect your requirements and if they lead you to a simple system - celebrate!
- This lie has a close sibling. Now, you’d think that if “we have big data” is a lie, then “we don’t have big data” must be the truth. But you’d be wrong. Just like its sibling, this is a way of explaining a choice of data storage system made without proper consideration of requirements, capabilities, performance tests, any tests at all or even any data at all. An example would be: “Why are you using Oracle to search text documents?” “Well, we don’t need Elastic, right? We don’t have big data or anything”. If you are using Oracle to do something that Elastic is really good at, there’s high probability that you are paying few million dollars too many for your text search. Same for using Oracle as a work queue. On a side note, it is really funny how I show large prospects how they can use Kafka for their work queue and therefore save few millions of dollars and suddenly 20,000 dollar becomes “why should I pay so much for open source?”. I’ll never understand that. Anyway, what’s the truth here?
- “Big Data” and “Not Big Data” are nearly meaningless. Talk to me in requirements. What’s your throughput? What is your latency tolerance. How long do you need to store your data? What’s the size of the working set? What are your data access patterns? What’s your reporting requirements? Lets talk about non-functional requirements too: What is your appetite for risk? Do you prefer a well-understood and stable system or a presentation with “lessons learned implementing System-Z” at a conference? What’s the outcome of failure like in your organization? How many other databases do you have? What is your operational culture like? Are you comfortable contributing to open source? I’ve seen an email from an engineer who decided to adopt a relatively new technology, and use one of the most cutting edge features in it. The email basically said “I’ve now lost all my credibility and I need to switch tracks or I’ll lose my job too”. This is heartbreaking to me. Given all the functional and non-functional factors, you should be able to make a strong, well-justified choice of a database. Without a single buzzword.
- This is a particularly annoying lie, because it is so easy to disprove. I have maybe 20 counter-examples. From a very diverse set of companies. Early on, I only ever heard this from “Developer Advocates” working for a certain cloud provider. But now I also hear it from customers.
- By now, if an architect tells me: “Mult-cloud architectures are not a thing”, you of know that they are very very attached to a specific cloud provider. You can tell for sure, if the next thing they say is:
- You really can’t argue with someone who didn’t really think about the architecture in the first place. But! If you get there first, you can get them to copy-paste YOUR architecture!
- Seriously though. There are lots of benefits to using lots of cloud managed services. You just need to have an escape hatch. A plan on what to do if things don’t turn out as expected. And this is true for most architecture decisions. https://riccomini.name/kafka-escape-hatch
- I have to admit that this is a lie that I fell for many many times before I discovered the truth. Actually, there are few versions of the truth.
- Sometimes it isn’t a lie as much as it is an aspiration. They want to do microservices. They are moving in that direction. They are just not there yet. This is great.
- It includes an implicit admission in what is really a grand universal truth: “Software architectures, like anything else, only exist in a state of change”. Like buddhist mandalas, they are great works of art, built in sand, to be swept away. In few month there will be somewhere else for you to migrate to. Maybe serverless.
- We have 50,000 different microservices that we can’t keep track of. Changing a simple config requires weeks of stitching together a workflow across at least 6 different services. Troubleshooting is impossible.
- How do you know if you have a distributed monolith? You try to do one of the things that microservices are supposed to make easy. Was it easy?
- The worst offender is leak of responsibilities. Microservices are supposed to contain entire context. But you see cases where the customer profile service needs to call the insurance quotes service whenever someone changes address. Some of the logic around quotes leaked from the quotes service into the customer profile service, where it does not belong.
- Now, if we had an event driven architecture, the customer profile service would publish changes in profile and the insurance quote service can decide how to use them. Now, don’t copy-paste my architecture… but you may want to look into this.
- As I said earlier, as a software architect… you have one job. Make sure your organization is standardized on a set of technologies and design principles. “Best of breed” can be a nice way to say “I’m not doing my job”.
- There is a cost to having a technology. What makes additional technology costly? learning, deployment, monitoring, finding all the “unexpected behaviors”. If a technology does not serve a purpose or “spark joy”, say “thank you” and figure out a plan to get rid of it. Carefully assess the technical and cultural fit of new technology vs the costs of adding another technology to the stack.
- Remember that the cost of adding new software is much higher if there are lots of unique integration points. Each integration adds its own risk, so definitely try to minimize those. We used to call it “integration tax” - it ain’t bad at first, but system #20 has to integrate with the 19 older things. Unless you take steps to control it with something like Kafka as central integration point. There is a tricky anti pattern here though…
- Sometimes architects think that just because their job is to get the organization to standardize, they are the only ones who can use new technologies. This is terrible and if you work in such a place, leave. As a developer, it robs you of your growth and joy. As an architect, it robs you of your chance to make an impact… because developers are unlikely to actually do what you tell them.
- I learned this from my customers. The most successful enterprise architects know how to detect great bets and work with the engineers and managers to help the rest of the organization adopt the new methods. You do it with workgroups, hackathons, office-hours, tech-talks, etc, etc. It is an immensely satisfying way to work - you get to see many engineers grow and your decisions take root and take the organization to the next level.
- Lyft had a great talk at Qcon NYC couple of years back on how they migrated from REST to GRPC, and the main challenges were getting company-wide participation – and they wrote many tools, including a legendary proxy to help gradually increase comfort level with the new technology. And please don’t tell me “Oh, this is Lyft. We can never do it here” because…
- From “We know almost nothing and call a vendor for everything” through “We know lots, do almost everything alone and also have a deep expert on retainer” all the way to “We employ 2+ committers on the project”.
- Remember that being an architect is all about changing how you think and approach problems, changing how others work and changing entire organizations.
- This is your job. Assess capacity for risk. Small organizations can feel bolder. Large organizations sometimes prefer to play it safe, and other times want to use their resources to do very bold things. Assess what matters to the business. Find the best solutions from large teams of engineers - and with sure hand, steer the organization in the right direction at the right speed.
- This can be a very benign lie. Agile manifesto has very reasonable ideas like “value people” and “value working software”. Why would anyone lie about something so reasonable?
- There is a wonderful document called “the half-arsed agile manifesto”. https://www.halfarsedagilemanifesto.org/
- I learned a lot preparing this presentation. Thank you for the opportunity to learn and share.