Design Tradeoffs in Distributed Systems- How Southwest Airlines Uses Geode
•
2 j'aime•4,374 vues
SpringOne Platform 2016 Speaker: Brian Dunlap; Tech Lead, Southwest Airlines. Distributed systems and fast data require new software patterns and implementation skills. Learn how Southwest Airlines uses Apache Geode, organizes team responsibilities, and approaches design tradeoffs. Drawing inspiration from real whiteboard conversations, we’ll explore: -Common development pitfalls -Environment capacity planning -Streaming data patterns, like consumer checkpointing -Support roles -Production lessons learned Every day, Apache Geode improves how Southwest Airlines schedules nearly 4,000 flights and serves over 500,000 passengers. It’s an essential component of Southwest’s ability to reduce flight delays and support future growth.
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (16)
Similaire à Design Tradeoffs in Distributed Systems- How Southwest Airlines Uses Geode
Similaire à Design Tradeoffs in Distributed Systems- How Southwest Airlines Uses Geode (20)
Plus de VMware Tanzu
Plus de VMware Tanzu (20)
Dernier
Dernier (20)
Design Tradeoffs in Distributed Systems- How Southwest Airlines Uses Geode
- 1. Design Tradeoffs In Distributed Systems How Southwest Uses Apache Geode
- 2. Brian Dunlap @brianwdunlap Technical Lead - Aircraft Systems March 8th, 2016
- 4. Optimize decisions with integrated schedule information.
- 5. Grow over the next 10 years.
- 6. Show a real-time view of our operational day… to 1,000s of web users.
- 7. Scale across data centers and ensure consistency.
- 8. Support our most critical operational systems.
- 10. Southwest’s Network Operations Control integrates decision makers. BOISE ALBANY OKLAHOMA CITY AUSTIN PANAMA CITY BEACH CHARLESTON GREENVILLE-SPARTANBURG TUCSON LUBBOCK AMARILLO MIDLAND/ODESSA EL PASO LITTLE ROCK NASHVILLE DALLAS (LOVE FIELD) SACRAMENTO OAKLAND SAN JOSE BURBANK LOS ANGELES (LAX) ORANGE COUNTY ONTARIO SAN DIEGO SAN FRANCISCO (SFO) BIRMINGHAM LOUISVILLE CLEVELAND OMAHA TULSA RENO/TAHOE HARLINGEN/SOUTH PADRE ISLAND PUERTO VALLARTA CORPUS CHRISTI ALBUQUERQUE DES MOINES MEMPHIS CABO SAN LUCAS/LOS CABOS ROCHESTER AKRON/ CANTON WICHITA PENSACOLA MEXICO CITY NASSAU PUNTA CANA SAN JUAN MONTEGO BAY ARUBA CANCÚN FLINT GRAND RAPIDS CHARLOTTE DAYTON MINNEAPOLIS/ ST. PAUL PHOENIX DENVER INDIANAPOLIS COLUMBUS RALEIGH/DURHAM CHICAGO (MIDWAY) FT. LAUDERDALE (MIAMI AREA) DETROIT HOUSTON (HOBBY) SEATTLE/TACOMA LAS VEGAS NEW ORLEANS ST. LOUIS MILWAUKEE BUFFALO/ NIAGARA FALLS ATLANTA ORLANDO FT. MYERS/NAPLES JACKSONVILLE TAMPA WEST PALM BEACH SAN ANTONIO KANSAS CITY BELIZE CITY SAN JOSÉ LIBERIA PORTLAND WASHINGTON, D.C. (REAGAN NATIONAL) RICHMOND MANCHESTER PROVIDENCE HARTFORD/SPRINGFIELD NORFOLK/VIRGINIA BEACH BOSTON LOGAN PHILADELPHIA BALTIMORE/WASHINGTON (BWI) WASHINGTON, D.C. (DULLES) PITTSBURGH NEW YORK (LAGUARDIA) LONG ISLAND/ISLIP NEW YORK (NEWARK) SALT LAKE CITY SPOKANE PORTLAND
- 11. NOC
- 12. CREW PASSENGER MAINTENANCE FLIGHT GATE CARGO AIRCRAFT FACILITY OPS SUITE CONSUMES 10MJMS MESSAGES DAILY
- 13. RECOVERY OPTIMIZATION USES OVER 1,000,000 SCHEDULES 4,000 FLIGHTS 700 AIRCRAFT 500K PASSENGERS / DAY
- 14. a response time measured in seconds RECOVERY OPTIMIZATION DRIVES
- 15. Luv!
- 18. What do you own? What do you need? How long can you keep it? Domain tradeoffs
- 19. Books by: @ericevans0 @VaughnVernon Get Organized! Domain Driven Design (DDD) CORE DOMAIN SUPPORTING DOMAIN UBIQUITOUS LANGUAGE AGGREGATES DOMAIN EVENTS
- 20. What do you own? (core) <invest> What do you need? (supporting) <simplify> How long can you keep it? <intentional>
- 21. Crew Maint Pax Cargo Flight Gate Existing domain silos… OVER 15 YEARS OF
- 22. Crew Maint Pax Cargo Flight Gate 100% 100% CORE DOMAINSSUPPORTING DOMAINS INCREMENTAL STEPS SELECTED INTEGRATION
- 23. What do you own? (core) <focus> What do you need? (supporting) <simplify> How long can you keep it? <intentional>
- 24. Adding is very easy. Watch out for data that’s around for too long. Do all of these data need to be in-memory? Data at rest for a long time? (>365 days) GEODE REGION SIZES
- 25. Determine if each subdomain should use Geode. Don’t make an automatic decision. Domain tradeoffs
- 26. Maybe it needs an entirely different home? Domain tradeoffs
- 28. How far? How fast? Pattern trade-offs
- 30. OLD NEW NORMALIZED JOINS REGIONS FOR READS REGIONS FOR AGGREGATES BLOCKING THREADS ASYNC - AKKA / ACTORS ACTIVE / PASSIVE ACTIVE / ACTIVE MUTABLE STATE IMMUTABILITY / EVENT SOURCING DATA CONVERGENCE CRUD CQRS / DDD EVENT DRIVEN ServiceManagerHandlerIm pl
- 31. We’re learning!
- 32. OLD NEW NORMALIZED JOINS REGIONS FOR READS REGIONS FOR AGGREGATES BLOCKING THREADS ASYNC - AKKA / ACTORS ACTIVE / PASSIVE ACTIVE / ACTIVE MUTABLE STATE IMMUTABILITY / EVENT SOURCING DATA CONVERGENCE CRUD CQRS / DDD EVENT DRIVEN
- 33. We write immutable domain events into event regions. Client’s receive events using Geode CQs. Client’s checkpoint their position into separate regions. Event regions expire messages. checkpointing
- 34. Akka Cluster manages Actor Singletons which coordinate parallel processing based on a logical groupId. Backpressure is implemented through a competing consumer pattern. Take a look at Akka Streams! All Geode replicate regions use distributed ack. We don’t want to converge. (some write wins) coordination (*important concept)
- 35. JMS adapter Command adapters Command handlers - to CQ clients View model builders - to CQ clients JMS publishers data flow
- 36. PUSH or PULL How do we scale expensive read I/ O? Contain expensive reads With CQRS view model builders, perform heavy state enriching “select *” once. Push read updates vs. polling (Geode CQs) Conflate triggering view model rebuild events
- 37. Be careful with timeouts! Be careful with alerts! Be careful with joins! Be careful with large values! Be careful with old habits! safety tips
- 39. Distributed systems are created by distributed teams. Communication coordination is a thing.
- 40. Integrate Geode security with a directory Tune JVM size and GC Deploy and upgrade environments Size and configure VMs Support production events Enable WAN Gateway Sender / Receivers Load snapshots between environments Automate starting and stopping clusters Teaching distributed concepts - like CAP How do we share new distributed system responsibilities? DBAs UNIX DEVs Middleware Release Management Offshore Support New Geode Team DevOps EARLIER IS BETTER
- 41. Learn to luv conversation tension. When there’s tension, you’re on the right track!
- 42. opssuite-all schedule-core.jar Use separate repos to help with boundaries. Align Teams with repo ownership. Minimize jar dependencies across teams.
- 43. EMBRACE a 100X MENTALITY! What does 100x mean? (msg/sec) Normal rate: 50 Busy rate: 500 Recover rate: 5,000 HOW FAST CAN WE RECOVER?
- 44. Create great learning resources Watch out for old habits!
- 46. Prefer less-shared disk I/O. (local to a VM rack, or dedicated) Prefer larger + fewer Geode nodes. (4 larger nodes vs. 8 smaller ones) Take advantage of availability zones (A CONVERSATION LEADERSHIP ACROSS TEAMS SHARED or SHARED LESS What infrastructure supports Geode?
- 47. Know your memory (and GC) limits. Watch out for slow heap growth that triggers continuous GC. -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=60 -Xloggc:/your/path/node-name.GC.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCCause -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=20 -XX:GCLogFileSize=5M Check out GCViewer for GC log analysis.
- 48. Essential tool for real-time decision optimization testing! Helpful for QA performance and functional testing. Wonderful Geode feature! WAN Gateway
- 49. Optimization binary consumes PDX via C++ Native Client Moving > 200 MB per optimization request Be careful with refactoring PDX data types! C++ Native Client