Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

ElasticSearch on AWS

196 vues

Publié le

ElasticSearch on AWS: Continuously Deployed, Immutable and Stateful

Publié dans : Internet
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

ElasticSearch on AWS

  1. 1. ElasticSearch on AWS Continuously Deployed, Immutable and Stateful
  2. 2. About me ● Philipp Garbe (@pgarbe) ● Husband ● Father ● AutoScout24 ● Former Microsoft-Fanboy ● Now Docker-Fanboy ;)
  3. 3. ● Use case: Logging for AS24 ● Challenge: How to handle the state? ○ Immutable ○ Continuously Deployed ○ Stateful ● Next challenges What to expect
  4. 4. Project Tatsu
  5. 5. Unified Logs CC
  6. 6. STRATEGIC GOALS Goals of the business side ARCHITECTURAL PRINCIPLES High-Level Principles DESIGN AND DELIVERY PRINCIPLES Tactical measures REDUCE TIME TO MARKET Speed, Fast Feedback COST EFFICIENCY Collect metrics to allow decisions cost vs. value. SUPPORT DATA-DRIVEN DECISIONS Listen to users and validate hypothesis. Provide as many relevant metrics & data as possible. YOU BUILT IT, YOU RUN IT The team is responsible for shaping, building, running and maintaining its products. Fast feedback from live and customers helps us to continuously improve. ORGANIZED AROUND BUSINESS CAPABILITIES Build teams around products not projects. Follow the domain and respect bounded contexts. Inverse Conway Maneuver. LOOSELY COUPLED By default avoid sharing and tight coupling, except for the big things in common. Don’t create the next monolith. MACRO AND MICRO ARCHITECTURE Clear separation. Autonomous micro services within the rules and constraints of the macro architecture. AWS FIRST Favor AWS platform service over managed service, over self-hosted OSS, over self-rolled solutions. DATA-DRIVEN / METRIC-DRIVEN Collect metrics from processes and applications. Analyze, alert and act on them. ELIMINATE ACCIDENTAL COMPLEXITY Strive to keep it simple. Focus on essential complexity. You build one, you delete one. AUTONOMOUS TEAMS Make fast local decisions. Be responsible. Know your boundaries. Share findings. INFRASTRUCTURE AS CODE Automate everything: Reproducible, traceable and tested. Immutable servers over snowflake servers. COLLABORATION CULTURE Engineers from all backgrounds work together in collaborative teams as engineers and share responsibilities. No silos. BE BOLD Go into production early. Value monitoring over tests. Recover and learn. Optimize for MTTR not MTBF. SECURITY, COMPLIANCE AND DATA PRIVACY Security must be included from the beginning and everybody’s concern. Keep data-privacy in mind. CONTAINMENT AND BOUNDARIES Align blast radius and vendor lock-in with the boundaries of the organization or business capabilities. Version 1.0 Icons made by Freepik from www.flaticon.com are licensed under CC BY 3.0 https://github.com/autoscout24/scout24-it-principles
  7. 7. INFRASTRUCTURE AS CODE Automate everything: Reproducible, traceable and tested. Immutable servers over snowflake servers.
  8. 8. Some numbers 12 nodes (m4.4xlarge, EBS gp2 á 2000 GB) 9 indices (one per day) 648 shards 7 Billion docs (new documents peak: 2,500 docs / sec) 16.99 TB of data
  9. 9. Challenge: How to handle the state?
  10. 10. Immutable Servers Rake CloudFormation ElasticSearch configuration
  11. 11. Continuously Deployed ● One CloudFormation stack per node ● Rolling updates through rake ● Every commit goes to production
  12. 12. Stateful Keep the data!
  13. 13. Monitoring ● CloudWatch ○ Cron-Lambda sends /_cluster/health metrics to CloudWatch ○ CloudWatch alarms based on these metrics ● DataDog ○ Deep Dive
  14. 14. Next challenges No AvailabilityZone awareness
  15. 15. Next challenges Mapping issues
  16. 16. Next challenges Long-running deployments (2-8h)
  17. 17. ?Questions Philipp Garbe http://garbe.io @pgarbe https://github.com/pgarbe