SlideShare une entreprise Scribd logo
1  sur  36
Télécharger pour lire hors ligne
From Duke of DevOps
To Queen of Chaos
APIdays.io Paris
December 11 & 12, 2018
Christophe ROCHEFOLLE
Director Operational Experience @OUI.sncf
@crochefolle
Experienced IT executive
providing tech & organization
to improve #quality & #agility
for IT systems,
#ChaosEngineering fan
Co-author of French
DevOps book
Who am I ?
French National Railway
Company
Founded in 1938.
First e-Commerce website
in France
IT Leader in mobility,
transform your journey into
an amazing experience
Where is my playground ?
99,997%
SLA availability
OUR RECORD
39
TICKETS SOLD by SECOND
SPEED RECORD
574.8
KM/H
2008 Andrew Shafer and Patrick Debois helds a "birds of a feather" session in
'Agile Toronto'
2009 “DevOpsDays” conference started in Belgium by Patrick Debois, and term
“DevOps" coined
2009 “10 Deploys per Day at Flickr” talk by John Allspaw and Paul Hammond
in “Velocity” conference
2009 In “Velocity” conference, Andrew Clay coined "Wall of confusion"
2009 Mike Rother wrote Toyota Kata and defined 'Improvement Kata'
2010 “Continuous Delivery” book from Jez Humble and David Farley, defined
"deployment pipeline"
2011 “The Phoenix Project” book from Gene Kim and Kevin Behr
2011 Amazon deploys to production every 11.6 seconds
2014 “DevOps for Dummies” book by Sanjeev Sharma
2014 Etsy deploys more than 50 times a day
2016 “The DevOps Handbook” book by Gene Kim and Jez Humble
2016 First “DevOpsREX” conference in Paris
2018 “Mettre en oeuvre DevOps – 2nd Edition” book by Alain Sacquet and me
2008 2010 2011 2014 2016 2018
DevOps
2009
DevOps: Shorten design to cash and
quick feedback
feedback
Duke of DevOps
Time is money.
Your TTM rocks !
You have a master in
CI/CD
Queen of Chaos
But the evil
is coming !!!
TIME
TTM
MTTR
slow fast
low
high
Increasing automation
Faster release cycles
Ephemeral knowledge
Increasing complexity
The automation paradox U-curve
For the first time, availability is
the main concern for IT European
management, before security.
Source: Master of Machines III
Real life
Focus was on the left side
CI/CD
Test automation
Application Lifecycle Management
Artifact management
IaaS / PaaS / CaaS
Deployment
RIGHT
LEFT
Time for Shift-Right
We need new ways to develop
reliability concern for our teams
…(an) error budget provides a clear,
objective metric that determines how
unreliable the service is allowed to
be…
SRE Error budget
• paying off some technical debt
• improve the logging to ease support
• add some additional integration or end-to-end tests
• do those first steps to enable blue/green
deployments
• implement service mesh
But, when was the last time that your product owner
willingly added any of those technical stories to the
next sprint?
Why having Error budget ?
SRE Error
budget
Where to start ?
1. Convert unavailability to cash
2. Define Service Level Objective with business team
3. Define Error budget
Availability = successful requests / (successful request + failed requests)
A failed request can be:
1. A 500 response, due to some bug.
2. No response, due to the service being down.
3. A slow response: if the client gives up before the response is available, it is as good as no response.
4. Incorrect data, due to some bug.
Error budget = (1 - availability) = failed requests / (successful requests + failed requests)
So if a service SLO is 99.9%, it has a 0.01% error budget. If the service is serves one million
request per quarter, the error budget tells us that it can fail up to ten thousand times.
SRE Error
budget
SRE Error
budget
How to use it ?
Company agreement:
Teams may no longer make any new release
without spending time improving the reliability
of the service when error budget is 0.
In fact, they better do improvement before it.
We need new ways to know
what f$$$ happens in production
Monitoring systems have not changed significantly in 20 years and has
fallen behind the way we build software.
Our software is now large distributed systems made up of many non-
uniform interacting components while the core functionality of
monitoring systems has stagnated.
Monitoring is dead
@grepory, Monitorama 2016
Why we need observability?
Observability
Complexity is exploding everywhere,
but our tools are designed for
a predictable world.
• Can you understand what’s happening inside your
code and systems, simply by asking questions
using your tools?
• Can you answer any new question you think of, or
only the ones you prepared for?
• Having to ship new code every time you want to
ask a new question … SUCKS.
Low
Medium
High
Microservice that does one thing
Function with no side effects
Monolith with logging
Monolith with tracing and logging
Monitoring
Thresholds, alerts, watching the
health of a system by checking for a
long list of symptoms. Black box-
oriented.
Observability
What can you learn about the
running state of a program by
observing its outputs?
(Instrumentation, tracing,
debugging)
Observability
What do we want ?
a system is observable
when your team can quickly
and reliably track down any
new problem with no prior
knowledge.
Observability
Where to start ?
Observability
• Rich instrumentation
• Events, not metrics
• No aggregation
• Few dashboards
• Test in production
Internal state from software
Wrap every network call, every data call
Structured data only
Arbitrarily wide events mean you can amass more and
more context over time. Use sampling to control costs
and bandwidth.
Aggregates destroy your precious details.
We need MORE detail and MORE context.
Dashboard focus on specific known possible failure. We
need to explore raw data to discover what we don’t
know. If you already know the answer, do self-healing !
Software engineers spend too much time looking at code in
elaborately falsified environments, and not enough time
observing it in the real world.
Need more information ?
https://www.d2si.io/observabilite
Follow @mipsytipsy
engineer/cofounder/CEO
“the only good diff is a red diff”
We need shit-right testing
RIGHT
LEFT
https://medium.com/@copyconstruct/testing-in-production-the-safe-way-18ca102d0ef1
The performance of complex systems
is typically optimized at the edge of
chaos, just before system behavior
will become unrecognizably turbulent.
Chaos Engineering
—Sidney Dekker, Drift Into Failure
How much confidence we can have in the
complex systems that we put into production?
Why do we need Chaos
Engineering ?
Chaos
Engineering
With so many interacting components, the number of
things that can go wrong in a distributed system is
enormous.
You’ll never be able to prevent all possible failure modes,
but you can identify many of the weaknesses in your
system before they’re triggered by these events.
Queen of Chaos
So, to fight the evil
Chaos
Engineering
Chaos engineering
is the discipline of experimenting
on a distributed system in order
to build confidence in the systems
capacity to withstand turbulent conditions
in production
Principles of Chaos Engineering
Chaos
Engineering
2004
Chaos
engineering
2010 2012 2016 2017 2018
2004
2010
2012
2016
2017
2018
Amazon—Jesse Robbins. Master of disaster
Netflix—Greg Orzell. @chaosimia - First implementation of
Chaos Monkey to enforce use of auto-scaled stateless services
NetflixOSS open sources simian army
Gremlin Inc founded
Netflix chaos eng book. Chaos toolkit open source project
Chaos concepts getting adopted widely !
Where to start ?
Chaos
Engineering
Hypothesis testing
We think we have safety margin in this dimension, let’s
carefully test to be sure
In production
Without causing an issue
1. Start by defining ‘steady state’ as some measurable output of a system that
indicates normal behavior.
2. Hypothesize that this steady state will continue in both the control group
and the experimental group.
3. Introduce variables that reflect real world events like servers that crash,
hard drives that malfunction, network connections that are severed, etc.
4. Try to disprove the hypothesis by looking for a difference in steady state
between the control group and the experimental group.
• Simulating the failure of an entire region or datacenter.
• Partially deleting Kafka topics over a variety of instances to recreate an issue that occurred in
production.
• Injecting latency between services for a select percentage of traffic over a predetermined period
of time.
• Function-based chaos (runtime injection): randomly causing functions to throw exceptions.
• Code insertion: Adding instructions to the target program and allowing fault injection to occur
prior to certain instructions.
• Time travel: forcing system clocks out of sync with each other.
• Executing a routine in driver code emulating I/O errors.
• Maxing out CPU cores on an Elasticsearch cluster.
Injecting Chaos
Chaos
Engineering
Different experiments for
every stage
Chaos
Engineering
Infrastructure
Switching
Application
PeopleGame days
Simian Army
chaostoolkit
ChAP
Gremlin
Our story of Chaos Engineering @OUI.sncf
2015
2016 2018
Birth of an
ambition :
Chaos Monkey
EXPERIMENTATION
INDUSTRIALIZATION
All critical
applications run
Chaos experiment
2017
OUR BESTIARY IS BORN IN OCTOBER
1ST DAYS OF CHAOS
Detection : 87%
Diagnostic : 73%
Resolution : 45%
RUN IN PRODUCTION
First Chaos Monkey in
production…
…and production is
still up
2ND DAYS OF CHAOS 3RD DAYS OF CHAOS
To follow our
experiment, birth of
the
https://days-of-chaos.slack.com
Paris Chaos Engineering Meetup
http://meetu.ps/c/3BMlX/xNjMx/f https://chaosengineering.slack.com
http://days-of-chaos.com/
https://medium.com/paris-
chaos-engineering-
community
SRE Error Budget
Observability
Test in production
Chaos Engineering
Continuous Quality
CI/CD
Test automation
Application Lifecycle Management
Artifact management
IaaS / PaaS / CaaS
Deployment
Thank you
And
Bon appetite !!!

Contenu connexe

Tendances

Chaos Engineering
Chaos EngineeringChaos Engineering
Chaos EngineeringYury Roa
 
Chaos Engineering: Injecting Failure for Building Resilience in Systems
Chaos Engineering: Injecting Failure for Building Resilience in SystemsChaos Engineering: Injecting Failure for Building Resilience in Systems
Chaos Engineering: Injecting Failure for Building Resilience in SystemsYury Roa
 
DevOps for Defenders in the Enterprise
DevOps for Defenders in the EnterpriseDevOps for Defenders in the Enterprise
DevOps for Defenders in the EnterpriseJames Wickett
 
Chaos Engineering 101: A Field Guide
Chaos Engineering 101: A Field GuideChaos Engineering 101: A Field Guide
Chaos Engineering 101: A Field Guidematthewbrahms
 
Chaos Engineering when you're not Netflix
Chaos Engineering when you're not NetflixChaos Engineering when you're not Netflix
Chaos Engineering when you're not NetflixMartez Reed
 
SecOps - Bringing Agility into Security
SecOps - Bringing Agility into SecuritySecOps - Bringing Agility into Security
SecOps - Bringing Agility into SecurityAtlassian
 
Craft 2019 - Security Chaos Engineering - Security Precognition
Craft 2019 - Security Chaos Engineering - Security PrecognitionCraft 2019 - Security Chaos Engineering - Security Precognition
Craft 2019 - Security Chaos Engineering - Security PrecognitionAaron Rinehart
 
Chaos Engineering – why we should all practice breaking things on purpose by ...
Chaos Engineering – why we should all practice breaking things on purpose by ...Chaos Engineering – why we should all practice breaking things on purpose by ...
Chaos Engineering – why we should all practice breaking things on purpose by ...Alex Cachia
 
The New Ways of Chaos, Security, and DevOps
The New Ways of Chaos, Security, and DevOpsThe New Ways of Chaos, Security, and DevOps
The New Ways of Chaos, Security, and DevOpsJames Wickett
 
GDS-Austin - DevSecOps & Security Chaos Engineering
GDS-Austin - DevSecOps & Security Chaos EngineeringGDS-Austin - DevSecOps & Security Chaos Engineering
GDS-Austin - DevSecOps & Security Chaos EngineeringAaron Rinehart
 
ChaoSlingr: Introducing Security based Chaos Testing
ChaoSlingr: Introducing Security based Chaos TestingChaoSlingr: Introducing Security based Chaos Testing
ChaoSlingr: Introducing Security based Chaos TestingAaron Rinehart
 
The Seven Habits of the Highly Effective DevSecOp
The Seven Habits of the Highly Effective DevSecOpThe Seven Habits of the Highly Effective DevSecOp
The Seven Habits of the Highly Effective DevSecOpJames Wickett
 
An Introduction to Chaos Engineering
An Introduction to Chaos EngineeringAn Introduction to Chaos Engineering
An Introduction to Chaos EngineeringGremlin
 
OWASP AppSec Global 2019 Security & Chaos Engineering
OWASP AppSec Global 2019 Security & Chaos EngineeringOWASP AppSec Global 2019 Security & Chaos Engineering
OWASP AppSec Global 2019 Security & Chaos EngineeringAaron Rinehart
 
DevSecOps and the New Path Forward
DevSecOps and the New Path ForwardDevSecOps and the New Path Forward
DevSecOps and the New Path ForwardJames Wickett
 
Adversary Driven Defense in the Real World
Adversary Driven Defense in the Real WorldAdversary Driven Defense in the Real World
Adversary Driven Defense in the Real WorldJames Wickett
 
DevOps - Understanding Core Concepts (Old)
DevOps - Understanding Core Concepts (Old)DevOps - Understanding Core Concepts (Old)
DevOps - Understanding Core Concepts (Old)Nitin Bhide
 
Making Observability Actionable At Scale - DBS DevConnect 2019
Making Observability Actionable At Scale - DBS DevConnect 2019Making Observability Actionable At Scale - DBS DevConnect 2019
Making Observability Actionable At Scale - DBS DevConnect 2019Squadcast Inc
 
DevOps for the Discouraged
DevOps for the Discouraged DevOps for the Discouraged
DevOps for the Discouraged James Wickett
 
Road to DevOps ROI
Road to DevOps ROIRoad to DevOps ROI
Road to DevOps ROICloudmunch
 

Tendances (20)

Chaos Engineering
Chaos EngineeringChaos Engineering
Chaos Engineering
 
Chaos Engineering: Injecting Failure for Building Resilience in Systems
Chaos Engineering: Injecting Failure for Building Resilience in SystemsChaos Engineering: Injecting Failure for Building Resilience in Systems
Chaos Engineering: Injecting Failure for Building Resilience in Systems
 
DevOps for Defenders in the Enterprise
DevOps for Defenders in the EnterpriseDevOps for Defenders in the Enterprise
DevOps for Defenders in the Enterprise
 
Chaos Engineering 101: A Field Guide
Chaos Engineering 101: A Field GuideChaos Engineering 101: A Field Guide
Chaos Engineering 101: A Field Guide
 
Chaos Engineering when you're not Netflix
Chaos Engineering when you're not NetflixChaos Engineering when you're not Netflix
Chaos Engineering when you're not Netflix
 
SecOps - Bringing Agility into Security
SecOps - Bringing Agility into SecuritySecOps - Bringing Agility into Security
SecOps - Bringing Agility into Security
 
Craft 2019 - Security Chaos Engineering - Security Precognition
Craft 2019 - Security Chaos Engineering - Security PrecognitionCraft 2019 - Security Chaos Engineering - Security Precognition
Craft 2019 - Security Chaos Engineering - Security Precognition
 
Chaos Engineering – why we should all practice breaking things on purpose by ...
Chaos Engineering – why we should all practice breaking things on purpose by ...Chaos Engineering – why we should all practice breaking things on purpose by ...
Chaos Engineering – why we should all practice breaking things on purpose by ...
 
The New Ways of Chaos, Security, and DevOps
The New Ways of Chaos, Security, and DevOpsThe New Ways of Chaos, Security, and DevOps
The New Ways of Chaos, Security, and DevOps
 
GDS-Austin - DevSecOps & Security Chaos Engineering
GDS-Austin - DevSecOps & Security Chaos EngineeringGDS-Austin - DevSecOps & Security Chaos Engineering
GDS-Austin - DevSecOps & Security Chaos Engineering
 
ChaoSlingr: Introducing Security based Chaos Testing
ChaoSlingr: Introducing Security based Chaos TestingChaoSlingr: Introducing Security based Chaos Testing
ChaoSlingr: Introducing Security based Chaos Testing
 
The Seven Habits of the Highly Effective DevSecOp
The Seven Habits of the Highly Effective DevSecOpThe Seven Habits of the Highly Effective DevSecOp
The Seven Habits of the Highly Effective DevSecOp
 
An Introduction to Chaos Engineering
An Introduction to Chaos EngineeringAn Introduction to Chaos Engineering
An Introduction to Chaos Engineering
 
OWASP AppSec Global 2019 Security & Chaos Engineering
OWASP AppSec Global 2019 Security & Chaos EngineeringOWASP AppSec Global 2019 Security & Chaos Engineering
OWASP AppSec Global 2019 Security & Chaos Engineering
 
DevSecOps and the New Path Forward
DevSecOps and the New Path ForwardDevSecOps and the New Path Forward
DevSecOps and the New Path Forward
 
Adversary Driven Defense in the Real World
Adversary Driven Defense in the Real WorldAdversary Driven Defense in the Real World
Adversary Driven Defense in the Real World
 
DevOps - Understanding Core Concepts (Old)
DevOps - Understanding Core Concepts (Old)DevOps - Understanding Core Concepts (Old)
DevOps - Understanding Core Concepts (Old)
 
Making Observability Actionable At Scale - DBS DevConnect 2019
Making Observability Actionable At Scale - DBS DevConnect 2019Making Observability Actionable At Scale - DBS DevConnect 2019
Making Observability Actionable At Scale - DBS DevConnect 2019
 
DevOps for the Discouraged
DevOps for the Discouraged DevOps for the Discouraged
DevOps for the Discouraged
 
Road to DevOps ROI
Road to DevOps ROIRoad to DevOps ROI
Road to DevOps ROI
 

Similaire à From Duke of DevOps to Queen of Chaos - Api days 2018

An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)Brian Brazil
 
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as CodeConfoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as CodeSteve Mercier
 
2014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.0
2014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.02014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.0
2014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.0Joakim Lindbom
 
A Tale of Contemporary Software
A Tale of Contemporary SoftwareA Tale of Contemporary Software
A Tale of Contemporary SoftwareYun Zhi Lin
 
Embracing Failure - AzureDay Rome
Embracing Failure - AzureDay RomeEmbracing Failure - AzureDay Rome
Embracing Failure - AzureDay RomeAlberto Acerbis
 
From Monoliths to Microservices at Realestate.com.au
From Monoliths to Microservices at Realestate.com.auFrom Monoliths to Microservices at Realestate.com.au
From Monoliths to Microservices at Realestate.com.auevanbottcher
 
Prometheus - Open Source Forum Japan
Prometheus  - Open Source Forum JapanPrometheus  - Open Source Forum Japan
Prometheus - Open Source Forum JapanBrian Brazil
 
Evolving to Cloud-Native - Nate Schutta (2/2)
Evolving to Cloud-Native - Nate Schutta (2/2)Evolving to Cloud-Native - Nate Schutta (2/2)
Evolving to Cloud-Native - Nate Schutta (2/2)VMware Tanzu
 
Dev Ops for systems of record - Talk at Agile Australia 2015
Dev Ops for systems of record - Talk at Agile Australia 2015Dev Ops for systems of record - Talk at Agile Australia 2015
Dev Ops for systems of record - Talk at Agile Australia 2015Mirco Hering
 
Moving to Microservices with the Help of Distributed Traces
Moving to Microservices with the Help of Distributed TracesMoving to Microservices with the Help of Distributed Traces
Moving to Microservices with the Help of Distributed TracesKP Kaiser
 
Accelerate your Application Delivery with DevOps and Microservices
Accelerate your Application Delivery with DevOps and MicroservicesAccelerate your Application Delivery with DevOps and Microservices
Accelerate your Application Delivery with DevOps and MicroservicesAmazon Web Services
 
2016 - 10 questions you should answer before building a new microservice
2016 - 10 questions you should answer before building a new microservice2016 - 10 questions you should answer before building a new microservice
2016 - 10 questions you should answer before building a new microservicedevopsdaysaustin
 
Workshop - The Little Pattern That Could.pdf
Workshop - The Little Pattern That Could.pdfWorkshop - The Little Pattern That Could.pdf
Workshop - The Little Pattern That Could.pdfTobiasGoeschel
 
Scaling Engineering with Docker
Scaling Engineering with DockerScaling Engineering with Docker
Scaling Engineering with DockerTom Leach
 
Monitoring As Code: How to Integrate App Monitoring Into Your Developer Cycle
Monitoring As Code: How to Integrate App Monitoring Into Your Developer CycleMonitoring As Code: How to Integrate App Monitoring Into Your Developer Cycle
Monitoring As Code: How to Integrate App Monitoring Into Your Developer CycleAtlassian
 
Code instrumentation
Code instrumentationCode instrumentation
Code instrumentationMennan Tekbir
 
Innovate Better Through Machine data Analytics
Innovate Better Through Machine data AnalyticsInnovate Better Through Machine data Analytics
Innovate Better Through Machine data AnalyticsHal Rottenberg
 
Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...
Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...
Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...Curiosity Software Ireland
 
The DevOps paradigm - the evolution of IT professionals and opensource toolkit
The DevOps paradigm - the evolution of IT professionals and opensource toolkitThe DevOps paradigm - the evolution of IT professionals and opensource toolkit
The DevOps paradigm - the evolution of IT professionals and opensource toolkitMarco Ferrigno
 

Similaire à From Duke of DevOps to Queen of Chaos - Api days 2018 (20)

An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)An Introduction to Prometheus (GrafanaCon 2016)
An Introduction to Prometheus (GrafanaCon 2016)
 
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as CodeConfoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
Confoo-Montreal-2016: Controlling Your Environments using Infrastructure as Code
 
2014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.0
2014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.02014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.0
2014-10 DevOps NFi - Why it's a good idea to deploy 10 times per day v1.0
 
A Tale of Contemporary Software
A Tale of Contemporary SoftwareA Tale of Contemporary Software
A Tale of Contemporary Software
 
Embracing Failure - AzureDay Rome
Embracing Failure - AzureDay RomeEmbracing Failure - AzureDay Rome
Embracing Failure - AzureDay Rome
 
From Monoliths to Microservices at Realestate.com.au
From Monoliths to Microservices at Realestate.com.auFrom Monoliths to Microservices at Realestate.com.au
From Monoliths to Microservices at Realestate.com.au
 
Prometheus - Open Source Forum Japan
Prometheus  - Open Source Forum JapanPrometheus  - Open Source Forum Japan
Prometheus - Open Source Forum Japan
 
Evolving to Cloud-Native - Nate Schutta (2/2)
Evolving to Cloud-Native - Nate Schutta (2/2)Evolving to Cloud-Native - Nate Schutta (2/2)
Evolving to Cloud-Native - Nate Schutta (2/2)
 
Dev Ops for systems of record - Talk at Agile Australia 2015
Dev Ops for systems of record - Talk at Agile Australia 2015Dev Ops for systems of record - Talk at Agile Australia 2015
Dev Ops for systems of record - Talk at Agile Australia 2015
 
Moving to Microservices with the Help of Distributed Traces
Moving to Microservices with the Help of Distributed TracesMoving to Microservices with the Help of Distributed Traces
Moving to Microservices with the Help of Distributed Traces
 
Accelerate your Application Delivery with DevOps and Microservices
Accelerate your Application Delivery with DevOps and MicroservicesAccelerate your Application Delivery with DevOps and Microservices
Accelerate your Application Delivery with DevOps and Microservices
 
2016 - 10 questions you should answer before building a new microservice
2016 - 10 questions you should answer before building a new microservice2016 - 10 questions you should answer before building a new microservice
2016 - 10 questions you should answer before building a new microservice
 
Workshop - The Little Pattern That Could.pdf
Workshop - The Little Pattern That Could.pdfWorkshop - The Little Pattern That Could.pdf
Workshop - The Little Pattern That Could.pdf
 
Model-Based Testing for Cypress
Model-Based Testing for CypressModel-Based Testing for Cypress
Model-Based Testing for Cypress
 
Scaling Engineering with Docker
Scaling Engineering with DockerScaling Engineering with Docker
Scaling Engineering with Docker
 
Monitoring As Code: How to Integrate App Monitoring Into Your Developer Cycle
Monitoring As Code: How to Integrate App Monitoring Into Your Developer CycleMonitoring As Code: How to Integrate App Monitoring Into Your Developer Cycle
Monitoring As Code: How to Integrate App Monitoring Into Your Developer Cycle
 
Code instrumentation
Code instrumentationCode instrumentation
Code instrumentation
 
Innovate Better Through Machine data Analytics
Innovate Better Through Machine data AnalyticsInnovate Better Through Machine data Analytics
Innovate Better Through Machine data Analytics
 
Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...
Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...
Curiosity and Sauce Labs present - When to stop testing: 3 dimensions of test...
 
The DevOps paradigm - the evolution of IT professionals and opensource toolkit
The DevOps paradigm - the evolution of IT professionals and opensource toolkitThe DevOps paradigm - the evolution of IT professionals and opensource toolkit
The DevOps paradigm - the evolution of IT professionals and opensource toolkit
 

Plus de Christophe Rochefolle

Agile Secteur Public - Numérique Responsable
Agile Secteur Public - Numérique ResponsableAgile Secteur Public - Numérique Responsable
Agile Secteur Public - Numérique ResponsableChristophe Rochefolle
 
Une App responsable pour de la mobilité durable
Une App responsable pour de la mobilité durableUne App responsable pour de la mobilité durable
Une App responsable pour de la mobilité durableChristophe Rochefolle
 
Cloud Expo Europe 2018 - "Et si on testait en production ?"
Cloud Expo Europe 2018 - "Et si on testait en production ?"Cloud Expo Europe 2018 - "Et si on testait en production ?"
Cloud Expo Europe 2018 - "Et si on testait en production ?"Christophe Rochefolle
 
Qualité Logiciel - Outils Open Source pour Java et Web
Qualité Logiciel - Outils Open Source pour Java et WebQualité Logiciel - Outils Open Source pour Java et Web
Qualité Logiciel - Outils Open Source pour Java et WebChristophe Rochefolle
 
Automatisation des tests - objectifs et concepts - partie 2
Automatisation des tests  - objectifs et concepts - partie 2Automatisation des tests  - objectifs et concepts - partie 2
Automatisation des tests - objectifs et concepts - partie 2Christophe Rochefolle
 
Automatisation des tests - objectifs et concepts - partie 1
Automatisation des tests  - objectifs et concepts - partie 1Automatisation des tests  - objectifs et concepts - partie 1
Automatisation des tests - objectifs et concepts - partie 1Christophe Rochefolle
 

Plus de Christophe Rochefolle (12)

Agile Secteur Public - Numérique Responsable
Agile Secteur Public - Numérique ResponsableAgile Secteur Public - Numérique Responsable
Agile Secteur Public - Numérique Responsable
 
Une App responsable pour de la mobilité durable
Une App responsable pour de la mobilité durableUne App responsable pour de la mobilité durable
Une App responsable pour de la mobilité durable
 
Cloud Expo Europe 2018 - "Et si on testait en production ?"
Cloud Expo Europe 2018 - "Et si on testait en production ?"Cloud Expo Europe 2018 - "Et si on testait en production ?"
Cloud Expo Europe 2018 - "Et si on testait en production ?"
 
Paris Chaos Engineering Meetup #6
Paris Chaos Engineering Meetup #6Paris Chaos Engineering Meetup #6
Paris Chaos Engineering Meetup #6
 
Qualité Logiciel - Outils Open Source pour Java et Web
Qualité Logiciel - Outils Open Source pour Java et WebQualité Logiciel - Outils Open Source pour Java et Web
Qualité Logiciel - Outils Open Source pour Java et Web
 
Qualité logiciel - Generalités
Qualité logiciel - GeneralitésQualité logiciel - Generalités
Qualité logiciel - Generalités
 
Automatisation des tests - objectifs et concepts - partie 2
Automatisation des tests  - objectifs et concepts - partie 2Automatisation des tests  - objectifs et concepts - partie 2
Automatisation des tests - objectifs et concepts - partie 2
 
Automatisation des tests - objectifs et concepts - partie 1
Automatisation des tests  - objectifs et concepts - partie 1Automatisation des tests  - objectifs et concepts - partie 1
Automatisation des tests - objectifs et concepts - partie 1
 
Paris Chaos Engineering Meetup #5
Paris Chaos Engineering Meetup #5Paris Chaos Engineering Meetup #5
Paris Chaos Engineering Meetup #5
 
Jftl 2018 chaos engineering
Jftl 2018   chaos engineeringJftl 2018   chaos engineering
Jftl 2018 chaos engineering
 
Paris Chaos Engineering Meetup #2
Paris Chaos Engineering Meetup #2Paris Chaos Engineering Meetup #2
Paris Chaos Engineering Meetup #2
 
Paris Chaos Engineering Meetup #1
Paris Chaos Engineering Meetup #1 Paris Chaos Engineering Meetup #1
Paris Chaos Engineering Meetup #1
 

Dernier

Tekom Netherlands | The evolving landscape of Simplified Technical English b...
Tekom Netherlands | The evolving landscape of Simplified Technical English  b...Tekom Netherlands | The evolving landscape of Simplified Technical English  b...
Tekom Netherlands | The evolving landscape of Simplified Technical English b...Shumin Chen
 
Introduction to Machine Learning Unit-2 Notes for II-II Mechanical Engineering
Introduction to Machine Learning Unit-2 Notes for II-II Mechanical EngineeringIntroduction to Machine Learning Unit-2 Notes for II-II Mechanical Engineering
Introduction to Machine Learning Unit-2 Notes for II-II Mechanical EngineeringC Sai Kiran
 
Searching and Sorting Algorithms
Searching and Sorting AlgorithmsSearching and Sorting Algorithms
Searching and Sorting AlgorithmsAshutosh Satapathy
 
0950_Rodriguez_200520_Work_done-GEOGalicia_ELAB-converted.pptx
0950_Rodriguez_200520_Work_done-GEOGalicia_ELAB-converted.pptx0950_Rodriguez_200520_Work_done-GEOGalicia_ELAB-converted.pptx
0950_Rodriguez_200520_Work_done-GEOGalicia_ELAB-converted.pptxssuser886c55
 
presentation by faizan[1] [Read-Only].pptx
presentation by faizan[1] [Read-Only].pptxpresentation by faizan[1] [Read-Only].pptx
presentation by faizan[1] [Read-Only].pptxkhfaizan534
 
Introduction to Data Structures .
Introduction to Data Structures        .Introduction to Data Structures        .
Introduction to Data Structures .Ashutosh Satapathy
 
Chapter 2 Canal Falls at Mnnit Allahabad .pptx
Chapter 2 Canal Falls at Mnnit Allahabad .pptxChapter 2 Canal Falls at Mnnit Allahabad .pptx
Chapter 2 Canal Falls at Mnnit Allahabad .pptxButcher771
 
electricity generation from food waste - based bioenergy with IOT.pptx
electricity generation from food waste - based bioenergy with IOT.pptxelectricity generation from food waste - based bioenergy with IOT.pptx
electricity generation from food waste - based bioenergy with IOT.pptxAravindhKarthik1
 
عناصر نباتية PDF.pdf architecture engineering
عناصر نباتية PDF.pdf architecture engineeringعناصر نباتية PDF.pdf architecture engineering
عناصر نباتية PDF.pdf architecture engineeringmennamohamed200y
 
introduction to python, fundamentals and basics
introduction to python, fundamentals and basicsintroduction to python, fundamentals and basics
introduction to python, fundamentals and basicsKNaveenKumarECE
 
Road to GDSC (Become the next GDSC lead)
Road to GDSC (Become the next GDSC lead)Road to GDSC (Become the next GDSC lead)
Road to GDSC (Become the next GDSC lead)GDSCNiT
 
Conventional vs Modern method (Philosophies) of Tunneling-re.pptx
Conventional vs Modern method (Philosophies) of Tunneling-re.pptxConventional vs Modern method (Philosophies) of Tunneling-re.pptx
Conventional vs Modern method (Philosophies) of Tunneling-re.pptxSAQIB KHURSHEED WANI
 
A brief about Jeypore Sub-station Presentation
A brief about Jeypore Sub-station PresentationA brief about Jeypore Sub-station Presentation
A brief about Jeypore Sub-station PresentationJeyporess2021
 
The Art of Cloud Native Defense on Kubernetes
The Art of Cloud Native Defense on KubernetesThe Art of Cloud Native Defense on Kubernetes
The Art of Cloud Native Defense on KubernetesJacopo Nardiello
 
PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...
PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...
PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...Luuk Brederode
 
Research paper publications: Meaning of Q1 Q2 Q3 Q4 Journal
Research paper publications: Meaning of Q1 Q2 Q3 Q4 JournalResearch paper publications: Meaning of Q1 Q2 Q3 Q4 Journal
Research paper publications: Meaning of Q1 Q2 Q3 Q4 JournalDr. Manjunatha. P
 
EJECTOR REFRIGERATION CYCLE WITH THE INJECTION OF A HIGH DENSITY FLUID INTO A...
EJECTOR REFRIGERATION CYCLE WITH THE INJECTION OF A HIGH DENSITY FLUID INTO A...EJECTOR REFRIGERATION CYCLE WITH THE INJECTION OF A HIGH DENSITY FLUID INTO A...
EJECTOR REFRIGERATION CYCLE WITH THE INJECTION OF A HIGH DENSITY FLUID INTO A...marijomiljkovic1
 
Advanced Additive Manufacturing by Sumanth A.pptx
Advanced Additive Manufacturing by Sumanth A.pptxAdvanced Additive Manufacturing by Sumanth A.pptx
Advanced Additive Manufacturing by Sumanth A.pptxSumanth A
 

Dernier (20)

Tekom Netherlands | The evolving landscape of Simplified Technical English b...
Tekom Netherlands | The evolving landscape of Simplified Technical English  b...Tekom Netherlands | The evolving landscape of Simplified Technical English  b...
Tekom Netherlands | The evolving landscape of Simplified Technical English b...
 
Introduction to Machine Learning Unit-2 Notes for II-II Mechanical Engineering
Introduction to Machine Learning Unit-2 Notes for II-II Mechanical EngineeringIntroduction to Machine Learning Unit-2 Notes for II-II Mechanical Engineering
Introduction to Machine Learning Unit-2 Notes for II-II Mechanical Engineering
 
Searching and Sorting Algorithms
Searching and Sorting AlgorithmsSearching and Sorting Algorithms
Searching and Sorting Algorithms
 
Industry perspective on cold in-place recycling
Industry perspective on cold in-place recyclingIndustry perspective on cold in-place recycling
Industry perspective on cold in-place recycling
 
0950_Rodriguez_200520_Work_done-GEOGalicia_ELAB-converted.pptx
0950_Rodriguez_200520_Work_done-GEOGalicia_ELAB-converted.pptx0950_Rodriguez_200520_Work_done-GEOGalicia_ELAB-converted.pptx
0950_Rodriguez_200520_Work_done-GEOGalicia_ELAB-converted.pptx
 
presentation by faizan[1] [Read-Only].pptx
presentation by faizan[1] [Read-Only].pptxpresentation by faizan[1] [Read-Only].pptx
presentation by faizan[1] [Read-Only].pptx
 
Introduction to Data Structures .
Introduction to Data Structures        .Introduction to Data Structures        .
Introduction to Data Structures .
 
Chapter 2 Canal Falls at Mnnit Allahabad .pptx
Chapter 2 Canal Falls at Mnnit Allahabad .pptxChapter 2 Canal Falls at Mnnit Allahabad .pptx
Chapter 2 Canal Falls at Mnnit Allahabad .pptx
 
electricity generation from food waste - based bioenergy with IOT.pptx
electricity generation from food waste - based bioenergy with IOT.pptxelectricity generation from food waste - based bioenergy with IOT.pptx
electricity generation from food waste - based bioenergy with IOT.pptx
 
عناصر نباتية PDF.pdf architecture engineering
عناصر نباتية PDF.pdf architecture engineeringعناصر نباتية PDF.pdf architecture engineering
عناصر نباتية PDF.pdf architecture engineering
 
introduction to python, fundamentals and basics
introduction to python, fundamentals and basicsintroduction to python, fundamentals and basics
introduction to python, fundamentals and basics
 
Road to GDSC (Become the next GDSC lead)
Road to GDSC (Become the next GDSC lead)Road to GDSC (Become the next GDSC lead)
Road to GDSC (Become the next GDSC lead)
 
Conventional vs Modern method (Philosophies) of Tunneling-re.pptx
Conventional vs Modern method (Philosophies) of Tunneling-re.pptxConventional vs Modern method (Philosophies) of Tunneling-re.pptx
Conventional vs Modern method (Philosophies) of Tunneling-re.pptx
 
A brief about Jeypore Sub-station Presentation
A brief about Jeypore Sub-station PresentationA brief about Jeypore Sub-station Presentation
A brief about Jeypore Sub-station Presentation
 
The Art of Cloud Native Defense on Kubernetes
The Art of Cloud Native Defense on KubernetesThe Art of Cloud Native Defense on Kubernetes
The Art of Cloud Native Defense on Kubernetes
 
PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...
PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...
PhD summary of Luuk Brederode, presented at 2023-10-17 to Veitch Lister Consu...
 
FOREST FIRE USING IoT-A Visual to UG students
FOREST FIRE USING IoT-A Visual to UG studentsFOREST FIRE USING IoT-A Visual to UG students
FOREST FIRE USING IoT-A Visual to UG students
 
Research paper publications: Meaning of Q1 Q2 Q3 Q4 Journal
Research paper publications: Meaning of Q1 Q2 Q3 Q4 JournalResearch paper publications: Meaning of Q1 Q2 Q3 Q4 Journal
Research paper publications: Meaning of Q1 Q2 Q3 Q4 Journal
 
EJECTOR REFRIGERATION CYCLE WITH THE INJECTION OF A HIGH DENSITY FLUID INTO A...
EJECTOR REFRIGERATION CYCLE WITH THE INJECTION OF A HIGH DENSITY FLUID INTO A...EJECTOR REFRIGERATION CYCLE WITH THE INJECTION OF A HIGH DENSITY FLUID INTO A...
EJECTOR REFRIGERATION CYCLE WITH THE INJECTION OF A HIGH DENSITY FLUID INTO A...
 
Advanced Additive Manufacturing by Sumanth A.pptx
Advanced Additive Manufacturing by Sumanth A.pptxAdvanced Additive Manufacturing by Sumanth A.pptx
Advanced Additive Manufacturing by Sumanth A.pptx
 

From Duke of DevOps to Queen of Chaos - Api days 2018

  • 1. From Duke of DevOps To Queen of Chaos APIdays.io Paris December 11 & 12, 2018 Christophe ROCHEFOLLE Director Operational Experience @OUI.sncf @crochefolle
  • 2. Experienced IT executive providing tech & organization to improve #quality & #agility for IT systems, #ChaosEngineering fan Co-author of French DevOps book Who am I ?
  • 3. French National Railway Company Founded in 1938. First e-Commerce website in France IT Leader in mobility, transform your journey into an amazing experience Where is my playground ? 99,997% SLA availability OUR RECORD 39 TICKETS SOLD by SECOND SPEED RECORD 574.8 KM/H
  • 4. 2008 Andrew Shafer and Patrick Debois helds a "birds of a feather" session in 'Agile Toronto' 2009 “DevOpsDays” conference started in Belgium by Patrick Debois, and term “DevOps" coined 2009 “10 Deploys per Day at Flickr” talk by John Allspaw and Paul Hammond in “Velocity” conference 2009 In “Velocity” conference, Andrew Clay coined "Wall of confusion" 2009 Mike Rother wrote Toyota Kata and defined 'Improvement Kata' 2010 “Continuous Delivery” book from Jez Humble and David Farley, defined "deployment pipeline" 2011 “The Phoenix Project” book from Gene Kim and Kevin Behr 2011 Amazon deploys to production every 11.6 seconds 2014 “DevOps for Dummies” book by Sanjeev Sharma 2014 Etsy deploys more than 50 times a day 2016 “The DevOps Handbook” book by Gene Kim and Jez Humble 2016 First “DevOpsREX” conference in Paris 2018 “Mettre en oeuvre DevOps – 2nd Edition” book by Alain Sacquet and me 2008 2010 2011 2014 2016 2018 DevOps 2009
  • 5. DevOps: Shorten design to cash and quick feedback feedback
  • 6. Duke of DevOps Time is money. Your TTM rocks ! You have a master in CI/CD
  • 7. Queen of Chaos But the evil is coming !!! TIME TTM MTTR slow fast low high Increasing automation Faster release cycles Ephemeral knowledge Increasing complexity The automation paradox U-curve
  • 8. For the first time, availability is the main concern for IT European management, before security. Source: Master of Machines III
  • 9. Real life Focus was on the left side CI/CD Test automation Application Lifecycle Management Artifact management IaaS / PaaS / CaaS Deployment
  • 11. We need new ways to develop reliability concern for our teams
  • 12. …(an) error budget provides a clear, objective metric that determines how unreliable the service is allowed to be… SRE Error budget
  • 13. • paying off some technical debt • improve the logging to ease support • add some additional integration or end-to-end tests • do those first steps to enable blue/green deployments • implement service mesh But, when was the last time that your product owner willingly added any of those technical stories to the next sprint? Why having Error budget ? SRE Error budget
  • 14. Where to start ? 1. Convert unavailability to cash 2. Define Service Level Objective with business team 3. Define Error budget Availability = successful requests / (successful request + failed requests) A failed request can be: 1. A 500 response, due to some bug. 2. No response, due to the service being down. 3. A slow response: if the client gives up before the response is available, it is as good as no response. 4. Incorrect data, due to some bug. Error budget = (1 - availability) = failed requests / (successful requests + failed requests) So if a service SLO is 99.9%, it has a 0.01% error budget. If the service is serves one million request per quarter, the error budget tells us that it can fail up to ten thousand times. SRE Error budget
  • 15. SRE Error budget How to use it ? Company agreement: Teams may no longer make any new release without spending time improving the reliability of the service when error budget is 0. In fact, they better do improvement before it.
  • 16. We need new ways to know what f$$$ happens in production
  • 17. Monitoring systems have not changed significantly in 20 years and has fallen behind the way we build software. Our software is now large distributed systems made up of many non- uniform interacting components while the core functionality of monitoring systems has stagnated. Monitoring is dead @grepory, Monitorama 2016
  • 18. Why we need observability? Observability Complexity is exploding everywhere, but our tools are designed for a predictable world. • Can you understand what’s happening inside your code and systems, simply by asking questions using your tools? • Can you answer any new question you think of, or only the ones you prepared for? • Having to ship new code every time you want to ask a new question … SUCKS.
  • 19. Low Medium High Microservice that does one thing Function with no side effects Monolith with logging Monolith with tracing and logging Monitoring Thresholds, alerts, watching the health of a system by checking for a long list of symptoms. Black box- oriented. Observability What can you learn about the running state of a program by observing its outputs? (Instrumentation, tracing, debugging) Observability
  • 20. What do we want ? a system is observable when your team can quickly and reliably track down any new problem with no prior knowledge. Observability
  • 21. Where to start ? Observability • Rich instrumentation • Events, not metrics • No aggregation • Few dashboards • Test in production Internal state from software Wrap every network call, every data call Structured data only Arbitrarily wide events mean you can amass more and more context over time. Use sampling to control costs and bandwidth. Aggregates destroy your precious details. We need MORE detail and MORE context. Dashboard focus on specific known possible failure. We need to explore raw data to discover what we don’t know. If you already know the answer, do self-healing ! Software engineers spend too much time looking at code in elaborately falsified environments, and not enough time observing it in the real world.
  • 22. Need more information ? https://www.d2si.io/observabilite Follow @mipsytipsy engineer/cofounder/CEO “the only good diff is a red diff”
  • 23. We need shit-right testing RIGHT LEFT
  • 25. The performance of complex systems is typically optimized at the edge of chaos, just before system behavior will become unrecognizably turbulent. Chaos Engineering —Sidney Dekker, Drift Into Failure
  • 26. How much confidence we can have in the complex systems that we put into production? Why do we need Chaos Engineering ? Chaos Engineering With so many interacting components, the number of things that can go wrong in a distributed system is enormous. You’ll never be able to prevent all possible failure modes, but you can identify many of the weaknesses in your system before they’re triggered by these events.
  • 27. Queen of Chaos So, to fight the evil Chaos Engineering
  • 28. Chaos engineering is the discipline of experimenting on a distributed system in order to build confidence in the systems capacity to withstand turbulent conditions in production Principles of Chaos Engineering Chaos Engineering
  • 29. 2004 Chaos engineering 2010 2012 2016 2017 2018 2004 2010 2012 2016 2017 2018 Amazon—Jesse Robbins. Master of disaster Netflix—Greg Orzell. @chaosimia - First implementation of Chaos Monkey to enforce use of auto-scaled stateless services NetflixOSS open sources simian army Gremlin Inc founded Netflix chaos eng book. Chaos toolkit open source project Chaos concepts getting adopted widely !
  • 30. Where to start ? Chaos Engineering Hypothesis testing We think we have safety margin in this dimension, let’s carefully test to be sure In production Without causing an issue 1. Start by defining ‘steady state’ as some measurable output of a system that indicates normal behavior. 2. Hypothesize that this steady state will continue in both the control group and the experimental group. 3. Introduce variables that reflect real world events like servers that crash, hard drives that malfunction, network connections that are severed, etc. 4. Try to disprove the hypothesis by looking for a difference in steady state between the control group and the experimental group.
  • 31. • Simulating the failure of an entire region or datacenter. • Partially deleting Kafka topics over a variety of instances to recreate an issue that occurred in production. • Injecting latency between services for a select percentage of traffic over a predetermined period of time. • Function-based chaos (runtime injection): randomly causing functions to throw exceptions. • Code insertion: Adding instructions to the target program and allowing fault injection to occur prior to certain instructions. • Time travel: forcing system clocks out of sync with each other. • Executing a routine in driver code emulating I/O errors. • Maxing out CPU cores on an Elasticsearch cluster. Injecting Chaos Chaos Engineering
  • 32. Different experiments for every stage Chaos Engineering Infrastructure Switching Application PeopleGame days Simian Army chaostoolkit ChAP Gremlin
  • 33. Our story of Chaos Engineering @OUI.sncf 2015 2016 2018 Birth of an ambition : Chaos Monkey EXPERIMENTATION INDUSTRIALIZATION All critical applications run Chaos experiment 2017 OUR BESTIARY IS BORN IN OCTOBER 1ST DAYS OF CHAOS Detection : 87% Diagnostic : 73% Resolution : 45% RUN IN PRODUCTION First Chaos Monkey in production… …and production is still up 2ND DAYS OF CHAOS 3RD DAYS OF CHAOS To follow our experiment, birth of the
  • 34. https://days-of-chaos.slack.com Paris Chaos Engineering Meetup http://meetu.ps/c/3BMlX/xNjMx/f https://chaosengineering.slack.com http://days-of-chaos.com/ https://medium.com/paris- chaos-engineering- community
  • 35. SRE Error Budget Observability Test in production Chaos Engineering Continuous Quality CI/CD Test automation Application Lifecycle Management Artifact management IaaS / PaaS / CaaS Deployment

Notes de l'éditeur

  1. Wouldn’t it be nice to spend the next sprint or two paying off some of that technical debt that your project had accrued? Wouldn’t it be nice to improve the logging to ease support? Or add some additional integration or end-to-end tests? Or maybe do those first steps to enable blue/green deployments? But, when was the last time that your product owner willingly added any of those technical stories to the next sprint?
  2. Zoom la prochaine, comment on y est passé