Kriss Rochefolle: "How to Convince Your Boss to Say "Yes!" to Chaos Engineering" - Chaos Conf 2018
2. « We will break
everything in
production, and
that will be fun! »
The «coming with
big clogs — Gros
Sabots» way
3. Experienced IT executive
providing tech & organization
to improve #quality & #agility
for IT systems,
#ChaosEngineering fan
Co-author of French
DevOps book
Who am I ?
4. French National Railway
Company
Founded in 1938.
First e-Commerce website in
France
IT Leader in mobility,
transform your journey into an
amazing experience
Where is my playground ?
99,997%
SLA availability
OUR RECORD
39
TICKETS SOLD by SECOND
SPEED RECORD
574.8
KM/H
5. Our story of Chaos Engineering
2015
2016 2018
Birth of an
ambition : Chaos
Monkey
EXPERIMENTATION
INDUSTRIALIZATION
All critical
applications run
Chaos experiment
2017
OUR BESTIARY IS BORN IN OCTOBER
1ST DAYS OF CHAOS
Detection : 87%
Diagnostic : 73%
Resolution : 45%
RUN IN PRODUCTION
First Chaos Monkey in
production…
…and production is
still up
2ND DAYS OF CHAOS 3RD DAYS OF CHAOS
To follow our
experiment, birth of
the
7. How to convince your boss
and make them say “Yes!” to
Chaos Engineering?
16. Adjust your story
to the different
directors of your
executive board
Sociodynamics
17. Foes Mutineer, Moaners and Opponents
They are a handful.
However they have the ability to cause great damage to the
synergy within the project team.
Opponents and Mutineers are against your project for various
reasons and they take any opportunity to antagonize the
project delivery.
This handful of individuals may rally those on the fence to be
antagonistic towards the project.
Established who they are, their concerns and evaluate if they
can be resolved e.g. do they want to be more involved.
If not, contain their impact on the passives as they would sway
against you.
18. Friends
These are a dream as they are Promoters.
They go out of their way to support the project.
They are dedicated to their task, quick to provide
resources and aid in issue resolution.
It is important to know these individuals and
maintain them as allies.
Ensure you understand what is critical to them to
make the delivery.
Zealots and Influencers
19. They define 40–80% of the stakeholders.
Despite ones excellent inter personal skills you may not be able
to reach all your stakeholders.
A key asset in winning the masses is using influencers who run
your campaign and positively influence the passives.
As passives are impressionable they are easily swayed based
on the feel in your project and it’s important they moved in
your preferred direction.
Winning the passives increase the synergy with your project
team and increase the chances of project success.
They need to feel part and parcel of the project i.e. are
consulted and informed in what is going on.
Fence Passives & Waverers
21. Very simple view of
our brain, it is far
more complex.
Once the conditions of a
serene exchange are here
and the players identified,
it is possible to proceed to
the next step.
MacLean Triune brain
Our target
25. Sometimes… it is just
a matter of time
It is during this type of incident
that you should not hesitate to be
opportunistic to advance your
pawns, to propose new practices
that will limit the impact in future
incidents.
26. Werner Vogels
VP & CTO at Amazon.com
“Everything fails
all the time”
The main point is no more to
avoid failures, but to limit
impact of those failures.
27. Resilience
« Did not even hurt me ! »
One of the best way to
deal with impact is
resilience —Ability to
recover quickly from
difficulties.
28. Better experiment in production
during day time with everyone
available, thus training teams and
limit impact,
…instead of react during night
outage with sleepy people and
limited resources.
The final goal of Chaos Engineering
is to sleep like a log, without
worrying about issues.
Sleep like a log
29. Red Queen hypothesis
Proposed in 1973 by Leigh Van Valen
The Red Queen Hypothesis in evolutionary
biology states that to survive an evolving
system, one must co-evolve with the best
traits to survive that system.
As the economy declines and evolves, your
role as entrepreneur is to develop survival
skills to keep your business in the running.
30. It’s all new, but not really
#disasterRecovery
#incidentManagementSystem
#CrisisManagement
32. …and it is just the beginning
Evolution of “Chaos Monkey” vs “Chaos
Engineering” since June 2010 on Google
Trends
Chaos Engineering au sein du
Technology Radar de ThoughtWorks
33. For the more adventurous,
highlight the innovative side
and precursors of these
practices.
Even if we are more and
more, it is a discipline that
remains to be explored.
Explore a new
playground
You may start by telling them you will break everything in production, and that will be fun! We call this in French “coming with big clogs — Gros Sabots”. They will see you coming from far, but except for noises, there is few chances you will hit the target.
Zoom la prochaine, comment on y est passé
They will probably ask for Return of Investment (R.O.I.). By arranging number of incidents, outage by year, cost of outage, and some hypothesis to reduce them, you may easily build a proposition with 2–3 years R.O.I.
However, between a project with new revenue and one which will maybe avoid losses, if they have to choose, the first one will be probably chosen first.
The rational approach is not the best one for a subject like Chaos Engineering. You must review your influence strategies.
It is not only your boss you have to convince, it should be the whole board of director.
You must review your influence strategies.
Biologist Henri Laborit wrote in 1976: “Faced to unknown experience, man has only three choices: fight, do nothing or flee “. This subject is quite new, with few visibility, it is important to allow your boss to discover the concept at his speed, to avoid instinctive rejection.
To be able to adjust your story to the different directors of your executive board, you need to identify the kind of players they are, among other things using sociodynamics.You will be able to focus by adapting your strategy and identifying the way they move and act. Using your energy on the right priorities to make them allies.
What are their major challenges?
What they may win or lose by collaborate with you?
What are their influences on other players?
Take time to identify the key issues at stake with decision-takers, as well as the brakes and leverages in relation to your subject. Concerns and objections are only excuses for your opponents to test you, but may also be real blocking points. You will win them by studying and including their views, but not necessarily during formal meeting…
Changing the global dynamic can take many forms. Sometimes it will be enough to change the views of some player on the subject …
Once you have identified your players, you should be adapt your story to their concerns, questions or objections. Never hesitate to play with emotions, they are a major factor in decision making.
The most obvious emotion to play with is fear: fear of major outage. The one which will impact your revenue.
For instance, 5-minute outage represent around one million dollars for Google/Alphabet, one hundred thousand dollars for Netflix and 3 million for Apple
Moreover, it is during this type of incident that you should not hesitate to be opportunistic to advance your pawns, to propose new practices that will limit the impact in future incidents.
Werner Vogels, Vice President & Chief Technology Officer at Amazon, introduce this subject by explaining that with size and continuous evolution of our system: “Everything fails all the time”.
Taken from Lewis Carroll's Through the Looking Glass, the Red Queen tells Alice as they run a race against each other, "It takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!"
You may also point out links with actual practice. For business continuity plan, you already do experiments in production, disaster recovery testing. This type of experiment is at the heart of the Chaos Engineering approach, the main difference is that we don’t want to do it once, manually. We want to do it automatically and continuously.
For the most concerned, do not hesitate to show that experiments of this type exist in more and more companies, as you can see in this mind map:
· Cloud and hosting companies: Amazon, Azure, Google, Digital Ocean, Pivotal Cloud Foundry, Twillio, Dropbox …
· Companies for final users: Netflix, Facebook, LinkedIn, Uber, OUI.sncf,…
· And even bank: Fidelity Investments.
In summary, as in the parable of the blind men and the elephant, every member of your executive board will see the point that will make them confident or even enthusiast.
If you still think that R.O.I is the only way that will work for you, and that this article is just bullshit, please have a look at Gary Yukl “Leadership in Organization”.
Rational persuasion like R.O.I only provides 23% of engagement whereas inspiration tactics (90%), consultation (55%) and personal appeals (42%) are far more efficient.
In any case, buying a beer (31%) is always better than pressure (3%). In French, we say that we do not suffer pressure, we drink it, as pressure (“pression” in French) is a synonym for beer.
Pioneers are brilliant people. They are able to explore never before discovered concepts, the uncharted land. They show you wonder but they fail a lot. Half the time the thing doesn't work properly. You wouldn't trust what they build. They create 'crazy' ideas. Their type of innovation is what we call core research. They make future success possible. Most of the time we look at them and go "what?", "I don't understand?" and "is that magic?". In the past, we often burnt them at the stake. They built the first ever electric source (the Parthian Battery, 400AD) and the first ever digital computer (Z3, 1943).
Settlers are brilliant people. They can turn the half baked thing into something useful for a larger audience. They build trust. They build understanding. They make the possible future actually happen. They turn the prototype into a product, make it manufacturable, listen to customers and turn it profitable. Their innovation is what we tend to think of as applied research and differentiation. They built the first ever computer products (e.g. IBM 650 and onwards), the first generators (Hippolyte Pixii, Siemens Generators).
Town Planners are brilliant people. They are able to take something and industrialise it taking advantage of economies of scale. This requires immense skill. You trust what they build. They find ways to make things faster, better, smaller, more efficient, more economic and good enough. They build the services that pioneers build upon. Their type of innovation is industrial research. They take something that exists and turn it into a commodity or a utility (e.g. with Electricity, then Edison, Tesla and Westinghouse). They are the industrial giants we depend upon.