1. Cisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. 1
What is Exactly Anti-Fragile
in DevOps?
Asher Sterkin
Distinguished Engineer, SPVSS, Cisco Video Systems, Israel
September 30, 2013

2. Cisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. 2

3. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3

4. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4
Antifragile
Some things benefit from shocks…
volatility, randomness, disorder,
and stressors and love adventure,
risk, and uncertainty… there is no
word for the exact opposite of
fragile. Let’s call it antifragile.
Nassim N. Taleb, “Antifragile. Things that gain from
disorder”

5. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5
The Book and Reactions
I think this concept is incredibly
powerful when applied to systems
and organizational architecture.
Jez Humble, “On Antifragility in Systems and
Organizational Structure”

6. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6
The Book and Reactions
The Netflix cloud architecture is
anti-fragile… The Netflix culture is
anti-fragile… Getting stronger
through failure is the basis of anti-
fragility. Avoiding failure at all costs
… makes you brittle and
vulnerable...
Adrian Cockcroft, “Looking back at 2012 with
pointers to 2013”

7. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7
The Book and Reactions
If the idea is nice and neat,
however, the book that houses it
is just the opposite. It is a big,
baggy, sprawling mess.
David Runciman, review of the book in Guardian
November 21, 2012

8. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8
Larger Body of Knowledge
• Complex Adaptive Systems
• Highly-Optimized Tolerance
• Technology Development Cycle
• Disruptive Innovations
• Product Development Flow
• Lean Start Up

9. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9
For Today
• De-fragilization
• Skin in the Game
• Barbell
• Asymmetric Pay-off

10. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10
DevOps Areas
OPSDEV
Area 1: extend delivery to
production
Area 2: extend operations
feedback to project
Area 3: embed project knowledge into operations
Area 4: embed operations knowledge into project
Patrick Debois: “Codifying devops practices”

11. Cisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. 11
De-Fragilization

12. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12
“Beauty plus pity-that is the closest we can
get to a definition of art. Where there is
beauty there is pity for the simple reason
that beauty must die: beauty always dies,
the manner dies with the matter, the world
dies with the individual.”
V. Nabokov, “Lecture on Metamorphosis”

13. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13
DevOps Areas
OPSDEV
Area 1: extend delivery to
production
Area 2: extend operations
feedback to project
Area 3: embed project knowledge into operations
Area 4: embed operations knowledge into project

14. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14
Continuous Delivery
... to exert a constant stress on
your delivery and deployment
process to reduce its fragility so
that releasing becomes a
boring, low-risk activity.
Jez Humble, “On Antifragility in Systems and
Organizational Structure”

15. Cisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. 15
Large batches increase cycle time
Large batches increase variability in flow
John Allspaw: “Ops Meta-Metrics”,
slides 103-109

16. Cisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. 16
Reducing batch size accelerates feedback
Reducing batch size reduces overhead
Reducing batch size reduces risk

17. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17
Batch size and Bottlenecks
Reduce batch
size before you
attack
bottlenecks

18. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18
DevOps Areas
OPSDEV
Area 1: extend delivery to
production
Area 2: extend operations
feedback to project
Area 3: embed project knowledge into operations
Area 4: embed operations knowledge into project

19. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19
“Resilience through Failure”

20. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20

21. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21

22. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22
Resilience to failure is a
lofty goal. It enables a
system to survive and
withstand failure. There's an
even higher peak to strive
for, however: making the
system stronger and better
with each failure.
A. Tseitlin, “The Antifragile Organization”
Culture of Continuous Learning

23. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23
E. Deming, “Out of Crisis”
“Confusing common
causes with special
causes will only make
things worse

24. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24
Resilience, Reliability, Robustness
Normal Operation
MTBF
Failure!
MTTD MTTR
Max Downtime
Normal Operation
MTBF
Max Data
Lost

25. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25
It’s me
“Obsessive protection of the
system against extremely rare
events makes it more fragile.
Resilience comes before the
last percentiles of reliability.

26. Cisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. 26

27. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27
DevOps Areas
OPSDEV
Area 1: extend delivery to
production
Area 2: extend operations
feedback to project
Area 3: embed project knowledge into operations
Area 4: embed operations knowledge into project

28. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28
Give ‘em pagers!

29. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29
“DevOps: architects,
developers, QA/QC,
system engineers, and IT
cooperate to maximize
the company value
It’s me again

30. Cisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. 30
Barbell

31. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31
DevOps Areas
OPSDEV
Area 1: extend delivery to
production
Area 2: extend operations
feedback to project
Area 3: embed project knowledge into operations
Area 4: embed operations knowledge into project

32. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32John Alsspaw, “Dev and Ops Collaboration”

33. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 33

34. Cisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. 34
Developers
• Responding to outages, takes
on-call
• Alerting systems thresholding,
design
• Architecture design and review
• Building metrics collection
• Application configuration
• Shipping public-facing code
• Responding to outages, takes
on-call
• Alerting systems thresholding,
design
• Architecture design and review
• Building metrics collection
• Application configuration
• Infrastructure
buildout/management
Operations
John Allspaw, “Reply to NoOps @ Netflix”

35. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 35
DevOps Areas
OPSDEV
Area 1: extend delivery to
production
Area 2: extend operations
feedback to project
Area 3: embed project knowledge into operations
Area 4: embed operations knowledge into project No, not really

36. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 36
Every analogy has its limit.
One just needs to learn where
to stop.

37. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 37
N. Taleb, “Antifragile”
“The first step to anti-fragility
consists in decreasing
downside…This brings us to the
solution in the form of barbell …
Away from Golden Middle.

38. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 38
Away from golden Middle
Risk
Aversion
Risk
Loving

39. Cisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. 39
Niek Bartholomeus,
“DevOps For Dinosaurs”

40. Cisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. 40Jeremy Edberg, “DevOps at Netflix”
Risk Aversion Risk Loving

41. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 41
N. Taleb, “Antifragile”
“The downside/lost should be
known and protected, not
probability

42. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 42
It’s me
“ You never know where from your next
outage or cyber attack will come from
and when.
The maximal downtime and maximal
data lost should be known and
guaranteed regardless of
probabilities.

43. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 43
Asymmetric Pay-Offs
x
Gain/Lossf(x)
Pain
Gain

44. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 44
DevOps Areas
OPSDEV
Area 1: extend delivery to
production
Area 2: extend operations
feedback to project
Area 3: embed project knowledge into operations
Area 4: embed operations knowledge into project

45. Cisco Confidential© 2013 Cisco and/or its affiliates. All rights reserved. 45

46. © 2013 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 46
N. Taleb
“Never be sucker. Period!

47. Thanks