Operations as a Strategic Weapon (Part 2)

Operations is a
Strategic Weapon
(Part Deux)

Saturday, September 24, 2011

Are you an ...

aaS

-If it's not it should be
-Let's take a look at GW.com.. yep
-they sell GW as a service GreenWidgits_aaS
-Obviously you better have kick “ass” green widgets; Pun intended....
-But today that is not enough....
----------------------------------------
-What does amazon and netﬂix get that barnes and noble and blockbuster don't ?
_aaS (circle the "S")

-Over the next 45 minutes I am going to try and convince you that the principles behind Devops are the best solution to
address the "S" in your aaS


-Most of you are probably familiar with “Software” TD
-I am going to tell you a story about “Infrastructure TD”


-How many ppl know who these guys are?
-The dudes who invented Facebook.. rght?
-However, this was not there ﬁrst venture.
-Green vs Red Widgits compnay story.
-After many arguments the both decide to split up. Cameron to make green widgets and Tyler
decides to make red widgets.
-They both go to their father each ask for a 1m dollars and the father asks how much money
are you going to make. They both say 10 million.
-Father calculates the ROR to be 900% and gives em each 1m.
10-1 = 9/1 = 900%
-However one of them lied.... The red widgets only return 233%
10-1-2 = 7/3 = 233%

Green vs Red
Widgits

1 Million 1 Million
2 Million No TD
10 Million Proﬁt 10 Million Proﬁt
10 - 3 = 7/3 10 - 1 = 9/1
233% ROR 900% ROR


-Father is not pleased with Rehaus.

It Gets Worse...

Technical Vicious Toxic
Terminal
Debt Cycle Operations


- It gets even worse. Rehause starts getting what Isreal Ghat of the Cutter Group call the vicious cycle of TD.
- You wind up fixing a lot of things that you didn’t fix in the first place..
- this pulls more resources from delivery good service and compound effect is that you are spending more and more
resources that you should have gotten right the firs time..
- but ever worse the effect of customer satisfaction starts loosing more business and the “V” cycle is out of control.
- TD->VTD->Toxic operations->Terminal

- I call this are you running a business or building a business?
- Toxic operations. Amertrade/etrade story

Tale of Two Startups


- Jesse Robbins my ex boss and CEO of Opscode/Chef did a great post on O’rielly rdar a few years ago called the Tale of
Two Startups.
- The chart looked like this ﬁrst 4 weeks.
- First chart legacy (I call it the non devops startup/project)
- Second Chart is the (secret sauce startup ... #devops)
- I played around with this using “R” to be cool and I came up with a=140% ROR and 700% ROR

The Meat to Math Ratio

http://radar.oreilly.com/2011/08/meat-to-math-ratio.html


- Alistar Croll has a great post called the meat to math ratio.
- Amazon had $12.95B in Q410 revenues and 33,700 employees, revenue per employee of $384,273.
- For Barnes & Noble: Barnes & Noble: $1.91B in Q410 revenues, and 35,000 employees, meaning a revenue per employee
of $54,571.
- Netﬂix: $444M in Q409 revenues, and 1,000 employees, meaning a revenue-per-employee of $444,000.
- Blockbuster: $400M in Q409 revenues. The company peaked at 60,000 employees i
- Dropbox: In Q211 Dropbox had $25M in revenues, and 74 employees, for a revenue per employee of $338K.


•I got into a Twttier argument that went like this...
• See “Cloud Gone Wrong”

Clouds Gone Wild

Busine
s s
Dev

AWS
business s3put s3
ssh
ideas !
apache/php Rightscale
memcached
mysql


-So letʼs start the story in the begining... Biz guy got a great idea for a service.

-They used a classic web2.0 app architecture: apache/php, memcached & mysql.
Development done on a single server
-Production ran on EC2 and they used rightscale server templates
-Release done by pushing code and assets to s3 buckets and then running a parallel SSH
scripts to distribute them
-This approach seemed to work... They got up to a few hundred nodes pretty fast.. business
was cooking

apache/php
p arty,
3rd s3
memcached
& are AWS mysql
w
mi ddle
Rightscale

yum
Business
apache/php puppet
memcached
mysql apache/php
s3
AWS s3 memcached
AWS mysql
Rightscale
Rightscale
s3put s3put
ssh ssh

Business2 Business3

-First service was such a huge success they decided to launch other sevices
-So they ”Copy and pasted” the whole architecture and lifecycle to launch the new businesses
-Each new group pushed assets to s3, scripted the distribution, and hacked the rightscripts and
templates
-Things were obviously getting more complicated, so they did what they were supposed to do
and added centralized tooling like puppet and yum
-They thought they were doing things the cloud way and that all would be ﬁne

“As-is”
operations server
templates

TAR
RPM restarts

TAR
TAR
code,
EC2
content pupp
deploys et RS
app RS deployment
devs deployment
S3

yum
repo
Provision-time builds

reconfigs

TAR EXE
TAR AUTO
server
TOOLS
code templates
TAR
RPM
middleware
devs platform CONTROL PROVISION RELEASE

system
eng


First we got the team on the whiteboard to map out the “as is” picture. This is a, believe it or
not, a simplified version of that.
Some of the highlights...
-First youʼll notice that different groups had their own path to production... different methods of
control, provisioning, and release.
-Each group and role seemed to have a different way to editing or storing config
-There were differing ways of packaging software... sometimes it might be a .tar.gz other times
it might be an RPM.
-Shockingly... Some things were even being built directly on production servers.
-There was no authoritative source of information is maintained about nodes, application
topology, software versions, etc....
-Everything was being stored in S3 buckets... which is great because itʼs so easy to use... but
itʼs unversioned and people would just upload whatever was newest for them. But that stuff
was never really tested in unison.. so old stuff wasnʼt working with new stuff... and its was
unclear what was different or why it was different.
-We can go on... but you should get the point that they had the right cloud, the right tools, and
lots of smart people, but it all got dangerously out of control very quickly

Other:

-Changes hit all customers at once => Puppet configs in unversioned S3 buckets
-Buggy node classification causing provisioning problems = > complex/long node classifier
script
-”Dead boxes” after provisioning => rightscript/puppet ordering problems
-”my box got clobbered!” => puppet, is it supposed to be on or off?
-new environment setup was taking longer and longer => from days to weeks because of

Tec
hnic
al


Infrastructure Deveopment Life Cycle
Everything starts
here
SVN

hud agent cfg
yum pup
son repo pet

packages
active users run packages

directory deck
commands
nag splunk
ios
resource model

sys cfg

right instance create
scale

events
new node Node a log data

CONTROL PROVISION RELEASE


I wonʼt go into too much detail about the tooling that was put into place to support all of this but
here are some highlights..
-took a loosely coupled toolchain approach... using mostly open source tools
-This became their standard stack of “operations middleware”. Of course, we are all used to
the notion of application middleware... but to an online service, the management infrastructure
is just as much a part of the service as the application itself.
-This operations middleware stack is a ﬁrst class citizen along with application stack and it all
goes through the same SDLC... everything is versioned, built, deployed, and packaged via the
same process
-Once in place, this middleware provides a single path for releasing, provisioning, and
controlling anything that goes into an environment.

Other:
Management infrastructure based on “swap-able” sets of integrated tools
- Organized into three rough categories: Control, Provisioning, Monitoring
- Control tools support routine and ad hoc procedures executed as commands/scripts
- Provisioning tools support package delivery and post install customization
- Monitoring tools actively check health and collect log data

When you think of middleware, you think of where your app code works.. but in the service
world, you have an operations middleware that is just as important. All the provisioning and
management stuff... itʼs just as important. Itʼs one and the same.

Solve the information problem... where is the system of record? easy in cloud to get basic
node data... that comes from the compute service... but what about everything else you need
to manage your infrastructure?

Meet your competition


•Betting on better experience
•Attacking / chased by the majors
•Real money at stake


One last quick tale for you all....

So what does it look like when you start with the end in mind...

Let’s look a hot startup called Wealthfront
-Its a ﬁnancial services company... but all you have to know that it manages real people’s
retirement funds. It’s not a game or a toy.
-They are being chased by the majors so they have to move fast and can’t screw up
-To make this work, they subscribe to the theories of these two gentlemen, steven blank and
eric reiss, and are using the Lean Startup methodology to run their business... and all you
really need to know about that right now is that it all hinges on getting new ideas in front of
your customers as fast as possible and getting feedback as fast as possible.

Customer discovery -> Customer Validation -> Customer Creation -> Company Building


So what was immediately identiﬁed as a major impediment to that fast cycle?

The traditional development and deployment cycle. Even the 1 - 4 weeks cycles that are so
common is too long to wait.


So they rewired it in classic Continuous Deployment fashion...
Get rid of the idea of batching up change for a release... They are always deploying as soon
as a a piece of code is ready
Don’t have staging... that’s a waste of time... production is the real environment anyway
Don’t have an explicit QA stage... built extensive automated testing in all across the lifecycle
and made quality be everyone’s responsibility
They are always deploying from trunk... so no need to worry about patching and merging
branches
oh and do they whole thing from check-in to running in production in 5 - 10 minutes
are they crazy? They don’t think so. They have extreme conﬁdence in this system and are
kicking ass with it.

go read: http://eng.wealthfront.com

The results are striking...

They are sometimes averaging 30 deployments to production a day. But they aren’t sacrificing
quality!

A release can get from check-in in to production in under 10 minutes but it’s got to pass over
6k tests and then there is a phased rollout with ongoing and extensive system and traffic
analysis post deployment.

The result is that their customers are happy. Their partners are happy. And more importantly
their team is happy...no burnout and releases are non-events. Want to go on vacation? No
problem, any developer can step in and take over your task or fix problem because of
extensive test coverage and confidence in their quality systems.

Operations is their strategic weapon

go read: http://codeascraft.etsy.com

The results are striking...

They are sometimes averaging 30 deployments to production a day. But they aren’t sacrificing
quality!

A release can get from check-in in to production in under 10 minutes but it’s got to pass over
6k tests and then there is a phased rollout with ongoing and extensive system and traffic
analysis post deployment.

The result is that their customers are happy. Their partners are happy. And more importantly
their team is happy...no burnout and releases are non-events. Want to go on vacation? No
problem, any developer can step in and take over your task or fix problem because of
extensive test coverage and confidence in their quality systems.

Operations is their strategic weapon

95% 5%

Necessary Liability Strategic Weapon


DevOps Cafe

subscribe in

iTunes

I also do a podcast with the famous cloud and IT management guru john willis.
Interview based series where we talk to all kinds of movers and shakers across the
development and operation spectrum.

Between DTO and doing the devops cafe content I get to talk to a lot of companies and see
what’s working and what isn’t working.

Let’s Talk....

@botchagalupe

john@dtosolutions.com


We do this stuff all day long for a lot of large and cutting edge clients... and we love talking
about DevOps so drop me a line anytime if you want to talk

Let’s Talk....

@botchagalupe dev2ops.org

john@dtosolutions.com


We do this stuff all day long for a lot of large and cutting edge clients... and we love talking
about DevOps so drop me a line anytime if you want to talk

Extra Slides


C
A
M
S

First get you mind around what you are looking for... Simple framework for categorizing
DevOps problems and solutions.

Culture
Automation
Measurement
Sharing

Operations as a Strategic Weapon (Part 2)

Recommandé

Recommandé

Contenu connexe

Similaire à Operations as a Strategic Weapon (Part 2)

Similaire à Operations as a Strategic Weapon (Part 2) (20)

Plus de John Willis

Plus de John Willis (20)

Dernier

Dernier (20)

Operations as a Strategic Weapon (Part 2)