My presentation at the CloudOps Workshop of Cloud Connect 2010

1. Cloudops fundamentals
System Management
Bret Piatt
Rackspace Hosting

2. Static configuration, meet the dodo bird
Severs "appear" based on demand..
Servers "disappear" and it isn't a problem "sometimes"..
Management systems must talk
to the cloud APIs. The source of
truth is no longer "the Visio"..
Diagrams now show functional elements, not specific
systems

3. "Instant" provisioning is not instant
To provision you have to get data to the new node....
5GB takes 7 minutes on a 100Mbps network..
5TB takes 12 hours on a 1Gbps network..
5PB takes a truck ... or 52 days on a 10Gbps network!

4. So what do I do with all of my tools?
Many tools end up in the garbage..
..unless they add (or you
write) a dynamic configuration
module..
..Devops, you may need to
code not just script..

5. Provision by API, configure by hand?!#?
Chef
Puppet

6. Dev & Ops, sitting in a tree
Your revision control system is not just for source code...
Use it to track all system configuration files...
Who hasn't had a dream about the day
they can see every change of a
configuration file on a server
over the entire lifespan?

7. Meet Pallet, your new cloud creating friend
Chef and Puppet are great for configuring systems...
...but we have to first... have systems to provision!
With Pallet you can create & configure
systems in a management shell..
;; We can create a node, by specifying a name tag and a template.
;; webserver-template is a vector specifying features we want in
;; our image.
(start-node context :webserver webserver-template)
;; At this point we can manage instance counts as a map.
;; e.g ensure that we have two webserver nodes
(with-node-templates templates
(converge context {:webserver 2}))
;; Images are configured differently between clouds and os's.
;; We might want to update our machines to use the latest
;; package manager. pallet has a couple of templates, and
;; you can add your own, see resources/bootstrap. Templates
;; have a default implementation, but can be specialised
;; for a given tag or operating system family.
(with-node-templates templates
(converge context {:webserver 1}
(bootstrap-with (bootstrap-template :ensure-resolve)
(bootstrap-template :update-pkg-mgr))))

8. Get it right once, never make a mistake again
Embrace TDD (Test Driven Design) in Ops..
Get the requirements from the business..
1. Maintain 1,000,000 open connections
2. Process 1,000 events per second
3. Run for 1 week without errors
Go build your tests..
Now hack the system until you have a configuration that
passes #1 and #2, then let the tests run until it passes #3..
..and since you built it with Chef/Puppet it is reusable!

9. Let the software work for you
Continuous Integration (CI) is for operations too!
Ops needs a development world view..
The job isn't done when, "Everything is working normal."
Continually try to improve configurations in the sand box..
..using TDD go from staging to production at improvement
milestones or a "sprint" release cycle...agile operations!

10. "Those lazy Ops guys.."
Unit tests now become
app monitoring
configuration files..
..eliminate the, "Our test
suite doesn't show the
error you see in
production" conflicts -- maintain a single suite.
def test_sign_up http://wiki.github.com/brynary/webrat/
visit "/" click_link "Sign up"
fill_in "Email", :with => "good@example.com"
select "Free account"
click_button "Register" ... end

11. The moral of the story, "Ops.."
.. is more programatic as APIs replace long lead time steps.
..needs to leverage development best practices.
..is not about just "keeping the lights on."
..is about optimizing everything below the app.
..must change the processes in use today.
..is more technical now than ever before.
..is becoming "software development" at the system level.

12. Questions?
Twitter: @bpiatt
bret.piatt@rackspace.com