Madrid DevOps Meeting in September 2013 about how to create a whole environment of immutable servers in AWS with Vagrant and Chef Server. More information about this topic available in http://juanvicenteherrera.eu
3. Index
1. Requisites
2. Vagrant Box
3. Vagrantfile
4. Vagrant Settings
5. Vagrant Commands
6. Provisioning with Chef
7. Multi machines
8. Version control
9. Cleanup
10.Conclusions
11.References
4. Why a common toolchain?
• Consistency
• Efficiency
• Safety
• Automation
6. Quick view
Chef Server
AMI base
Dev/Ops workstation
(Vagrant+knife)
VPC Subnet
App instances
VPC Subnet
Virtual Private Cloud
AWS Cloud
1. Create instances from AMI
base and make chef
bootstrap
2. Apply roles
7. Vagrant Box
Base Box: The base box is simply a saved hard-disk of a Virtual Machine
created with VirtualBox. It can contain anything but it needs at least :
• Ruby
• VirtualBox guest additions
• Puppet or Chef
Vagrant AWS Example Box
Vagrant providers each require a custom provider-specific box format. These
files compose the contents of a box for the AWS provider.
• README.md
• Vagrantfile
• metadata.json
To turn this into a box:
$ tar cvzf aws.box ./metadata.json ./Vagrantfile
8. Vagrantfile EC2
Vagrant.configure("2") do |config| # Version 2 Vagrant syntax
config.vm.box = "dummy"
config.omnibus.chef_version = :latest
config.vm.provider :aws do |aws, override|
aws.keypair_name = "bq”
aws.access_key_id = ”xxxxxxxxx"
aws.secret_access_key = ”xxxxxxxxxxxxxxxxxx"
aws.ami = "ami-3ad1af53”
override.ssh.username = "ec2-user”
override.ssh.private_key_path = "/Users/juanvi/keypairs/bq.pem"
end
end
9. Vagrant Settings
Creating Vagrant File
mkdir [project name]
cd [project name]
Vagrant init -> creates Vagrantfile
Vagrant ec2 test 2 ec2 without chef management
$ vagrant plugin install vagrant-aws
$ vagrant box add dummy https://github.com/mitchellh/vagrant-
aws/raw/master/dummy.box
$ vagrant up --provider=aws
This box works by using Vagrant's built-in Vagrantfile merging to setup
defaults for AWS. These defaults can easily be overwritten by higher-
level Vagrantfiles (such as project root Vagrantfiles).
10. Vagrant Commands
vagrant box # Commands to manage system boxes
vagrant destroy # Destroy the environment, deleting the created virtual
machines
vagrant halt # Halt the running VMs in the environment
vagrant help [TASK] # Describe available tasks or one specific task
vagrant init [box_name] [box_url] # Initializes the current folder for Vagrant usage
vagrant package # Package a Vagrant environment for distribution
vagrant provision # Rerun the provisioning scripts on a running VM
vagrant reload # Reload the environment, halting it then restarting it.
vagrant resume # Resume a suspended Vagrant environment.
vagrant ssh # SSH into the currently running Vagrant environment.
vagrant ssh_config # outputs .ssh/config valid syntax for connecting to this
environment via ssh
vagrant status # Shows the status of the current Vagrant environment.
vagrant suspend # Suspend a running Vagrant environment.
vagrant up # Creates the Vagrant environment
vagrant version # Prints the Vagrant version information
11. Provisioning with Chef
web.vm.provision :chef_client do |chef|
chef.chef_server_url =
"https://api.opscode.com/organizations/juanvi"
chef.validation_key_path = "/Users/juanvi/chef-
repo/.chef/juanvi-validator-new.pem"
chef.validation_client_name = "juanvi-validator"
# Provision with the database role
chef.add_role("webserver")
# Set the environment for the chef server
chef.environment = ”prod"
12. Multi Machines
Vagrant.configure("2") do |config|
config.vm.define :web do |web|
web.vm.box = "apache"
end
web.vm.provision :chef_client do |chef|
chef.chef_server_url = chef_server_url
chef.validation_key_path = validation_key_path
chef.validation_client_name = validation_client_name
…….
end
config.vm.define :db do |db|
db.vm.box = "mysql"
end
db.vm.provision :chef_client do |chef|
chef.chef_server_url = chef_server_url
chef.validation_key_path = validation_key_path
chef.validation_client_name = validation_client_name
…
end
end
13. Provisioning with Chef
Provisioning only one kind of server executing
chef_client
$ vagrant provision web --provision-with
chef_client
Provisioning the whole platform executing chef_client
$ vagrant provision --provision-with
chef_client
After editing the Vagrantfile you need to 'reboot' the
machine to take this settings
$ vagrant reload
14. Version control with git
Version control
Now is a good time to version control our awesome-
vagrant project
$ cd
$ git init
$ git add Vagrantfile
$ git commit -m "This was just my first commit"
15. Cleanup
$ vagrant plugin install vagrant-butcher
then you can terminate a instance and deregister in the Chef
server executing:
$ vagrant destroy -f
[Butcher] knife.rb location set to
'/path/to/knife.rb'
[Butcher] Chef node 'node_name' successfully
butchered from the server...
[Butcher] Chef client 'node_name' successfully
butchered from the server...
[default] Forcing shutdown of VM...
[default] Destroying VM and associated drives...
16. • The task of writing recipes is not picked up by all team
members, and seem to stay the main job of the system
oriented people on the team.
• Reading Chef Cookbooks help people understand
what is needed and makes it easy to point out what
needs to be changed
Conclusions
17. Conclusions
• Before committing cookbooks-> destroy 'development'
box and re-provision a new box to see if this works.
(immutable servers)
• The longer the provision takes, the less frequent
people do it. It's important to keep that process as
fast as possible.
18. Conclusions
• Installation problems would get noticed far
sooner in the process.
• People would only do a full rebuild in the
morning when getting their coffee.
19. Conclusions
• It clearly helps everybody to have a consistent
environment to develop against, the latest version is just
one git pull away.
• The central approach drives people to
• do frequent commits
• do stable commits.
20. Conclusions
• Vagrant is an essential part of the DevOps process:
it is the solution to developing, testing and deploying
in the same environment. It thus ensures a smoother
transition of your project from the dev team to the ops
team.
• Chef has to be the only way to manage the
configuration; NO MORE MANUAL CHANGES
21. Hands On
A typical situation in a company that makes software is the
arrival of a new developer that has the tough task to install
a lot of tools in his workstation that in the best case takes
days.
We are going to create a whole developer stack in AWS
not for this new developer but for a complete new
developer team!
After make magic with Vagrant and Chef the unique task
for all of the members of the team to begin to work with the
environment is clone the git repository, receive the chef
and EC2 certificates/keypairs from their project leader.
22. Hands On
1. Download project (private repository)
2. Configure ~/.chef/aws_credentials.rb with your AWS security credentials
3. Check that you have in ~/.chef the chef and aws credentials and keypairs.
4. Well you are ready to create the whole demo environment!
5. $ gem install vagrant
6. $ vagrant plugin install vagrant-aws
7. $ vagrant plugin install vagrant-omnibus
8. $ environment=prod vagrant up --provider=aws
9. Tweak in the /etc/hosts url wiki.juanvicenteherrera.es pointing
to elastic ip assigned to ws.(only for test, in prod env change
dns entry or proxywebserver virtualhost)
10. Check the url http://wiki.juanvicenteherrera.es
11. Congrats! IT’S MAGIC! You have got your first complete environment created with
Vagrant and Chef!
Notes: if you don’t want to import a initial db dump…
1. This step can be avoid changing node attribute db_load to 0 in
environments/prod.json.
23. Create a new demo environment called demo
1. $ cp environments/prod.json environments/demo.json
2. Change id env and ips
3. $ knife environment from file environments/demo.json
4. Change values for all of the attributes with the appropiate demo values
5. $ vi Vagrantfile -> add the environment in the array of the allowed
environments
6. if (environment_id =~ /^(prod|qa|dev|demo)$/)
Add demo and app db password in encrypted databags for the new
environment
7. $ knife data bag edit --secret-file ~/.chef/encrypted_data_bag_secret secrets
php_app
8. $ knife data bag edit --secret-file ~/.chef/encrypted_data_bag_secret secrets
postgresql
9. $ environment=demo vagrant up --provider=aws
10.Tweak in the /etc/hosts url wiki.juanvicenteherrera.es
pointing to elastic ip assigned to ws.(only for test, in
prod env change dns entry or proxywebserver virtualhost)
11. Check the url http://wiki.juanvicenteherrera.es
Hands On
24. Video with the execution of the hands on available
at:
https://vimeo.com/76336356
For more information about this hands on check
http://juanvicenteherrera.eu/2013/09/04/immutable-
servers-with-vagrant-and-chef-chapter-i/
Hands On
25. References
• Video of this presentation recorded in Madrid DevOps’ September meeting
• https://vimeo.com/76334136
• https://vimeo.com/76334134
• Vagrant: Up and Running, Mitchell Hashimoto
http://www.amazon.com/Vagrant-Up-Running-Mitchell-
Hashimoto/dp/1449335837/ref=sr_1_1?ie=UTF8&qid=1375433466&sr=8-
1&keywords=vagrant+up
• http://juanvicenteherrera.eu/2013/09/04/immutable-servers-with-vagrant-
and-chef-chapter-i/
• https://github.com/mitchellh/vagrant-aws
• http://docs.vagrantup.com/v2/cli/index.html
• http://red-badger.com/blog/2013/02/21/automating-your-infrastructure-with-
vagrant-chef-from-development-to-the-cloud/
• http://docs-v1.vagrantup.com/v1/docs/provisioners/chef_server.html
27. irc channel #madrid-devops.
Web https://sites.google.com/site/madridevops/
Twitter https://twitter.com/MadridDevops
Grupo de Google
https://groups.google.com/d/forum/madriddevops
GitHub https://github.com/madrid-devops
Group information