1. Proprietary and
Chef on SmartOS
Eric Saxby
@sax @ecdysone @sax

2. Who am I?
Proprietary and
■ Application developer
operational experience with many
technologies, project by project
■ BSD/AIX/Ubuntu
Solaris in 2002, but I was very much
out of my element
■ Switched to DevOps-y team 18 months ago
Multiple back end services for a large e-commerce site,
transitioning to SmartOS
■ Now I’m at Wanelo

3. From a certain point of view...
Proprietary and

4. From a certain point of view...
Proprietary and

5. What is Wanelo?
Proprietary and
■ Wanelo (“Wah-nee-lo” from Want, Need
Love) is a global platform for shopping.

6. Proprietary and
Marketing-free shopping across 100s of
thousands of unique stores

7. Proprietary and
Personal feed of products from any store on
the internet

8. Technology overview
Proprietary and
■ MRI Ruby 1.9.3 & Rails 3.2
■ PostgreSQL 9.2.4, Solr 3.6
■ Joyent Cloud, SmartOS
ZFS, ARC, raw IO performance, SmartOS, CPU bursting, dTrace
■ Circonus, Chef + Opscode
Monitoring, graphing, alerting, automation
■ Amazon S3 + Fastly CDN
■ NewRelic, statsd, Graphite, nagios

9. What’s SmartOS?
Proprietary and
■ Illumos branch optimized for cloud
computing
■ Developed by Joyent for their public
cloud

10. What’s Illumos?
Proprietary and
■ It’s what OpenSolaris became after Oracle
killed the project
■ Umbrella for various distributions, each
committed to pushing their improvements
upstream
■ http://wiki.illumos.org/display/illumos/About+illumos

11. What does SmartOS look like?
Proprietary and
■ Compute Node — physical server
■ Global Zone — host OS (SmartOS)
■ Non-Global Zone — like a virtual machine, with
native system calls (no fake hardware layer)
■ Very secure
■ Can run KVM for guest OS (Ubuntu, Centos)

12. How is it deployed?
Proprietary and
■ Can manage from global zone (imgadm,
zoneadm)
■ Tools provide APIs
■ Smart Data Center (Joyent’s tools, can be licensed)
■ Project FIFO (SDC API in free package)
■ Joyent Public Cloud
■ Many compute nodes working in a cluster,
PXE booted from a head node

13. Why should I care?
Proprietary and

14. Why should I care?
Proprietary and
■ ZFS
File system built for speed and data integrity

15. Why should I care?
Proprietary and
■ Visibility tools
dtrace, kstat, snoop, truss
■ ZFS
File system built for speed and data integrity

16. ■ Service Management Facility (SMF)
If init.d and monit and god were one thing, and
actually awesome
Why should I care?
Proprietary and
■ Visibility tools
dtrace, kstat, snoop, truss
■ ZFS
File system built for speed and data integrity

17. ■ Service Management Facility (SMF)
If init.d and monit and god were one thing, and
actually awesome
Why should I care?
Proprietary and
■ Visibility tools
dtrace, kstat, snoop, truss
■ ZFS
File system built for speed and data integrity
■ Application Latency
Zones are OS virtualization, so faster
Processes are scheduled in global zone kernel,
not in a hardware virtualization layer

18. ■ Service Management Facility (SMF)
If init.d and monit and god were one thing, and
actually awesome
Why should I care?
Proprietary and
■ Visibility tools
dtrace, kstat, snoop, truss
■ ZFS
File system built for speed and data integrity
■ Application Latency
Zones are OS virtualization, so faster
Processes are scheduled in global zone kernel,
not in a hardware virtualization layer

19. ■ Service Management Facility (SMF)
If init.d and monit and god were one thing, and
actually awesome
Why should I care?
Proprietary and
■ Visibility tools
dtrace, kstat, snoop, truss
■ ZFS
File system built for speed and data integrity
■ Application Latency
Zones are OS virtualization, so faster
Processes are scheduled in global zone kernel,
not in a hardware virtualization layer

20. Lower latency == less cost
Proprietary and

21. Lower latency == less cost
Proprietary and
■ Requests/second of single process =~
request latency

22. Lower latency == less cost
Proprietary and
■ # processes required =~
requests/second of site
■ Requests/second of single process =~
request latency

23. ■ # cores, RAM required =~
# processes
Lower latency == less cost
Proprietary and
■ # processes required =~
requests/second of site
■ Requests/second of single process =~
request latency

24. ■ # cores, RAM required =~
# processes
Lower latency == less cost
Proprietary and
■ # processes required =~
requests/second of site
■ Requests/second of single process =~
request latency
$$$

25. On to Chef!
Proprietary and

26. Terminology
Proprietary and
■ Image / Dataset — OS at a particular version,
snapshotted at base state
■ Flavor / Package— RAM, CPU shares
■ API URL — Each data center has its own URL
■ Server ID / Zonename — Each zone gets a
UUID

27. knife-joyent
Proprietary and

28. Installation/Configuration
Proprietary and
■ Update knife.rb
■ Add to Gemfile
knife[:joyent_username] = 'sax'
knife[:joyent_keyname] = 'EricSaxby'
knife[:joyent_keyfile] = "#{ENV['HOME']}/.ssh/id_rsa"
knife[:joyent_api_url] = 'https://us-sw-1.api.joyentcloud.com/'
■ Add first public key in cloud API
https://my.joyentcloud.com
gem 'knife-joyent'

29. Managing keys
Proprietary and
■ No role based access, but at least you can
make each user upload their own key
knife joyent key add -f ~/.ssh/id_rsa -k KeyName
knife joyent key delete KeyName
■ Passphrase protected keys are annoying
Each API request includes data signed with the private
key. Ruby does not have a good way of signing private
keys with ssh-agent.

30. Creating servers!
Proprietary and
■ See what images are available
knife joyent image list
cf7e2f40-9276-11e2-af9a-0bad2233fb0b base64 1.9.1 smartos
f4bc70ca-5e2c-11e1-8380-fb28785857cb smartosplus64 3.1.0 smartos
da144ada-a558-11e2-8762-538b60994628 ubuntu-12.04 2.4.1 linux
■ base / base64 — minimal install, you add what
you need
■ smartosplus — many more things pre-
installed, but can get in the way
13328c9a-9173-11e2-a9a5-2ff43d306c21 ws2008ent-r2-sp1 2.0.2 windows

31. Creating servers!
Proprietary and
■ See what flavors are available
knife joyent flavor list
Name RAM Disk Swap
Extra Small 512 MB 0 GB 15 GB 1 GB
Small 1GB 1 GB 30 GB 2 GB
Medium 2GB 2 GB 60 GB 4 GB
Medium 4GB 4 GB 120 GB 8 GB
Large 8GB 8 GB 240 GB 16 GB
Large 16GB 16 GB 480 GB 32 GB
■ Custom networking can be done in a custom
flavor (ie public or private VLAN, routes)

32. Creating servers already!
Proprietary and
knife joyent server create
--image cf7e2f40-9276-11e2-af9a-0bad2233fb0b
--flavor 'Medium 2GB'
-N server.domain.com
-E environment
-d distro
-r run_list
■ No Omnibus, so you have to provide your own
distro bootstrap template
https://gist.github.com/sax/5457464

33. knife joyent server list
See what's there...
Proprietary and
a597a3a7-3fdf-481f-af08-e7c1e0ae7dca admin.prod running smartmachine
sdc:sdc:base64:1.8.1 8.19.1.1 10.100.1.1 8 GB 240 GB
5c066e6e-8af2-4d4f-a81e-c8e2691ae8a0 demo.dev running smartmachine
sdc:sdc:base64:1.8.1 10.12.1.1 165.225.1.1 8 GB 240 GB
b3370d52-3bed-462e-857a-e17eba15ab06 app010.c1.prod running smartmachine
sdc:sdc:base64:1.8.1 10.100.1.2 165.225.1.2 8 GB 240 GB
■ ID / zonename
■ Name
■ Run state
■ Type
■ Image
■ IP addresses
■ RAM
■ Disk

34. Other management
Proprietary and
knife joyent server delete <server_id>
knife joyent server start <server_id>
knife joyent server stop <server_id>
knife joyent server reboot <server_id>
knife joyent server resize <server_id> -f <flavor>
knife joyent snapshot create <server_id> <snapshot_name>
■ Snapshots are full ZFS snapshots
Copy-on-write snapshot of local file system.
Each snapshot is locally mounted in zone at
/checkpoints

35. So now you have a
smartmachine...
Proprietary and

36. What's different?
Proprietary and
■ Things you expect in /usr/local are in /opt/local
■ For historical reasons
■ If you're used to Linux, this can be annoying
■ Joyent is working on a more Linux friendly image
■ For now, add /opt/local/bin to PATH
■ Many configs are in /opt/local/etc instead of /etc
■ Some utilities are different
■ This is not the grep you're looking for....
■ Symlink your "correct" version into /opt/local/bin
■ Add /opt/local/lib to CFLAGS and LDFLAGS

37. Caveats?
Proprietary and
■ Zones inside of zones inside of...
■ Vagrant does not currently work with SmartOS
■ VirtualBox only works in Bridged network mode
■ Local integration tests do not work

38. Where are all the things?
Proprietary and
■ Services
■ svcs -a
■ svcadm < enable | disable | clear > service
■ Packages
■ pkgin search packagename
■ pkgin -y install packagename

39. Public vs. Private IP
Proprietary and
■ ipaddr_extensions gem
■ Adds 'privateaddress' attribute to ohai
■ Useful to add this to bootstrap
■ Smartmachines may have a public IP and a
private IP
■ Recipes can be configured to use ipaddress or
privateaddress

40. System preparation
Proprietary and
■ smartos cookbook
■ https://github.com/modcloth-cookbooks/smartos
■ fixes chef providers
■ smartmachine_functions
■ links nicer utils into /opt/local/bin
■ https://github.com/higanworks-cookbooks/
smartmachine_functions
■ fixes chef providers
■ provides access to Joyent metadata API
or

41. Useful LWRPs
Proprietary and

42. SMF
Proprietary and
■ https://github.com/modcloth-cookbooks/smf
■ Chef knows how to use SMF, not how to configure it
■ Uses nokogiri, which requires libxml2
smf 'postgres' do
user 'postgres'
group 'postgres'
project 'postgres'
start_command 'postgres-service.sh start'
stop_command 'postgres-service.sh stop'
working_directory '/var/pgsql/data'
environment 'PATH' => '/opt/postgres/bin'
end

43. SMF (cnt'd)
Proprietary and
smf 'postgres' do
user 'postgres'
group 'postgres'
project 'postgres'
start_command 'postgres-service.sh start'
stop_command 'postgres-service.sh stop'
stop_timeout 120
restart_command 'postgres-service.sh restart'
refresh_command 'postgres-service.sh reload'
working_directory '/var/pgsql/data'
environment 'PATH' => '/opt/postgres/bin'
end
service 'postgres' do
supports :status => true,
:restart => true, :reload => true
end

44. SMF (cnt'd)
Proprietary and
smf 'postgres' do
user 'postgres'
group 'postgres'
project 'postgres'
start_command 'postgres-service.sh start'
stop_command 'postgres-service.sh stop'
stop_timeout 120
restart_command 'postgres-service.sh restart'
refresh_command 'postgres-service.sh reload'
working_directory '/var/pgsql/data'
environment 'PATH' => '/opt/postgres/bin'
end
service 'postgres' do
supports :status => true,
:restart => true, :reload => true
end

45. Resource Control / Projects
Proprietary and
■ https://github.com/wanelo-chef/resource-control
■ configure max file descriptors, shared memory, etc
■ Bunch up master/worker processes to view in
prstat -J
resource_control_project "postgres" do
comment "PostgreSQL 9.2"
users "postgres"
project_limits "max-shm-memory" => 12000000,
"max-lwps" => 6
process_limits "max-file-descriptor" => {
"value" => 32768, "deny" => true
}
action :create
end

46. Role Based Access Control
Proprietary and
■ https://github.com/modcloth-cookbooks/rbac
■ Allows delegation of authority without sudo
■ Implementation currently too simple, only useful for SMF
delegation
rbac 'solr' do
user 'wanelo'
action :add_management_permissions
end

47. Contributing to cookbooks
Proprietary and
■ ~95% just require SMF, correct package names
■ ~5% of those need a special init script
■ The rest usually require custom compile
`postgres -D /path/to/data` not granular enough
`pg_ctl -D /path/to/data < start | stop | reload | refresh >`
--with-libraries=/opt/local/lib
--with-includes=/opt/local/include
LDFLAGS='-R/opt/local/lib -L/opt/local/lib'

48. Comments? Questions? Find me.
https://github.com/wanelo
https://github.com/wanelo-chef
https://github.com/wanelo-chef/smartos-chef-repo
Proprietary and
@sax @ecdysone @sax