What is Metadata and how it is represented in Chef and Puppet? What are the challenges for metadata sharing across providers and possible solutions.

1. A metadata ocean in
Puppet and Chef
FOSDEM’14
Marc Cluet
DevOps & Automation Advisory
Rackspace
http://www.flickr.com/photos/elias_daniel/7470265444/

2. Who am I?!
Marc Cluet (@lynxman)
More than 16 years of experience as a
SysAdmin
Founding member of Juju and MAAS
while at Canonical
Leading a DevOps Engineering Team at
Rackspace
Likes DevOps, programming and long
walks on the beach
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

3. What is Rackspace?!
Founded in 1998 in San Antonio, TX by
three guys that wanted to create a hosting
company
Home of Fanatical Support /o/
Second biggest Public Cloud in the world
OpenStack Project co-founder
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

4. What is Metadata?!
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
/http://www.flickr.com/photos/boojee/3743753784/

5. What is Metadata?!
Metadata is “data about data”!
Structural Metadata!
Descriptive Metadata!
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
http://www.flickr.com/photos/ggvaidya/8256283184/

6. Metadata in Chef!
Metadata is compiled on the client
at execution time!
Metadata is always stored in its
entirety in the node[] dictionary!
$node[var] = val
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

7. Metadata in Chef!
You can add metadata in your!
Attribute File!
Node/Recipe!
Environment!
Role!
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

8. Organisation in Chef!
http://docs.opscode.com/essentials_environments.html
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

9. Metadata in Puppet!
Metadata is always compiled on
execution time
Metadata vars can’t be reassigned!
$::var = val
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

10. Metadata in Puppet!
You can add metadata in your!
!Class vars!
!Nodes!
!Top Scope!
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

11. Scoping in Puppet!
http://docs.puppetlabs.com/puppet/latest/reference/lang_scope.html
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

12. Hiera in Puppet!
Solves metadata fragmentation issues!
Defines hierarchy for easy organisation!
All in clean and properly styled in either
YAML or JSON!
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

13. Hiera in Puppet!
:backends:
- yaml
- json
:yaml:
:datadir: /etc/puppet/hieradata
:json:
:datadir: /etc/puppet/hieradata
:hierarchy:
- "%{::clientcert}"
- "%{::custom_location}"
- common
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

14. RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
http://www.flickr.com/photos/musebrarian/3289649684/

15. Bad metadata practices!
In Chef!
Not keeping consistency between
cookbook scope and environment
scope!
Not removing metadata from nodes
when deprecated!
Directly assigning metadata in a
cookbook without a default attribute!
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

16. Bad metadata practices!
In Puppet!
Directly assigning metadata in a class
without a default value!
Assigning vars in nodes.pp!
Not assigning global vars in site.pp!
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

17. RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
http://www.flickr.com/photos/deepstereo/9616389940/

18. Good behaviours!
Separate vars from execution!
Do not store vars on the execution
area (classes/cookbooks)!
High abstraction, good hierarchy!
Metadata is all about context!
!
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

19. Good behaviours!
Global
Environment
Node
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

20. RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
http://www.flickr.com/photos/monica_andre/3119991125/

21. Metadata Across Providers!
Provisioner
Config Mgmt
cloud-init
Puppet
$dog = cute
$::dog = big
WHO IS RIGHT?!?!
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

22. Metadata Across Providers!
RACKSPACE® HOSTING | WWW.RACKSPACE.CO.UK
http://www.flickr.com/photos/pss/233841567/

23. Hard Problem to Solve!
Metadata brain split problem!
Gets worse by adding more players!
System view varies!
Not same info from bios than from a
user level program as facter or ohai!
!
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

24. Possible Solutions!
Central DB using something fast!
!Redis, mongoDB, mySQL!
Metadata translators might fix it?!
!Write for everything!
Creates new SPoF!
!Hard to solve without throwing $$!
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

25. Hard Problem to Solve!
http://xkcd.com/927/
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

26. Possible Solutions!
Scope vars from top of chain!
Pass them over as ext facts!
$dog in cloud-init becomes $::dog in
puppet!
RACKSPACE® HOSTING
|
WWW.RACKSPACE.CO.UK

27. @lynxman!
http://slideshare.net/lynxmanuk/!
http://www.rackspace.co.uk/technicalservices/ !
!
RACKSPACE® HOSTING
|
UK SALES: +44 (0)20 8712 6507
RACKSPACE® HOSTING
|
© RACKSPACE US, INC.
|
5 MILLINGTON ROAD
|
HAYES, UNITED KINGDOM UB3 4AZ
|
UK SUPPORT: 0800 988 0300
|
WWW.RACKSPACE.CO.UK
RACKSPACE® AND FANATICAL SUPPORT® ARE SERVICE MARKS OF RACKSPACE US, INC. REGISTERED IN THE UNITED STATES AND OTHER COUNTRIES.
|
WWW.RACKSPACE.CO.UK
27