This is the talk I gave at Config Management Camp 2016 in Ghent introducing Jerakia as a lookup tool that can be used in place of, or along side of hiera to solve some of the edge cases around data separation

1. Solving real world data
problems with Jerakia
Craig Dunn, Config Management Camp, Ghent 2016

2. • Best practice
• Code base design
• Workflow mangement
• Scaling Puppet
• Installation and support
• Module writing
• Throughout Europe
www.enviatics.com

3. • Puppet user since 2008
• IT consultant for 15+ years
• Active community member
• The “Roles and Profiles” guy
• Problem solver
• Lives in Málaga, Spain.
• …. and hotels
• Daddy!
www.craigdunn.org
Craig Dunn
@crayfishx

4. A brief history of Puppet

5. In the beginning…
• Over complex code
• Unsharable modules
• Making simple changes required alot of skill.
The embedded data era

6. class ntp {
if $env == ‘dev’ {
$server = ‘dev.ntp.local’
} else {
if $hostname == ‘gateway’ {
$server = ‘pool.ntp.org’
} else {
$server = ‘prod.ntp.local’
}
}
…
}

7. And then…

8. Hiera
The dawn of the data separation era

9. • Separation of data from code
• Module authors could write sharable re-usable code
• Code was less complex and more readable
• The Forge became useful
• Managing data became a lot easier

10. Hierarchical Search

11. Pluggable
• Pluggable interchangable backends
• Data can be sourced from multiple formats
• hiera-eyaml
• hiera-mysql
• hiera-http
• hiera-redis
• hiera-consul

13. Managing our data is now
a critical part of configuration management

14. Infrastructure grows and requirements
get more complex

16. • Different teams and customers require different hierarchies
• A particular application needs to source data from a different place
• Control access to sub-sets of data for teams within an organisation
• Dynamically generate the lookup hierarchy at runtime
• Group together application specific data into separate files
• Manage encrypted data from any data source
• Global hiera.yaml file creates restrictions

17. Introducing Jerakia
jerakia.io

18. Jerakia
• Data lookup tool
• Open source
• Extendable framework
• Solving the most complex edge cases

19. Jerakia
• Can be used as a Hiera backend
• Can be wired directly into Puppet as a data binding terminus
• Drop in replacement for Hiera, or not.

20. Why Jerakia?

21. One design goal…

22. Flexibility

23. • Lookup behaviour written in Ruby DSL
• Almost everything is pluggable
• Inter-changable data sources
• Easy integration
• Hiera compatible*

24. $ gem install jerakia

25. $ puppet module install crayfishx/jerakia

26. • A request is received containing a key and a namespace
• A policy is chosen to perform the request
• One or more lookups are called to act on the request
• A response is sent back to the requestor
• Container for lookups
• Written in Ruby DSL
• Different policies for different apps
Policy File

27. An Example Jerakia Policy File
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end

28. An Example Jerakia Policy File
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end

29. An Example Jerakia Policy File
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end

30. An Example Jerakia Policy File
policy :main do
lookup :default do
datasource :file, {
:docroot => "/var/jerakia/data",
:format => :yaml,
:searchpath => [
"host/#{scope[:hostname]}",
"env/#{scope[:env]}",
"common",
]
}
end
end

31. • Lookups are contained within policies
• A policy can contain multiple lookups
• A lookup always contains at least a data source
Lookups

32. Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup

33. Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup
Request consists of a
lookup key, a namespace
and some metadata

34. Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup
Information to be
used in determining
how data is looked up

35. Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup
Lookup plugins can read
and modify the scope and
request objects

36. Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup
A pluggable data source is
used to lookup data

37. Scope
Handler
Request
Lookup
Plugins
Data Source
Output Filter
Response Data
Anatomy of a Jerakia lookup
Data returned from the
datasource is passed to a
pluggable output filter

38. Lookup methods

39. confine / exclude
Invalidates a lookup unless/if the criteria is met
confine request.namespsace[0], "apache"
confine request.namespsace[0], [
/website_.*/,
"apache",
"php"
]

40. Stop
Do not proceed to the next lookup if this lookup is valid
lookup :special do
…
confine request.namespsace[0], "apache"
stop
end
lookup :main do
…

41. Datasources
• Easily pluggable and extendable
• File and HTTP datasources shipped out-of-the-box

42. Datasources
datasource :name, { :option => “value”… }

43. Datasource definition
lookup :main do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"host/#{scope[:certname]}",
"env/#{scope[:environment]}",
"common",
]
}
end
/var/lib/jerakia/env/dev/apache.yaml

44. lookup :main do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"host/#{scope[:certname]}",
"env/#{scope[:environment]}",
"common",
]
}
end
/var/lib/jerakia/env/dev/apache.yaml
Datasource definition

45. lookup :main do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"host/#{scope[:certname]}",
"env/#{scope[:environment]}",
"common",
]
}
end
/var/lib/jerakia/env/dev/apache.yaml
Datasource definition

46. /var/lib/jerakia/env/dev/apache.yaml
/var/lib/jerakia/env/dev/apache.d/www_corp_com.yaml
/var/lib/jerakia/env/dev/apache.d/www_acme_net.yaml
/var/lib/jerakia/env/dev/apache.d/www_fake_org.yaml
Fragments
• Introduced in 0.4
• If a .d directory is found, files within are
concatenated
• One document is returned

47. Data Layout
:searchpath => [
"host/#{scope[:certname]}",
"env/#{scope[:environment]}",
]
# cat /var/lib/jerakia/env/dev/apache.yaml
—-
port: 80
# cat /var/lib/jerakia/env/dev.yaml
—-
apache::port: 80
Hiera
Jerakia

48. Plugins
• Access to request and scope
• Can read or modify on-the-fly
• Re-usable
• Cleaner code in policy files

49. class Jerakia::Lookup::Plugin
module Mything
def do_something
…
end
end
end
Writing plugins
• Written as Ruby extensions
• Can be placed in the plugin dir
• Or shipped as rubygems

50. lookup :main, :use => :mything do
plugin.mything.do_something
…
end
Using plugins
• Plugins are loaded into the lookup
• Referenced as plugin.name.method
lookup :main, :use => [ :mything, :foo ] do
…
end

51. lookup :main, :use => :hiera do
plugin.hiera.rewrite_lookup
datasource :file, {
:docroot => "/var/lib/jerakia",
:format => :yaml,
:searchpath => [
"env/#{scope[:environment]}",
"common",
]
end
The hiera plugin
• Provides compatibility to hiera filesystem layouts
• Shipped with Jerakia
# cat /var/lib/jerakia/env/dev.yaml
—-
apache::port: 80

52. Output filters
• Pluggable
• Specified in the lookup
• Parses data returned from the datasource

53. Output filters
• Two are currently shipped
• Encryption (provided by eyaml*)
• Strsub
*https://github.com/TomPoulton/hiera-eyaml

54. Output filters
lookup :main do
…
output_handler :encryption
end

55. Output filters
lookup :main do
…
output_handler :encryption
end

56. Example User Story
• Team in Ireland manage PHP/Apache
• Autonomous team that don’t manage infra
• Their optimal hierarchy is different from “ours”
• “We” need to service them from Puppet
• They must not modify infra services
• “We” also manage PHP/Apache for other clients

57. policy :default do
lookup :main, do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"hostname/#{scope[:fqdn]}",
"environment/#{scope[:environment]}",
"common"
],
}
end
end
Our main lookup is
responsible for the entire
infrastructure

58. policy :default do
lookup :ireland do
datasource :file, {
:format => :yaml,
:docroot => "/var/external/data/ie",
:searchpath => [
"project/#{scope[:project]}",
"common",
]
}
end
lookup :main, do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"hostname/#{scope[:fqdn]}",
"environment/#{scope[:environment]}",
"common"
],
}
end
end
Lookup for the Ireland
team added above the
main lookup with
separate docroot and
searchpath

59. policy :default do
lookup :ireland do
datasource :file, {
:format => :yaml,
:docroot => "/var/external/data/ie",
:searchpath => [
"project/#{scope[:project]}",
"common",
]
}
confine scope[:location], "ie"
confine request.namespace[0], [
"apache",
"php",
]
end
lookup :main, do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"hostname/#{scope[:fqdn]}",
"environment/#{scope[:environment]}",
"common"
],
}
end
end
Only use this lookup if the
requestor location is IE
and the namespace is
apache or php

60. policy :default do
lookup :ireland do
datasource :file, {
:format => :yaml,
:docroot => "/var/external/data/ie",
:searchpath => [
"project/#{scope[:project]}",
"common",
]
}
confine scope[:location], "ie"
confine request.namespace[0], [
"apache",
"php",
]
stop
end
lookup :main, do
datasource :file, {
:format => :yaml,
:docroot => "/var/lib/jerakia",
:searchpath => [
"hostname/#{scope[:fqdn]}",
"environment/#{scope[:environment]}",
"common"
],
}
end
end
If this lookup is valid then
do not proceed to the
main lookup, even if data
is not found.

61. Command line
$ jerakia lookup port —namespace apache
$ jerakia help lookup
Usage:
jerakia lookup [KEY]
Options:
c, [--config=CONFIG] # Configuration file
p, [--policy=POLICY] # Lookup policy
# Default: default
n, [--namespace=NAMESPACE] # Lookup namespace
t, [--type=TYPE] # Lookup type
# Default: first
s, [--scope=SCOPE] # Scope handler
# Default: metadata
[--scope-options=key:value] # Key/value pairs to be passed to the scope handler
m, [--merge-type=MERGE_TYPE] # Merge type
# Default: array
l, [--log-level=LOG_LEVEL] # Log level
v, [--verbose], [--no-verbose] # Print verbose information
D, [--debug], [--no-debug] # Debug information to console, implies --log-level debug
d, [--metadata=key:value] # Key/value pairs to be used as metadata for the lookup
Lookup [KEY] with Jerakia

62. Integration with Puppet
—-
:backends:
- jerakia
[master]
. . .
data_binding_terminus = jerakia

63. Roadmap &
Contributing

64. Upcoming in 0.5
• Data Schemas
• Better REST client/server
• Deep merge behaviour
• Lookup plugin “load method”

65. Contributions wanted
• Code maturity
• Caching
• Features
• Bugfixes
• Documentation
• #jerakia (freenode) Sponsored by

66. Jerakia 1.0

67. Thank you
Questions?
jerakia.io
@crayfishx