PuppetDB: One Year Faster - Deepak Giridharagopal, Puppet Labs

1. PuppetDB
(ﾉ°ヮ°)ﾉ*:･ﾟ✧2014
✧

2. RUN
PDB

3. deepak giridharagopal
deepak @puppetlabs.com
@grim_radical

4. Puppet
agent
Puppet
master
facts
PuppetDBPuppetDB

5. Puppet
agent
Puppet
master
facts
PuppetDBPuppetDB

6. Puppet
agent
Puppet
master PuppetDBfacts
PuppetDB

7. Puppet
agent
Puppet
master
facts
PuppetDB
PuppetDB

8. Puppet
agent
Puppet
master
facts
PuppetDBPuppetDB
Yum!

9. Puppet
agent
Puppet
master PuppetDBPuppetDB

10. Puppet
agent
Puppet
master PuppetDBPuppetDB
catalogcatalog

11. catalog
Puppet
agent
Puppet
master PuppetDBPuppetDB
catalog

12. catalog
Puppet
agent
Puppet
master
PuppetDB
PuppetDB
catalog

13. Puppet
agent
catalog
Puppet
master
facts
PuppetDBPuppetDB
Yum!

14. Puppet
agent
Puppet
master PuppetDBPuppetDB
catalog

15. Puppet
agent
Puppet
master PuppetDBPuppetDB
report

16. Puppet
agent
Puppet
master PuppetDBPuppetDB
report

17. report
Puppet
agent
Puppet
master PuppetDBPuppetDB

18. report
Puppet
agent
Puppet
master
PuppetDB
PuppetDB

19. Puppet
agent
Puppet
master
facts
PuppetDBPuppetDB
Yum!

20. Puppet
agent
Puppet
master PuppetDBPuppetDB

22. Puppet
master
catalog
PuppetDB
catalog

23. Puppet
master
catalog
PuppetDB
catalog

24. Puppet
master PuppetDB
catalog

25. catalog
Puppet
master PuppetDB
catalog

26. Puppet
master
PuppetDcatalog

27. catalog
Puppet
master
PuppetDcatalog

28. Puppet
master
Pupcatalog

29. catalog
Puppet
master
Pupcatalog

30. Software should be
self-regulating

31. goo.comfoo.com bar.com
STORAGE
bar.com
Self-regulating catalog & fact storage!

32. goo.comfoo.com bar.com
STORAGE
bar.combar.combar.combar.com
Self-regulating report storage!

33. goo.comfoo.com bar.com
STORAGE
Self-regulating node storage!

34. goo.comfoo.com bar.com
STORAGE
bar.com
Deduplication of catalogs!
foo.com goo.com

35. /commands MQ Parse
Delayed
Dead Letter
Office
Process
UUID

36. Querying

37. File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
baz.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
goo.com
/v4/resources

38. File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
baz.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
goo.com
/v4/resources/Service

39. File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
baz.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
goo.com
/v4/resources/Service/foo

40. File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
baz.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
goo.com
/v4/nodes/foo.com/resources

41. File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
baz.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
goo.com
/v4/nodes/foo.com/resources
/Package

42. File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
baz.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
goo.com
/v4/nodes/foo.com/resources
/Package/foo

43. • Can query facts, nodes,
resources, reports, events,
metrics
• Advanced queries via AST-
based language
• Aggregates, ordering, paging,
streaming, subqueries

45. A long time ago in a galaxy
far, far away…
!
!
uh, like here a year ago…

46. Just released 1.4.0,
Around 11k deployments,
Basic streaming support,
Other stuff…

47. We’ve been busy!

48. Soft write failures
Paging support
Resource containment paths
in events
Event aggregates
Differential fact storage

49. Differential edge storage
Pervasive streaming of query
results
3dfx Voodoo2 and
Soundblaster AWE32 support
Improved de-duplication

50. Resource parameter caching
PostgreSQL Hot Standby
support for faster reads
Debugging of de-duplication
algorithm
Full director’s commentary

51. Compressed responses
A pony
Certificate chain support
Support for puppet
environments
Event subqueries

52. Prepared statement caching
Direct POST of json data in
terminus
Brings Aeris from Final
Fantasy VII back to life
Profiling support for terminus

53. Faster message parsing
Structured fact storage and
querying
Unified query subsystem in v4
API
Riker’s beard

54. docs.puppetlabs.com/
puppetdb/latest/
release_notes.html

55. We can’t get through
that entire list, but I’ll try
to highlight a few shiny
bits

56. 1.
Differential storage

57. goo.comfoo.com bar.com
STORAGE
bar.comfoo.com

58. bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[zzz]
Exec[echo foo]
Exec[echo bar]

59. STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[zzz]
Exec[echo foo]
Exec[echo bar]

60. STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[zzz]
Exec[echo foo]
Exec[echo bar]

61. STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[zzz]
Exec[echo foo]
Exec[echo bar]

62. STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
bar.com
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[zzz]
Exec[echo foo]
Exec[echo bar]

63. • Edges and facts, too
• Trades reads for writes,
which is a good tradeoff.
• PostgreSQL’s heap-only-
tuples help a lot.

64. ~90%fewer writes

66. Thanks to the folks at
Spotify, the community,
etc. that helped us with
this!

67. 2.
More effective
de-duplication

68. Order matters!

69. {"foo" => "goo",
"bar" => "baz"}
{"bar" => "baz",
"foo" => "goo"}
904d4d…
11c05d…

70. • Restructuring data prior to
hashing results in much
fewer false negatives
• The fastest way to persist
data is to already have it
persisted!

71. ~60-70%boost for
users with ordinarily
low dedupe rates

72. Thanks to the folks at
CERN that helped us
debug this!

73. 3.
Hot standby

74. PuppetDB PostgreSQL
WRITE
READ

75. PuppetDB
PostgreSQL
WRITE
READ
PostgreSQL
standby
REPLICATION

76. Less I/O contention for
reads and writes yields
better throughput

77. 3.
Environment support

78. STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TEST
bar.com
PROD

79. STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TEST
bar.com
PROD

80. STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TEST
bar.com
PROD
All Files,
please!
Anything for
you! !

81. STORAGE
File[/foo]
File[/bar] Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TEST
bar.com
PROD
WTF?!
I’m trying my
best! !

82. STORAGE
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TEST
bar.com
PROD
All Files for
env TEST, plz!
Anything for
you! !
File[/foo]

83. STORAGE
File[/foo]
File[/bar]
Package[foo]
Package[bar]
Service[foo]
Service[bar]
Exec[echo foo]
Exec[echo bar]
foo.com
TEST
bar.com
PROD
Thanks!
We’re friends
again! ❤️

84. • Any reception or
transmission of data now
includes the environment
where possible
• Queries can be isolated to a
single environment

85. 3.
Unified query engine

86. PuppetDB
Parse
Map valid
fields
Map valid
operators
Compile to
SQL
PostgreSQL
Parse
Map valid
fields
Map valid
operators
Compile to
SQL
Parse
Map valid
fields
Map valid
operators
Compile to
SQL
Parse
Map valid
fields
Map valid
operators
Compile to
SQL
Facts
Resources
Nodes
Reports
Parse
Map valid
fields
Map valid
operators
Compile to
SQL
Events
QUERY

87. PuppetDB
Parse to AST
Term
rewriting
Apply
operators
Compile to
SQL
PostgreSQL
QUERY

88. • Common query engine
underpinning v4 API
• Operators are available
uniformly across all v4
endpoints
• We can add new endpoints
and fields much faster

89. 4.
Structured/Trusted
fact support

90. {
"cpus" : {
"cpu1" : {
"bogomips": 6000,
}
},
"networking" : {
"eth0" : {
"ipaddresses" : [ "1.1.1.5" ],
"macaddresses" : [ "aa:bb:cc:dd:ee:00" ]
}
}
}

91. ["=", "path",
["networking",
"eth0",
"macaddresses",
0]]

92. ["~>", "path",
["networking",
"eth.*",
"macaddresses",
".*"]]

93. !
• PostgreSQL’s pg_trgm index
• Trusted facts

94. So where are we now?

95. More features, but also
more speed

96. Language bindings:
Ruby, Python,
JavaScript, C/C++, Go,
Clojure, Java…

97. Trapperkeeper
https://github.com/puppetlabs/trapperkeeper

98. Puppetboard
https://github.com/nedap/puppetboard

99. Puppet Explorer
https://github.com/spotify/puppetexplorer

100. A new deployment
every 10 minutes

101. Coming soon:
more efficient GC

102. Coming soon:
simplified query syntax

103. Coming soon:
historical data

104. Thanks!