SaltStack is fast, among other things. At it's core, SaltStack is a real-time infrastructure management tool that utilizes a high-speed communication channel for remote execution and configuration management. Proper systems administration is not possible without both. This presentation will demonstrate how to create real-time interaction between your code and your infrastructure using SaltStack.

1. Real-time infrastructure management with Salt
Seth House <seth@eseth.com>
OpenWest Conference 2013
2013-05-03
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 1 / 34

2. Salt internals (briefly)
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 2 / 34

3. Salt internals (briefly)
Outline
1 Salt internals (briefly)
2 Salt speed
3 Minion data
4 Events
5 Schedules
6 What’s coming
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 2 / 34

4. Salt internals (briefly)
Salt internals
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 3 / 34

5. Salt internals (briefly)
Master
salt-master -d
Open two ports (pub/sub & reply channel)
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 4 / 34

6. Salt internals (briefly)
Minions
salt-minion -d
Connect to the master
No open ports required
Listens for pubs from the master
/etc/salt/minion:
#master: salt
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 5 / 34

7. Salt internals (briefly)
Execution modules
Contain all the functionality
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 6 / 34

8. Salt internals (briefly)
Execution example
salt ’web-*’ network.interfaces
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 7 / 34

9. Salt internals (briefly)
State modules
Wrap execution modules
Before-check
test=true
Call out to execution modules
After-check
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 8 / 34

10. Salt internals (briefly)
State module example
top.sls:
base:
’web-*’:
- httpd
httpd.sls:
httpd:
pkg:
- installed
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 9 / 34

11. Salt speed
Outline
1 Salt internals (briefly)
2 Salt speed
3 Minion data
4 Events
5 Schedules
6 What’s coming
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 10 / 34

12. Salt speed
Why is Salt fast?
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 11 / 34

13. Salt speed
Communication
ZeroMQ
msgpack
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 12 / 34

14. Salt speed
pub/sub
Asynchronous
Minions determine targeting match
Minions do all the work
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 13 / 34

15. Minion data
Outline
1 Salt internals (briefly)
2 Salt speed
3 Minion data
4 Events
5 Schedules
6 What’s coming
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 14 / 34

16. Minion data
Sharing minion data
<-- Live -- Recent -- Historic -->
peer interface Salt Mine Returners
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 15 / 34

17. Minion data
Peer
/etc/salt/master:
peer:
lb-.*:
- network.interfaces
Be mindful of data security
Communication still goes through the master
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 16 / 34

18. Minion data
Peer example
Configuring haproxy.cfg:
{% for server,ip in
salt[’publish.publish’](
’web*’,
’network.interfaces’,
[’eth0’]).items() %}
server {{ server }} {{ ip[0] }}:80 check
{% endfor %}
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 17 / 34

19. Minion data
Salt Mine
/etc/salt/{master,minion}:
mine_functions:
network.interfaces: [eth0]
mine_interval: 60
New in Salt v0.15
Either master or minion config
Be mindful of data security
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 18 / 34

20. Minion data
Salt Mine example
Configuring haproxy.cfg:
{% for server,ip in
salt[’mine.get’](
’web-*’,
’network.interfaces’,
[’eth0’]).items() %}
server {{ server }} {{ ip[0] }}:80 check
{% endfor %}
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 19 / 34

21. Minion data
Returners
/etc/salt/{master,minion}:
redis.db: 0
redis.host: myredis
redis.port: 6379
Minions write directly
Can be read into Pillar via ext_pillar
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 20 / 34

22. Minion data
Returner full-circle example
Collect the data:
salt ’web-*’ network.interfaces eth0
--return redis_return
Fetch the data via a custom ext_pillar module.
Use the data:
{% for server,ip in
salt[’pillar.get’](’web.ip_addrs’, {}).items() %}
server {{ server }} {{ ip[0] }}:80 check
{% endfor %}
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 21 / 34

23. Events
Outline
1 Salt internals (briefly)
2 Salt speed
3 Minion data
4 Events
5 Schedules
6 What’s coming
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 22 / 34

24. Events
Events
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 23 / 34

25. Events
Fire events
salt ’lb-*’ event.fire_master
refresh_pool loadbalancer
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 24 / 34

26. Events
Watch for events (manually)
Some assembly required
salt/tests/eventlisten.py
Coming soon to salt-api
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 25 / 34

27. Events
Reactor (react to events)
/etc/salt/master:
reactor:
- loadbalancer:
- /src/reactor/refresh_pool.sls
/src/reactor/refresh_pool.sls:
{% if data[’type’] == ’refresh_pool’ %}
highstate_run:
cmd.state.highstate:
- tgt: lb-*
{% endif %}
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 26 / 34

28. Schedules
Outline
1 Salt internals (briefly)
2 Salt speed
3 Minion data
4 Events
5 Schedules
6 What’s coming
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 27 / 34

29. Schedules
Schedules
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 28 / 34

30. Schedules
Add events
/etc/salt/{master,minion} (or pillar):
schedule:
highstate:
function: state.highstate
minutes: 60
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 29 / 34

31. Schedules
Stats gathering
schedule:
uptime:
function: status.uptime
seconds: 60
returner: redis
meminfo:
function: status.meminfo
minutes: 5
returner: redis
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 30 / 34

32. What’s coming
Outline
1 Salt internals (briefly)
2 Salt speed
3 Minion data
4 Events
5 Schedules
6 What’s coming
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 31 / 34

33. What’s coming
What’s coming
Salt v.0.next
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 32 / 34

34. What’s coming
Monitoring states
Configure inline with existing states
Individual components are in place
Needed: glue
Needed: alerting
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 33 / 34

35. What’s coming
Data resolution
Time-series data
Thin and/or summarize older and older data
Free with some returners
Seth House <seth@eseth.com> (OpenWest Conference 2013)Real-time infrastructure management with Salt 2013-05-03 34 / 34