8. A Bit of History
Initially written in Python
Utilized Twisted framework
Historical Data stored in relational database
It worked, but it did not perform
8 Oracle OpenWorld 2011
9. Why?
Global Interpreter Lock (GIL)
caused performance problems
Relational Database not efficient
for time-series data
9 Oracle OpenWorld 2011
10. So…
Why switch to Clojure?
It is hip
It was designed with multi-threading in mind
It is a functional language
It uses the JVM
We can use all the Java libraries lying around
Homoiconic
10 Oracle OpenWorld 2011
11. “And there was much rejoicing”
-- Monty Python and the Holy Grail
11 Oracle OpenWorld 2011
12. So, this is how we did it
12 Oracle OpenWorld 2011
13. Loader
Takes XML and dumps it on a
Message Bus (RabbitMQ)
Nothing much to see here but…
13 Oracle OpenWorld 2011
14. Data is Code
So, how do we store the configurations we want for the various datacenters?
As code … data … code …
[“dc1” “url1” “type1”
“dc2” “url2” “type2”]
The configs are just Clojure code and they make sense
14 Oracle OpenWorld 2011
15. RabbitMQ
That is easy
We will just use the RabbitMQ Java API
We will create Clojure centric data structures
This whole Java interoperability is kind of nice …
things just kind of work
15 Oracle OpenWorld 2011
16. Also!
If code is data … then we can just send the code over
RabbitMQ
16 Oracle OpenWorld 2011
17. Wait – What?
We don’t need any funky configurations?
We don’t need to use XML?
We don’t need to use JSON?
If it is Clojure talking to Clojure we can just use data (or is it
code, I am confused)
17 Oracle OpenWorld 2011
18. Persister
Takes the data off the bus and writes it to disk
The Java ecosystem has tools for that, too
Jrobin
We now have our own little timeseries database and we didn’t
really have to work for it.
18 Oracle OpenWorld 2011
19. Consumer
Takes metrics and does stuff with them
Checks
Computes
Aggregates
Historical Aggregates
19 Oracle OpenWorld 2011
22. Threading
All those metrics are Clojure Agents, so I don’t
have to worry about it
All 16 of my processors get used
Life is easy
22 Oracle OpenWorld 2011
24. WWW
We are not web developer types, which is fine,
Clojure (plus some libraries) makes that easy, too
Compojure
Hiccup
So, no HTML. Just code
[:a {:href “/”}]
24 Oracle OpenWorld 2011
25. Query
We need a way to query the data in real time
Clojure is homiconic
So…
We will just create our own DSL
25 Oracle OpenWorld 2011
26. The DSL
It is just code
We can use existing Clojure functions plus new ones like:
where
select
pivot
filter
sort
format
sum-by and agg-by
26 Oracle OpenWorld 2011
28. Explanation
Looks a lot like Linux pipes
Which is a good way to think about it
Clojure way of reading it is:
(sort (select (filter (pivot (where)))))
28 Oracle OpenWorld 2011
30. DevOps
What we needed (and what we got)
Reports
Ad-hoc Queries
Corrective actions?
Make the app smarter?
30 Oracle OpenWorld 2011
31. Corrective Actions
Write little python scripts that
pull data and take actions
This was so easy that we had to do it
Simple, repetitive actions are now fully automated
Life is better
31 Oracle OpenWorld 2011
32. App Smarter
App now uses the monitoring to feed intelligently
Less operator interaction needed
More time spent solving real problems
32 Oracle OpenWorld 2011
