Slides for Venturing into Datomic talk

1. Venturing into
Datomic

2. Kirill Salykin
Former .NET, Ruby developer
Now happy clojurist at AdGoji :)

3. Adgoji is a media agency.
We help advertisers find value on mobile.
From our offices in Amsterdam, we connect
advertisers with the right* mobile users
worldwide. To do so, we have built a Clojure
based real-time bidding platform that buys
targeted mobile advertisement from the largest
market places in the world.
• and by “right”, we mean users willing to see
and/or interact with (click, install, purchase)

4. Kudos to Carlo
For organising meetup every month

5. What is Datomic?
According to Wikipedia: Datomic is a distributed
database and implementation of Datalog.It
has ACID transactions, joins, and a logical query language,
Datalog.
Designed by Rich Hickey

6. Architecture
Datomic (on-prem):
- Peer
- Transactor
- Storage

8. Peer
- Lives in the app
- Does query
- Persistent index + Live index
- Receives updates from transactor

9. Transactor
- Handles transactions
- Commits results to the Storage
- Single threaded
- Transmit changes to connected peers
- Live index
- Index in background, place index to the storage

10. Storage
- Stores the data :)
- Supported storages: DynamoDB, SQL,
Cassandra

11. Information model
Datomic uses Triple to describe data - Datom
Entity Attribute Value
1 :likes “pizza”
1 :likes “lasagna”
1 :first-name “Kirill”

12. Information model 2
Datomic extends triple to include operation
Add or Retract
Entity Attribute Value Op (Add?)
1 :likes “pizza” True
1 :likes “lasagna” True
1 :first-name “Kirill” True
1 :likes “lasagna” False

13. Information model 3
Every added/retracted Datum also references
transaction
Entity Attribute Value Op (Add?) Transaction
1 :likes “pizza” True 2
1 :likes “lasagna” True 2
1 :first-name “Kirill” True 2
1 :likes “lasagna” False 3

14. Information model 4
Schema
{:db/ident :person/name
:db/valueType :db.type/string
:db/cardinality :db.cardinality/one}
valueType:
- keyword
- string
- boolean
- long
- bigint
- float
- double
- bigdec
- ref
- instant
- uuid
- uri
- bytes
cardinality: one or many

15. Transactions
- Serialised
- No `begin transaction` `commit/rollback`
- Updates indices in the background
- Sends updates to the connected peers
- String tempids (references)
- Returns db-before, db-after and tempids
resolution

16. Transactions 2
- Indexes are B-tree-like structures: sorted, immutable, persistent,
1,000+ branching factor, custom sort order, fast lookup and range
scans, able to be efficiently merged (details unknown).
- Index trees are shallow, no more than three level deep: root node,
directories and segments as leafs.

17. Transactions 3
- Constraints via transaction functions
- Runs during transaction
- On exception - transaction rollbacks

18. Query
- Directly via indices
- Query API + Entity API
- Pull API

19. Query 2
- Indicies:
- Covering (contains data, not reference)
- Element of index is not datom, but segment
- Segment is from 1,000 to 20,000 datoms (~50 Kb)
- EAVT
- AEVT
- VAET
- AVET
- Log index
- Keeps fetched segments in the memory

20. Features
- No query planner (you responsible for optimal query)
- Query result must feet in the memory (no dump to file,
as sql does)
- Schema alteration can be tricky (even not possible sometimes)
- No compound indices
- Sort done manually
- Pagination done manually (no offset/limit in query defined)
- Cant add `nil` attribute value - retract only
- RetractEntity to retract all attributes of entity
- Stu’s 10 billion datoms (soft limit)
- Huge heap - may influence GC?
- GC is a manual operation (garbage segments may still be used?
run rarely, e.g. once a week)
- Memcache support out-of-the-box
- Segments easy to cache - immutable
- Excision - not to correct, but because forced (for instance GDPR)

21. Resources
- Rich Hickey: Deconstructing the Database
https://www.youtube.com/watch?v=Cym4TZwTCNU
- Nitika Prokopov: Unofficial guide to Datomic internals
http://tonsky.me/blog/unofficial-guide-to-datomic-internals
- Stu Halloway: Day of Datomic
https://docs.datomic.com/on-prem/day-of-datomic.html
- http://www.learndatalogtoday.org
- Official docs: https://docs.datomic.com/on-prem/index.html