1) DAQDB is a key-value store designed for data acquisition systems to provide fast pre-computing and long-term storage of large volumes of data from experiments like the LHC. 2) It uses optimized data structures like adaptive radix tries and distributed locking to process over 20,000 data fragments every millisecond from multiple sources at throughput of over 100 Gbps. 3) The storage is distributed across persistent memory and NVMe devices to maximize performance while ensuring reliability and persistence of data.

1. Jakub Radtke, Maciej Maciejewski
April 2019

2. SPDK, PMDK & Vtune™ Summit 2
DataAcQuisiontypicalusecase
Key – timestamp, camera ID, geo, ...
Value - frame
Processing at the edge
Filtered data
Cloud storage
DAQ DB enables better ways
to process data at the edge
Data
stream

3. SPDK, PMDK & Vtune™ Summit 3
LHCexperimentswillbeproducing
©CERN

4. SPDK, PMDK & Vtune™ Summit 4
Dataacquisitionsystems(DAQ)
1 in 100
20..40x
5..10x
>20k data fragments every 1ms
©Atlas Experiment©Atlas Experiment

5. SPDK, PMDK & Vtune™ Summit 5
PMDK
Persistent Memory Development Kit
• Optimal performance of persistent memory
SPDK
Storage Performance Development Kit
• User-mode access to NVMe devices (SSDs)

6. SPDK, PMDK & Vtune™ Summit 6
DAQDB–aKVSforDAQ
• First-line buffer for fast pre-computing and
second-line buffer for longer term storage
• Data structure based on optimized
Adaptive Radix Trie
• Distributed locking
Compute farm
~2k nodes
Offline
storage
DAQDB nodes ~500
NVMe
Replication
Persistent
Memory
Key/Value
O(100) Gbps O(100) Gbps
Request router
©Atlas Experiment

7. SPDK, PMDK & Vtune™ Summit 7
DAQ–specificAPI
struct MinidaqKey {
uint64_t eventId; uint16_t subdetectorId; uint16_t runId; }
User-defined key structure
kvPairVector = kvs->GetRange(keyMin, keyMax)
Range queries with
compound keys
kvs->GetRangeAsync(keyMin, keyMax, cb)
Asynchronous mode for
even higher performance
eventKey = kvs->GetAny(options=(lock))
Distributed locking for
next event retrieval
value = kvs->Alloc(key, 10 * 1024)
DAQDB memory allocator
minimizing copy operations

8. SPDK, PMDK & Vtune™ Summit 8
reserve/publishwork-flow
• Application requests
allocation for key data
AllocKey()
• Application requests
allocation for value data
AllocValue()
• Application moves the
value data
• Signals it has finished
Put()
• Optional
• Changes the location of
object from AEP to NVMe
• Changes locks state
• Instructs persistency
Update()
DAQ DB tree
makes object
visible
Copy the value
data to SPDK
ring buffer
Publish actions
Cancel actions
pmemobj_x
reserve()
DAQ DB tree
DAQ DB tree
DAQ DB tree
©MuhannadAjjan - CC BY-SA 4.0

9. SPDK, PMDK & Vtune™ Summit
 Throughput matching
2x100Gbps
 Distributed
Range Queries
 RDMA networking
9
DAQDB
Multi-TB/s hot
storage solution of a
petascale capacity
for DAQ

10. https://github.com/daq-db/daqdb