A presentation about creating message queues in relational database. Presented at SQL BITS X

1. Fireside
Grade of the Steel
Thomas Kejser

2. Grade of the Steel
1. Pick a simple workload
2. Do the naïve thing
3. Tune the … out of it
4. Find the limit
5. Get smart (optional)

3. Grade of the Steel Examples
• 16M BULK Rows/sec
• 50M rows SELECT (4 x 10Gbit)
• Backup at 6GB/sec
• 1.1M INSERT/sec

4. Step 1 – Theory and Baseline
Commit SQLIO.EXE
BufferOffset –t1
–o1
–b60
...
Slot 1
Slot
–kW
...
n -fsequential
Writer COREINFO.EXE > info.txt
Queue
Log Writer

5. Step 2 – Design Workload
File 1 FG 1 Table 1
File 2 FG 2 Table 2
SQLCMD
Stripe
File 3 FG 3 Threads
Table 3
File X FG X Table X

6. Step 3 – Tune
• Add Files / Tables
Theory Measure
• Thread Sweet spot
• Affinitize I/O
• Placement of Log Change

7. Messages Queues
Part 1 – Naïve

8. Problem Statement
Push Pop
Queue Structure
300B Msg
Msg Msg Msg Msg
msg
Ordered

9. The Naïve Approach
Push Pop
• Seek first value page • Seek last value page
• INSERT a row at page • DELETE/Output key on page
Min Max Min Max
Msg Msg Msg

10. Why this doesn’t Scale!
Root
Min Min Min
Min Min Min Min Min Min
Min Max

11. Messages Queues
Part 2 – Getting smart

12. Summarising the Problem
• Hot stuff
– Root
– Min page
– Max page
– Intermediate pages
• Alloc/Dealloc
• BUT: We Must have
order!

13. Cooling it down

14. Allocations - What if?...
Push Pop
• Seek first value page • Seek last value page
• UPDATE Reference Count • UPDATE Reference Count
Min Max Min Max
Msg++ Msg--

15. Dissipate the Heat
Last Digit = 0 Last Digit = 1 Last Digit = 2
Min Max Min Max Min Max
Msg-- Msg++ Msg-- Msg++ Msg-- Msg++

16. Eliminating Thread Contention
VERY
fast!
PopSequence++ PushSequence++
Queue Structure
4 5 6 7 8
Ordered

17. Ring Buffering
PushSequence+=2 PopSequence++ PushSequence+=1
Mod 100 Mod 100 Mod 100
MyQ
Slot:0 Slot:42 Slot: 97 Slot: 98 Slot: 99
Msg: 200 Msg: 142 Msg: 197 Msg: 198 Msg: 199
Ordered

18. Summing it up
• UPDATE instead of
INSERT/DELETE
• Multi partitions =
multiple trees
• Ring buffer using modulo
• Sweet spot concurrency