1. Transient Detection
and SQL
Gijs Molenaar
!
Anton Pannekoek Institute (UvA)
!
@gijzelaerr
gijs@pythonic.nl

2. LOFAR
•
Radio telescope
•
~10–240 MHz
•
Software telescope
•
omni-directional

3. AARTFAAC
•
Lofar sub project
•
24/7 full sky transient
detection
•
‘Changes’
•
99% Python code

4. Transients
•
Super-nova
•
Gamma-ray burst
•
Pulsars
•
exo-planets
•
?

5. Images
•
10-240 MHz
•
Split in 10 bands
•
1 image cube per second
•
future 10 images per second

6. Data
•
Source
•
Position
•
Flux (intensity)
•
Shape (2d gaussian)

7. Data rate
•
500 - 5000 inserts per second
•
Higher in other modes

8. Calculations
•
Associate in frequency
•
Associate in time

9. Quite a lot of data
•
move calculations to data (base)

10. nonosql
•
Structured data
•
Natural sharding by
sky position
•
MonetDB

11. Why MonetDB
•
Dutch (CWI)
•
Fast with huge
datasets
•
Developers next door

12. calculations to data
•
No moving around data
•
one huge big monolithic query per ‘operation’

13. Example

14. HORRIBLE
•
Huge monolithic queries
•
hard to read
•
hard to reason about
•
hard to test
•
hard to maintain
•
hard to port

15. Solution?
•
Functions?

16. NO!
•
DB revisions, versioning hell
•
For every code change new version
•
Schema migrations
•
Init functions on startup?
•
DB specific

17. SOLUTION

18. Why?
•
Object Relational Mapper
•
Better Query Builder Language!
•
Support multiple database dialect
•
Functions per dialect
!
•
SQLAlchemy -> SQL is like C -> assembler

19. example

20. Why not earlier?
•
Started by non programmers
•
No MonetDB dialect for SQLAlchemy

21. sqlalchemy-monetdb
•
pip install sqlalchemy-monetdb
•
https://github.com/gijzelaerr/sqlalchemy-monetdb
•
https://github.com/gijzelaerr/djonet
•
(banana)

22. Questions?