Soumettre la recherche
Mettre en ligne
LOFAR - finding transients in the radio spectrum
•
Télécharger en tant que ODP, PDF
•
0 j'aime
•
560 vues
Gijs Molenaar
Suivre
Technologie
Signaler
Partager
Signaler
Partager
1 sur 18
Télécharger maintenant
Recommandé
Richard Salter: Using the Titanium OpenGL Module
Richard Salter: Using the Titanium OpenGL Module
Axway Appcelerator
Fantastic caches and where to find them
Fantastic caches and where to find them
Alexey Tokar
Toy Model Overview
Toy Model Overview
Brandon McKinzie
Java JVM Memory Cheat Sheet
Java JVM Memory Cheat Sheet
Mark Papis
Cassandra Lunch #59 Functions in Cassandra
Cassandra Lunch #59 Functions in Cassandra
Anant Corporation
Python memory managment. Deeping in Garbage collector
Python memory managment. Deeping in Garbage collector
Jose Manuel Ortega Candel
Parallel computing with GPars
Parallel computing with GPars
Pablo Molnar
WebGL
WebGL
EU Edge
Recommandé
Richard Salter: Using the Titanium OpenGL Module
Richard Salter: Using the Titanium OpenGL Module
Axway Appcelerator
Fantastic caches and where to find them
Fantastic caches and where to find them
Alexey Tokar
Toy Model Overview
Toy Model Overview
Brandon McKinzie
Java JVM Memory Cheat Sheet
Java JVM Memory Cheat Sheet
Mark Papis
Cassandra Lunch #59 Functions in Cassandra
Cassandra Lunch #59 Functions in Cassandra
Anant Corporation
Python memory managment. Deeping in Garbage collector
Python memory managment. Deeping in Garbage collector
Jose Manuel Ortega Candel
Parallel computing with GPars
Parallel computing with GPars
Pablo Molnar
WebGL
WebGL
EU Edge
Matematika Dasar Bab II Fungsi Real
Matematika Dasar Bab II Fungsi Real
Adhi99
"Metrics: Where and How", Vsevolod Polyakov
"Metrics: Where and How", Vsevolod Polyakov
Yulia Shcherbachova
Java program-to-calculate-area-and-circumference-of-circle
Java program-to-calculate-area-and-circumference-of-circle
University of Essex
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Nicholas Peihl
Gaucheで本を作る
Gaucheで本を作る
guest7a66b8
실시간 대중교통 경로 탐색
실시간 대중교통 경로 탐색
지승 한
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
NETWAYS
Presto Bangalore Meetup1 Presto Raptor@ola
Presto Bangalore Meetup1 Presto Raptor@ola
Shubham Tagra
MongoDB - Warehouse and Aggregator of Events
MongoDB - Warehouse and Aggregator of Events
Maxim Ligus
Compiler basics: lisp to assembly
Compiler basics: lisp to assembly
Phil Eaton
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Nikita Koksharov
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Austin Benson
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
PROIDEA
Lrz kurse: r as superglue
Lrz kurse: r as superglue
Ferdinand Jamitzky
Kapacitor Alert Topic handlers
Kapacitor Alert Topic handlers
InfluxData
2010 jan 12
2010 jan 12
leroy walker
The impact of supercomputers on MSR
The impact of supercomputers on MSR
Yasutaka Kamei
MongoDB - visualisation of slow operations
MongoDB - visualisation of slow operations
Kay1A
C coroutine
C coroutine
Chien-Wei Huang
Purely Functional Data Structures ex3.3 leftist heap
Purely Functional Data Structures ex3.3 leftist heap
Tetsuro Nagae
Sorry - How Bieber broke Google Cloud at Spotify
Sorry - How Bieber broke Google Cloud at Spotify
Neville Li
Lofar python meetup jan9 2013
Lofar python meetup jan9 2013
Gijs Molenaar
Contenu connexe
Tendances
Matematika Dasar Bab II Fungsi Real
Matematika Dasar Bab II Fungsi Real
Adhi99
"Metrics: Where and How", Vsevolod Polyakov
"Metrics: Where and How", Vsevolod Polyakov
Yulia Shcherbachova
Java program-to-calculate-area-and-circumference-of-circle
Java program-to-calculate-area-and-circumference-of-circle
University of Essex
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Nicholas Peihl
Gaucheで本を作る
Gaucheで本を作る
guest7a66b8
실시간 대중교통 경로 탐색
실시간 대중교통 경로 탐색
지승 한
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
NETWAYS
Presto Bangalore Meetup1 Presto Raptor@ola
Presto Bangalore Meetup1 Presto Raptor@ola
Shubham Tagra
MongoDB - Warehouse and Aggregator of Events
MongoDB - Warehouse and Aggregator of Events
Maxim Ligus
Compiler basics: lisp to assembly
Compiler basics: lisp to assembly
Phil Eaton
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Nikita Koksharov
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Austin Benson
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
PROIDEA
Lrz kurse: r as superglue
Lrz kurse: r as superglue
Ferdinand Jamitzky
Kapacitor Alert Topic handlers
Kapacitor Alert Topic handlers
InfluxData
2010 jan 12
2010 jan 12
leroy walker
The impact of supercomputers on MSR
The impact of supercomputers on MSR
Yasutaka Kamei
MongoDB - visualisation of slow operations
MongoDB - visualisation of slow operations
Kay1A
C coroutine
C coroutine
Chien-Wei Huang
Purely Functional Data Structures ex3.3 leftist heap
Purely Functional Data Structures ex3.3 leftist heap
Tetsuro Nagae
Tendances
(20)
Matematika Dasar Bab II Fungsi Real
Matematika Dasar Bab II Fungsi Real
"Metrics: Where and How", Vsevolod Polyakov
"Metrics: Where and How", Vsevolod Polyakov
Java program-to-calculate-area-and-circumference-of-circle
Java program-to-calculate-area-and-circumference-of-circle
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Gaucheで本を作る
Gaucheで本を作る
실시간 대중교통 경로 탐색
실시간 대중교통 경로 탐색
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
Presto Bangalore Meetup1 Presto Raptor@ola
Presto Bangalore Meetup1 Presto Raptor@ola
MongoDB - Warehouse and Aggregator of Events
MongoDB - Warehouse and Aggregator of Events
Compiler basics: lisp to assembly
Compiler basics: lisp to assembly
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
Lrz kurse: r as superglue
Lrz kurse: r as superglue
Kapacitor Alert Topic handlers
Kapacitor Alert Topic handlers
2010 jan 12
2010 jan 12
The impact of supercomputers on MSR
The impact of supercomputers on MSR
MongoDB - visualisation of slow operations
MongoDB - visualisation of slow operations
C coroutine
C coroutine
Purely Functional Data Structures ex3.3 leftist heap
Purely Functional Data Structures ex3.3 leftist heap
Similaire à LOFAR - finding transients in the radio spectrum
Sorry - How Bieber broke Google Cloud at Spotify
Sorry - How Bieber broke Google Cloud at Spotify
Neville Li
Lofar python meetup jan9 2013
Lofar python meetup jan9 2013
Gijs Molenaar
HBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Spark Summit
MongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & Analytics
Server Density
Apache Spark v3.0.0
Apache Spark v3.0.0
Jean-Georges Perrin
Bringing code to the data: from MySQL to RocksDB for high volume searches
Bringing code to the data: from MySQL to RocksDB for high volume searches
Ivan Kruglov
Graphite
Graphite
Oleg Obleukhov
Riak add presentation
Riak add presentation
Ilya Bogunov
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
MongoDB
Scio - Moving to Google Cloud, A Spotify Story
Scio - Moving to Google Cloud, A Spotify Story
Neville Li
Tweaking perfomance on high-load projects_Думанский Дмитрий
Tweaking perfomance on high-load projects_Думанский Дмитрий
GeeksLab Odessa
Infrastructure Monitoring with Postgres
Infrastructure Monitoring with Postgres
Steven Simpson
Scalable IoT platform
Scalable IoT platform
Swapnil Bawaskar
Realtime Statistics based on Apache Storm and RocketMQ
Realtime Statistics based on Apache Storm and RocketMQ
Xin Wang
Hadoop at aadhaar
Hadoop at aadhaar
Regunath B
Lens: Data exploration with Dask and Jupyter widgets
Lens: Data exploration with Dask and Jupyter widgets
Víctor Zabalza
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
DataStax
DataStax and Esri: Geotemporal IoT Search and Analytics
DataStax and Esri: Geotemporal IoT Search and Analytics
DataStax Academy
JavaOne_2010
JavaOne_2010
Tadaya Tsuyukubo
Similaire à LOFAR - finding transients in the radio spectrum
(20)
Sorry - How Bieber broke Google Cloud at Spotify
Sorry - How Bieber broke Google Cloud at Spotify
Lofar python meetup jan9 2013
Lofar python meetup jan9 2013
HBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon 2015: OpenTSDB and AsyncHBase Update
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
MongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & Analytics
Apache Spark v3.0.0
Apache Spark v3.0.0
Bringing code to the data: from MySQL to RocksDB for high volume searches
Bringing code to the data: from MySQL to RocksDB for high volume searches
Graphite
Graphite
Riak add presentation
Riak add presentation
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Scio - Moving to Google Cloud, A Spotify Story
Scio - Moving to Google Cloud, A Spotify Story
Tweaking perfomance on high-load projects_Думанский Дмитрий
Tweaking perfomance on high-load projects_Думанский Дмитрий
Infrastructure Monitoring with Postgres
Infrastructure Monitoring with Postgres
Scalable IoT platform
Scalable IoT platform
Realtime Statistics based on Apache Storm and RocketMQ
Realtime Statistics based on Apache Storm and RocketMQ
Hadoop at aadhaar
Hadoop at aadhaar
Lens: Data exploration with Dask and Jupyter widgets
Lens: Data exploration with Dask and Jupyter widgets
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
DataStax and Esri: Geotemporal IoT Search and Analytics
DataStax and Esri: Geotemporal IoT Search and Analytics
JavaOne_2010
JavaOne_2010
Dernier
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
Delhi Call girls
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
carlostorres15106
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
HostedbyConfluent
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
BookNet Canada
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
ThousandEyes
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Katpro Technologies
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Safe Software
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Patryk Bandurski
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
hans926745
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
shyamraj55
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
2toLead Limited
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
Padma Pradeep
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
soniya singh
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
Radu Cotescu
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
Maria Levchenko
Azure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & Application
AndikSusilo4
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
Delhi Call girls
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
Michael W. Hawkins
Benefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other Frameworks
Softradix Technologies
Dernier
(20)
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Kotlin Multiplatform & Compose Multiplatform - Starter kit for pragmatics
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
Transcript: #StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
Integration and Automation in Practice: CI/CD in Mule Integration and Automat...
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Automating Business Process via MuleSoft Composer | Bangalore MuleSoft Meetup...
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Tech-Forward - Achieving Business Readiness For Copilot in Microsoft 365
Install Stable Diffusion in windows machine
Install Stable Diffusion in windows machine
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
Azure Monitor & Application Insight to monitor Infrastructure & Application
Azure Monitor & Application Insight to monitor Infrastructure & Application
08448380779 Call Girls In Civil Lines Women Seeking Men
08448380779 Call Girls In Civil Lines Women Seeking Men
GenCyber Cyber Security Day Presentation
GenCyber Cyber Security Day Presentation
Benefits Of Flutter Compared To Other Frameworks
Benefits Of Flutter Compared To Other Frameworks
LOFAR - finding transients in the radio spectrum
1.
LOFAR Transient Detection
Pipeline Gijs Molenaar gijs@pythonic.nl
2.
Agenda ●
Transient detection ● Pipeline layout ● Software
3.
Transients Detection ●
Static sources are boring ● Transient – something that changes ● Huge amount of data – automation
4.
Image data
5.
The data ●
1 datacube per second ● 10 frequency bands ● In the future 10 images per second ● In the future 4 different polarization ● Non stop
6.
Transient detection Pipeline
Extract metadata Quality check Source extraction Source extraction Association Detection Classification Database
7.
●
The glue ● Distributing computation ● Image processing ● Statistics ● Source extraction ● Database interactions
8.
Distributed Computation ●
Home made libary ● SSH based ● Difficult to debug ● Difficult to profile ● Doing research on Celery and Hadoop
9.
Database ●
Move calculation to the data ● Highly structured ● Independent data ● Naturally separable by sky coordinates ● ~100 TB/year ● 10.000 insert/second
10.
MonetDB ●
Relational ● Column store DB ● Fast ● Auto tuning! ● Developers next door (CWI)
11.
Challenges ●
Debugging queries ● MonetDB still in active development
12.
INSERT INTO tempbasesources (xtrsrc_id ,datapoints ,I_peak_sum ,I_peak_sq_sum ,weight_peak_sum ,weight_I_peak_sum ,weight_I_peak_sq_sum ) SELECT
b0.xtrsrc_id ,b0.datapoints + 1 AS datapoints ,b0.I_peak_sum + x0.I_peak AS i_peak_sum ,b0.I_peak_sq_sum + x0.I_peak * x0.I_peak AS i_peak_sq_sum ,b0.weight_peak_sum + 1 / (x0.I_peak_err * x0.I_peak_err) AS weight_peak_sum ,b0.weight_I_peak_sum + x0.I_peak / (x0.I_peak_err * x0.I_peak_err) AS weight_i_peak_sum ,b0.weight_I_peak_sq_sum + x0.I_peak * x0.I_peak / (x0.I_peak_err * x0.I_peak_err) AS weight_i_peak_sq_sum FROM basesources b0 ,extractedsources x0 WHERE x0.image_id = @imageid AND b0.zone BETWEEN CAST(FLOOR((x0.decl - @theta) / x0.zoneheight ) AS INTEGER) AND CAST(FLOOR((x0.decl + @theta) / x0.zoneheight ) AS INTEGER) AND ASIN(SQRT((x0.x - b0.x)*(x0.x - b0.x) +(x0.y - b0.y)*(x0.y - b0.y) +(x0.z - b0.z)*(x0.z - b0.z) )/2 ) / SQRT(x0.ra_err * x0.ra_err + b0.ra_err * b0.ra_err +x0.decl_err * x0.decl_err + b0.decl_err * b0.decl_err) < @assoc_r;
13.
MonetDB and Python ●
We maintain the MonetDB Python API ● http://pypi.python.org/pypi/python-monetdb/ ● Problems? Ask me :)
14.
Djonet ●
MonetDB backend for Django ● https://github.com/gijzelaerr/djonet ● brew install monetdb ● pip install python-monetdb djonet ● Contributions are welcome!
15.
VO events ●
Standardized language ● Report observations of astronomical events ● Hey world, check this supernova out over there ● http://comet.transientskp.org
16.
Visualisation ●
Web interface ● Django! ● Not public (yet)
17.
More ●
http://www.transientskp.org/ ● http://www.lofar.org/ ● http://www.aartfaac.org/
18.
Questions?
Télécharger maintenant