Soumettre la recherche
Mettre en ligne
LOFAR - finding transients in the radio spectrum
•
Télécharger en tant que ODP, PDF
•
0 j'aime
•
560 vues
Gijs Molenaar
Suivre
Technologie
Signaler
Partager
Signaler
Partager
1 sur 18
Télécharger maintenant
Recommandé
Richard Salter: Using the Titanium OpenGL Module
Richard Salter: Using the Titanium OpenGL Module
Axway Appcelerator
Fantastic caches and where to find them
Fantastic caches and where to find them
Alexey Tokar
Toy Model Overview
Toy Model Overview
Brandon McKinzie
Java JVM Memory Cheat Sheet
Java JVM Memory Cheat Sheet
Mark Papis
Cassandra Lunch #59 Functions in Cassandra
Cassandra Lunch #59 Functions in Cassandra
Anant Corporation
Python memory managment. Deeping in Garbage collector
Python memory managment. Deeping in Garbage collector
Jose Manuel Ortega Candel
Parallel computing with GPars
Parallel computing with GPars
Pablo Molnar
WebGL
WebGL
EU Edge
Recommandé
Richard Salter: Using the Titanium OpenGL Module
Richard Salter: Using the Titanium OpenGL Module
Axway Appcelerator
Fantastic caches and where to find them
Fantastic caches and where to find them
Alexey Tokar
Toy Model Overview
Toy Model Overview
Brandon McKinzie
Java JVM Memory Cheat Sheet
Java JVM Memory Cheat Sheet
Mark Papis
Cassandra Lunch #59 Functions in Cassandra
Cassandra Lunch #59 Functions in Cassandra
Anant Corporation
Python memory managment. Deeping in Garbage collector
Python memory managment. Deeping in Garbage collector
Jose Manuel Ortega Candel
Parallel computing with GPars
Parallel computing with GPars
Pablo Molnar
WebGL
WebGL
EU Edge
Matematika Dasar Bab II Fungsi Real
Matematika Dasar Bab II Fungsi Real
Adhi99
"Metrics: Where and How", Vsevolod Polyakov
"Metrics: Where and How", Vsevolod Polyakov
Yulia Shcherbachova
Java program-to-calculate-area-and-circumference-of-circle
Java program-to-calculate-area-and-circumference-of-circle
University of Essex
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Nicholas Peihl
Gaucheで本を作る
Gaucheで本を作る
guest7a66b8
실시간 대중교통 경로 탐색
실시간 대중교통 경로 탐색
지승 한
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
NETWAYS
Presto Bangalore Meetup1 Presto Raptor@ola
Presto Bangalore Meetup1 Presto Raptor@ola
Shubham Tagra
MongoDB - Warehouse and Aggregator of Events
MongoDB - Warehouse and Aggregator of Events
Maxim Ligus
Compiler basics: lisp to assembly
Compiler basics: lisp to assembly
Phil Eaton
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Nikita Koksharov
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Austin Benson
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
PROIDEA
Lrz kurse: r as superglue
Lrz kurse: r as superglue
Ferdinand Jamitzky
Kapacitor Alert Topic handlers
Kapacitor Alert Topic handlers
InfluxData
2010 jan 12
2010 jan 12
leroy walker
The impact of supercomputers on MSR
The impact of supercomputers on MSR
Yasutaka Kamei
MongoDB - visualisation of slow operations
MongoDB - visualisation of slow operations
Kay1A
C coroutine
C coroutine
Chien-Wei Huang
Purely Functional Data Structures ex3.3 leftist heap
Purely Functional Data Structures ex3.3 leftist heap
Tetsuro Nagae
Sorry - How Bieber broke Google Cloud at Spotify
Sorry - How Bieber broke Google Cloud at Spotify
Neville Li
Lofar python meetup jan9 2013
Lofar python meetup jan9 2013
Gijs Molenaar
Contenu connexe
Tendances
Matematika Dasar Bab II Fungsi Real
Matematika Dasar Bab II Fungsi Real
Adhi99
"Metrics: Where and How", Vsevolod Polyakov
"Metrics: Where and How", Vsevolod Polyakov
Yulia Shcherbachova
Java program-to-calculate-area-and-circumference-of-circle
Java program-to-calculate-area-and-circumference-of-circle
University of Essex
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Nicholas Peihl
Gaucheで本を作る
Gaucheで本を作る
guest7a66b8
실시간 대중교통 경로 탐색
실시간 대중교통 경로 탐색
지승 한
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
NETWAYS
Presto Bangalore Meetup1 Presto Raptor@ola
Presto Bangalore Meetup1 Presto Raptor@ola
Shubham Tagra
MongoDB - Warehouse and Aggregator of Events
MongoDB - Warehouse and Aggregator of Events
Maxim Ligus
Compiler basics: lisp to assembly
Compiler basics: lisp to assembly
Phil Eaton
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Nikita Koksharov
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Austin Benson
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
PROIDEA
Lrz kurse: r as superglue
Lrz kurse: r as superglue
Ferdinand Jamitzky
Kapacitor Alert Topic handlers
Kapacitor Alert Topic handlers
InfluxData
2010 jan 12
2010 jan 12
leroy walker
The impact of supercomputers on MSR
The impact of supercomputers on MSR
Yasutaka Kamei
MongoDB - visualisation of slow operations
MongoDB - visualisation of slow operations
Kay1A
C coroutine
C coroutine
Chien-Wei Huang
Purely Functional Data Structures ex3.3 leftist heap
Purely Functional Data Structures ex3.3 leftist heap
Tetsuro Nagae
Tendances
(20)
Matematika Dasar Bab II Fungsi Real
Matematika Dasar Bab II Fungsi Real
"Metrics: Where and How", Vsevolod Polyakov
"Metrics: Where and How", Vsevolod Polyakov
Java program-to-calculate-area-and-circumference-of-circle
Java program-to-calculate-area-and-circumference-of-circle
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Mashup OpenStreetMap and Wikidata to Create Useful Vector Data
Gaucheで本を作る
Gaucheで本を作る
실시간 대중교통 경로 탐색
실시간 대중교통 경로 탐색
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
OSDC 2016 - Chronix - A fast and efficient time series storage based on Apach...
Presto Bangalore Meetup1 Presto Raptor@ola
Presto Bangalore Meetup1 Presto Raptor@ola
MongoDB - Warehouse and Aggregator of Events
MongoDB - Warehouse and Aggregator of Events
Compiler basics: lisp to assembly
Compiler basics: lisp to assembly
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Java data structures powered by Redis. Introduction to Redisson @ Redis Light...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
Data Structures and Performance for Scientific Computing with Hadoop and Dumb...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
[4DEV][Łódź] Ivan Vaskevych - InfluxDB and Grafana fighting together with IoT...
Lrz kurse: r as superglue
Lrz kurse: r as superglue
Kapacitor Alert Topic handlers
Kapacitor Alert Topic handlers
2010 jan 12
2010 jan 12
The impact of supercomputers on MSR
The impact of supercomputers on MSR
MongoDB - visualisation of slow operations
MongoDB - visualisation of slow operations
C coroutine
C coroutine
Purely Functional Data Structures ex3.3 leftist heap
Purely Functional Data Structures ex3.3 leftist heap
Similaire à LOFAR - finding transients in the radio spectrum
Sorry - How Bieber broke Google Cloud at Spotify
Sorry - How Bieber broke Google Cloud at Spotify
Neville Li
Lofar python meetup jan9 2013
Lofar python meetup jan9 2013
Gijs Molenaar
HBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Spark Summit
MongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & Analytics
Server Density
Apache Spark v3.0.0
Apache Spark v3.0.0
Jean-Georges Perrin
Bringing code to the data: from MySQL to RocksDB for high volume searches
Bringing code to the data: from MySQL to RocksDB for high volume searches
Ivan Kruglov
Graphite
Graphite
Oleg Obleukhov
Riak add presentation
Riak add presentation
Ilya Bogunov
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
MongoDB
Scio - Moving to Google Cloud, A Spotify Story
Scio - Moving to Google Cloud, A Spotify Story
Neville Li
Tweaking perfomance on high-load projects_Думанский Дмитрий
Tweaking perfomance on high-load projects_Думанский Дмитрий
GeeksLab Odessa
Infrastructure Monitoring with Postgres
Infrastructure Monitoring with Postgres
Steven Simpson
Scalable IoT platform
Scalable IoT platform
Swapnil Bawaskar
Realtime Statistics based on Apache Storm and RocketMQ
Realtime Statistics based on Apache Storm and RocketMQ
Xin Wang
Hadoop at aadhaar
Hadoop at aadhaar
Regunath B
Lens: Data exploration with Dask and Jupyter widgets
Lens: Data exploration with Dask and Jupyter widgets
Víctor Zabalza
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
DataStax
DataStax and Esri: Geotemporal IoT Search and Analytics
DataStax and Esri: Geotemporal IoT Search and Analytics
DataStax Academy
JavaOne_2010
JavaOne_2010
Tadaya Tsuyukubo
Similaire à LOFAR - finding transients in the radio spectrum
(20)
Sorry - How Bieber broke Google Cloud at Spotify
Sorry - How Bieber broke Google Cloud at Spotify
Lofar python meetup jan9 2013
Lofar python meetup jan9 2013
HBaseCon 2015: OpenTSDB and AsyncHBase Update
HBaseCon 2015: OpenTSDB and AsyncHBase Update
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
MongoDB: Optimising for Performance, Scale & Analytics
MongoDB: Optimising for Performance, Scale & Analytics
Apache Spark v3.0.0
Apache Spark v3.0.0
Bringing code to the data: from MySQL to RocksDB for high volume searches
Bringing code to the data: from MySQL to RocksDB for high volume searches
Graphite
Graphite
Riak add presentation
Riak add presentation
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Ensuring High Availability for Real-time Analytics featuring Boxed Ice / Serv...
Scio - Moving to Google Cloud, A Spotify Story
Scio - Moving to Google Cloud, A Spotify Story
Tweaking perfomance on high-load projects_Думанский Дмитрий
Tweaking perfomance on high-load projects_Думанский Дмитрий
Infrastructure Monitoring with Postgres
Infrastructure Monitoring with Postgres
Scalable IoT platform
Scalable IoT platform
Realtime Statistics based on Apache Storm and RocketMQ
Realtime Statistics based on Apache Storm and RocketMQ
Hadoop at aadhaar
Hadoop at aadhaar
Lens: Data exploration with Dask and Jupyter widgets
Lens: Data exploration with Dask and Jupyter widgets
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
Monitoring Cassandra at Scale (Jason Cacciatore, Netflix) | C* Summit 2016
DataStax and Esri: Geotemporal IoT Search and Analytics
DataStax and Esri: Geotemporal IoT Search and Analytics
JavaOne_2010
JavaOne_2010
Dernier
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
Gabriella Davis
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
Earley Information Science
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Miguel Araújo
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Katpro Technologies
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
The Digital Insurer
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
Puma Security, LLC
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
The Digital Insurer
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
Delhi Call girls
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Roshan Dwivedi
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
Rafal Los
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
Maria Levchenko
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
hans926745
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen Frames
Sinan KOZAK
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
vu2urc
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
Radu Cotescu
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
Results
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
Enterprise Knowledge
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
apidays
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
Enterprise Knowledge
Dernier
(20)
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptx
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Factors to Consider When Choosing Accounts Payable Services Providers.pptx
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
Axa Assurance Maroc - Insurer Innovation Award 2024
Axa Assurance Maroc - Insurer Innovation Award 2024
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
[2024]Digital Global Overview Report 2024 Meltwater.pdf
[2024]Digital Global Overview Report 2024 Meltwater.pdf
Unblocking The Main Thread Solving ANRs and Frozen Frames
Unblocking The Main Thread Solving ANRs and Frozen Frames
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
A Call to Action for Generative AI in 2024
A Call to Action for Generative AI in 2024
IAC 2024 - IA Fast Track to Search Focused AI Solutions
IAC 2024 - IA Fast Track to Search Focused AI Solutions
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf
LOFAR - finding transients in the radio spectrum
1.
LOFAR Transient Detection
Pipeline Gijs Molenaar gijs@pythonic.nl
2.
Agenda ●
Transient detection ● Pipeline layout ● Software
3.
Transients Detection ●
Static sources are boring ● Transient – something that changes ● Huge amount of data – automation
4.
Image data
5.
The data ●
1 datacube per second ● 10 frequency bands ● In the future 10 images per second ● In the future 4 different polarization ● Non stop
6.
Transient detection Pipeline
Extract metadata Quality check Source extraction Source extraction Association Detection Classification Database
7.
●
The glue ● Distributing computation ● Image processing ● Statistics ● Source extraction ● Database interactions
8.
Distributed Computation ●
Home made libary ● SSH based ● Difficult to debug ● Difficult to profile ● Doing research on Celery and Hadoop
9.
Database ●
Move calculation to the data ● Highly structured ● Independent data ● Naturally separable by sky coordinates ● ~100 TB/year ● 10.000 insert/second
10.
MonetDB ●
Relational ● Column store DB ● Fast ● Auto tuning! ● Developers next door (CWI)
11.
Challenges ●
Debugging queries ● MonetDB still in active development
12.
INSERT INTO tempbasesources (xtrsrc_id ,datapoints ,I_peak_sum ,I_peak_sq_sum ,weight_peak_sum ,weight_I_peak_sum ,weight_I_peak_sq_sum ) SELECT
b0.xtrsrc_id ,b0.datapoints + 1 AS datapoints ,b0.I_peak_sum + x0.I_peak AS i_peak_sum ,b0.I_peak_sq_sum + x0.I_peak * x0.I_peak AS i_peak_sq_sum ,b0.weight_peak_sum + 1 / (x0.I_peak_err * x0.I_peak_err) AS weight_peak_sum ,b0.weight_I_peak_sum + x0.I_peak / (x0.I_peak_err * x0.I_peak_err) AS weight_i_peak_sum ,b0.weight_I_peak_sq_sum + x0.I_peak * x0.I_peak / (x0.I_peak_err * x0.I_peak_err) AS weight_i_peak_sq_sum FROM basesources b0 ,extractedsources x0 WHERE x0.image_id = @imageid AND b0.zone BETWEEN CAST(FLOOR((x0.decl - @theta) / x0.zoneheight ) AS INTEGER) AND CAST(FLOOR((x0.decl + @theta) / x0.zoneheight ) AS INTEGER) AND ASIN(SQRT((x0.x - b0.x)*(x0.x - b0.x) +(x0.y - b0.y)*(x0.y - b0.y) +(x0.z - b0.z)*(x0.z - b0.z) )/2 ) / SQRT(x0.ra_err * x0.ra_err + b0.ra_err * b0.ra_err +x0.decl_err * x0.decl_err + b0.decl_err * b0.decl_err) < @assoc_r;
13.
MonetDB and Python ●
We maintain the MonetDB Python API ● http://pypi.python.org/pypi/python-monetdb/ ● Problems? Ask me :)
14.
Djonet ●
MonetDB backend for Django ● https://github.com/gijzelaerr/djonet ● brew install monetdb ● pip install python-monetdb djonet ● Contributions are welcome!
15.
VO events ●
Standardized language ● Report observations of astronomical events ● Hey world, check this supernova out over there ● http://comet.transientskp.org
16.
Visualisation ●
Web interface ● Django! ● Not public (yet)
17.
More ●
http://www.transientskp.org/ ● http://www.lofar.org/ ● http://www.aartfaac.org/
18.
Questions?
Télécharger maintenant