SlideShare une entreprise Scribd logo
1  sur  25
IGA Workflow
Alessandro Gervaso   Vittorio Zamboni
Introduction
IGA Workflow is a web-based tool created
using the Django Framework.
It is intended as a management tool for the IGA
wet-lab.

We use it to track the lifecycle of biological
samples, from vial to file.
Overview
● Laboratory management, from sample to
  flowcell
● Bioinformatic analyses
● Pipelines management
● Technology
● Other applications and future developments
Biology for dummies
Technology - Basics
Database server
● Postgresql
● Redis
Workers
● Celery
Web server
● nginx + uWSGI
Overview - Lab
● SAMPLE: The basic unit in the lab

● LIBRARY: a treated sample with an attached
  chemical TAG.

● POOL: a set of libraries, ready to be placed
  on a flowcell's lane.
Overview - Lab
Overview - Lab
The main challenge was to replace notebooks
with a tool that allows to:
● insert samples, libraries, pools, and edit
   them;
● create lanes and runs and the configuration
   files for the physical sequencer;
● collect the sequencer results and map them
   in a easy way
Almost done!
Overview - Lab
We started using the basic Django admin
                     BUT

● the page loading was slow
● due to the admin's nature we lacked
  flexibility
● we were forcing the lab people procedures
● management was cumbersome
Overview - Analyses
After the physical sequencing the raw data
(basecalls) must be converted in FASTQ files.

The FASTQ files are FASTA files with some
embedded quality stats.

They are the starting point for almost every
genomic analysis.
Overview - Analyses
To optimize time and resources we use a
cluster of Celery workers.

● we track the software packages used
● we track their parameters
● we create a set of useful stats
Technology
Additions:

● Informational celery tasks (list directories
  content, copy files between devices with
  dbus plugin and UDisks instead of ugly
  hacks, ...)
Overview - Pipelines
FASTQ files     alignments and assemblies

Each analysis use different software in
sequence or in parallel.
Using hundreds of samples, the analyses can't
be handled manually.
Pipelines
The results of each pipelines (like previous
analyses) must be tracked.

Since CLI based software is not user-friendly,
we develop a graphical pipeline builder.

Users are able to choose and combine different
softwares to perform their own analyses.
Pipeline
● Workers have different queues in order to
  satisfy different tasks

● Worker's tasks talk each other with Redis to
  avoid inconsistencies and to improve
  performances
Pipelines
Different queues (stored in Redis dbs):

  ○   available
  ○   queued
  ○   active
  ○   completed
  ○   error
Pipelines
Video demonstration of a working pipeline with
use of different kind of steps.
(1 video: 1 minute)
Under development
● simple interface that allow customers to:
  ○ insert their samples directly
  ○ watch the results of their pipeline in a genome
      browser - also made with Django (see below)!

● barcodes

● genome browser (like GMOD GBrowse, but
  with the greatness of Python instead of the
  confusion of Perl)
Genome browser
An application that allows browsing a genome's
annotations (like genes, or where reads are
aligned).

Actually, the best web genome browser is
GMOD Gbrowse.
Genome browser
The challenge is to develop a genome browser
that has a set of basic features and could
accept plugins for particular type of data - like
GMOD Gbrowse.

In addition, it must be quick and easy to
manage - NOT like GMOD Gbrowse.
Genome browser
Video demonstration of the genome browser.
(2 videos: 1 minute + 1 minute)
Acknowledgments
● The wet-lab     Teams that developed:
  ladies at IGA   ● Django
                  ● JQuery
                  ● nginx
● WEBdeBS         ● uWSGI
                  ● Celery
                  ● Redis
                  ● pip and virtualenv
                  ● PostgreSQL
                  ● All the open source projects involved

Contenu connexe

Tendances

Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...Develcz
 
Bgoug 2019.11 test your pl sql - not your patience
Bgoug 2019.11   test your pl sql - not your patienceBgoug 2019.11   test your pl sql - not your patience
Bgoug 2019.11 test your pl sql - not your patienceJacek Gebal
 
Grunt - The JavaScript Task Runner
Grunt - The JavaScript Task RunnerGrunt - The JavaScript Task Runner
Grunt - The JavaScript Task RunnerMohammed Arif
 
Killer R10K Workflow - PuppetConf 2014
Killer R10K Workflow - PuppetConf 2014Killer R10K Workflow - PuppetConf 2014
Killer R10K Workflow - PuppetConf 2014Puppet
 
Stop Sucking at Building Stuff!
Stop Sucking at Building Stuff!Stop Sucking at Building Stuff!
Stop Sucking at Building Stuff!Puppet
 
Sandstorm.io and OpenDocument
Sandstorm.io and OpenDocumentSandstorm.io and OpenDocument
Sandstorm.io and OpenDocumentAudrey Tang
 

Tendances (10)

Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
Daniel Steigerwald: EsteJS - javascriptové aplikace robusně, modulárně a komf...
 
Bgoug 2019.11 test your pl sql - not your patience
Bgoug 2019.11   test your pl sql - not your patienceBgoug 2019.11   test your pl sql - not your patience
Bgoug 2019.11 test your pl sql - not your patience
 
Grunt - The JavaScript Task Runner
Grunt - The JavaScript Task RunnerGrunt - The JavaScript Task Runner
Grunt - The JavaScript Task Runner
 
Killer R10K Workflow - PuppetConf 2014
Killer R10K Workflow - PuppetConf 2014Killer R10K Workflow - PuppetConf 2014
Killer R10K Workflow - PuppetConf 2014
 
Stop Sucking at Building Stuff!
Stop Sucking at Building Stuff!Stop Sucking at Building Stuff!
Stop Sucking at Building Stuff!
 
What grunt?
What grunt?What grunt?
What grunt?
 
OpenCms Days 2012 - Developing OpenCms with Gradle
OpenCms Days 2012 - Developing OpenCms with GradleOpenCms Days 2012 - Developing OpenCms with Gradle
OpenCms Days 2012 - Developing OpenCms with Gradle
 
Groovy and noteworthy
Groovy and noteworthyGroovy and noteworthy
Groovy and noteworthy
 
Sandstorm.io and OpenDocument
Sandstorm.io and OpenDocumentSandstorm.io and OpenDocument
Sandstorm.io and OpenDocument
 
React Django Presentation
React Django PresentationReact Django Presentation
React Django Presentation
 

Similaire à IGA Workflow Management & Analysis

Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...Sauce Labs
 
DevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and ProjectsDevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and ProjectsFedir RYKHTIK
 
SciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programmingSciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programmingSamuel Lampa
 
ElasTest presentation in MadridJUG (Madrid December 2017)
ElasTest presentation in MadridJUG (Madrid December 2017)ElasTest presentation in MadridJUG (Madrid December 2017)
ElasTest presentation in MadridJUG (Madrid December 2017)ElasTest Project
 
Reproducibility in artificial intelligence
Reproducibility in artificial intelligenceReproducibility in artificial intelligence
Reproducibility in artificial intelligenceCarlos Toxtli
 
2016_04_04_CNI_Spring_Meeting_Microservices
2016_04_04_CNI_Spring_Meeting_Microservices2016_04_04_CNI_Spring_Meeting_Microservices
2016_04_04_CNI_Spring_Meeting_MicroservicesJason Varghese
 
Software Delivery in 2016 - A Continuous Delivery Approach
Software Delivery in 2016 - A Continuous Delivery ApproachSoftware Delivery in 2016 - A Continuous Delivery Approach
Software Delivery in 2016 - A Continuous Delivery ApproachGiovanni Toraldo
 
Jenkinsconf Presentation - Advance jenkins management with multiple projects.
Jenkinsconf Presentation - Advance jenkins management with multiple projects.Jenkinsconf Presentation - Advance jenkins management with multiple projects.
Jenkinsconf Presentation - Advance jenkins management with multiple projects.Ohad Basan
 
TDD with Python and App Engine
TDD with Python and App EngineTDD with Python and App Engine
TDD with Python and App EngineRicardo Bánffy
 
ElasTest presentation in Panel Sistemas company (Madrid December 2017)
ElasTest presentation in Panel Sistemas company (Madrid December 2017)ElasTest presentation in Panel Sistemas company (Madrid December 2017)
ElasTest presentation in Panel Sistemas company (Madrid December 2017)ElasTest Project
 
Introduzione a junit + integrazione con archibus
Introduzione a junit + integrazione con archibusIntroduzione a junit + integrazione con archibus
Introduzione a junit + integrazione con archibusDavide Fella
 
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie..."How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...Fwdays
 
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08Борис Зора
 
OpenProdoc 0.8 Load Tests
OpenProdoc 0.8 Load TestsOpenProdoc 0.8 Load Tests
OpenProdoc 0.8 Load Testsjhierrot
 

Similaire à IGA Workflow Management & Analysis (20)

Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
Slow, Flaky and Legacy Tests: FTFY - Our New Testing Strategy at Net-A-Porter...
 
DevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and ProjectsDevOps for TYPO3 Teams and Projects
DevOps for TYPO3 Teams and Projects
 
SciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programmingSciPipe - A light-weight workflow library inspired by flow-based programming
SciPipe - A light-weight workflow library inspired by flow-based programming
 
ElasTest presentation in MadridJUG (Madrid December 2017)
ElasTest presentation in MadridJUG (Madrid December 2017)ElasTest presentation in MadridJUG (Madrid December 2017)
ElasTest presentation in MadridJUG (Madrid December 2017)
 
Reproducibility in artificial intelligence
Reproducibility in artificial intelligenceReproducibility in artificial intelligence
Reproducibility in artificial intelligence
 
2016_04_04_CNI_Spring_Meeting_Microservices
2016_04_04_CNI_Spring_Meeting_Microservices2016_04_04_CNI_Spring_Meeting_Microservices
2016_04_04_CNI_Spring_Meeting_Microservices
 
Software Delivery in 2016 - A Continuous Delivery Approach
Software Delivery in 2016 - A Continuous Delivery ApproachSoftware Delivery in 2016 - A Continuous Delivery Approach
Software Delivery in 2016 - A Continuous Delivery Approach
 
ElasTest Webinar
ElasTest WebinarElasTest Webinar
ElasTest Webinar
 
Jenkinsconf Presentation - Advance jenkins management with multiple projects.
Jenkinsconf Presentation - Advance jenkins management with multiple projects.Jenkinsconf Presentation - Advance jenkins management with multiple projects.
Jenkinsconf Presentation - Advance jenkins management with multiple projects.
 
TDD with Python and App Engine
TDD with Python and App EngineTDD with Python and App Engine
TDD with Python and App Engine
 
ElasTest presentation in Panel Sistemas company (Madrid December 2017)
ElasTest presentation in Panel Sistemas company (Madrid December 2017)ElasTest presentation in Panel Sistemas company (Madrid December 2017)
ElasTest presentation in Panel Sistemas company (Madrid December 2017)
 
Go at Skroutz
Go at SkroutzGo at Skroutz
Go at Skroutz
 
Introduzione a junit + integrazione con archibus
Introduzione a junit + integrazione con archibusIntroduzione a junit + integrazione con archibus
Introduzione a junit + integrazione con archibus
 
Cypress Testing.pptx
Cypress Testing.pptxCypress Testing.pptx
Cypress Testing.pptx
 
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie..."How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
"How to Use Bazel to Manage Monorepos: The Grammarly Front-End Team’s Experie...
 
Integration testing - A&BP CC
Integration testing - A&BP CCIntegration testing - A&BP CC
Integration testing - A&BP CC
 
Mono Repo
Mono RepoMono Repo
Mono Repo
 
Cypress for Testing
Cypress for TestingCypress for Testing
Cypress for Testing
 
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
Continuous Delivery with Jenkins declarative pipeline XPDays-2018-12-08
 
OpenProdoc 0.8 Load Tests
OpenProdoc 0.8 Load TestsOpenProdoc 0.8 Load Tests
OpenProdoc 0.8 Load Tests
 

Plus de WEBdeBS

Legal Pirlo
Legal PirloLegal Pirlo
Legal PirloWEBdeBS
 
Nodejsconf 2012
Nodejsconf 2012Nodejsconf 2012
Nodejsconf 2012WEBdeBS
 
Mockup, wireframe e visual: una breve introduzione
Mockup, wireframe e visual: una breve introduzioneMockup, wireframe e visual: una breve introduzione
Mockup, wireframe e visual: una breve introduzioneWEBdeBS
 
Djangoday lt 20120420
Djangoday lt 20120420Djangoday lt 20120420
Djangoday lt 20120420WEBdeBS
 
Unbit djangoday 20120419
Unbit djangoday 20120419Unbit djangoday 20120419
Unbit djangoday 20120419WEBdeBS
 
Django mongodb -djangoday_
Django mongodb -djangoday_Django mongodb -djangoday_
Django mongodb -djangoday_WEBdeBS
 
Virtualenv
VirtualenvVirtualenv
VirtualenvWEBdeBS
 
Django e il Rap Elia Contini
Django e il Rap Elia ContiniDjango e il Rap Elia Contini
Django e il Rap Elia ContiniWEBdeBS
 
PepperTweet - Project presentation Startup Weekend Brescia
PepperTweet - Project presentation Startup Weekend BresciaPepperTweet - Project presentation Startup Weekend Brescia
PepperTweet - Project presentation Startup Weekend BresciaWEBdeBS
 
Peppertweet - Presentazione Startup Weekend Brescia
Peppertweet - Presentazione Startup Weekend BresciaPeppertweet - Presentazione Startup Weekend Brescia
Peppertweet - Presentazione Startup Weekend BresciaWEBdeBS
 
NoSql Day - Chiusura
NoSql Day - ChiusuraNoSql Day - Chiusura
NoSql Day - ChiusuraWEBdeBS
 
NoSql Day - Apertura
NoSql Day - AperturaNoSql Day - Apertura
NoSql Day - AperturaWEBdeBS
 

Plus de WEBdeBS (14)

Legal Pirlo
Legal PirloLegal Pirlo
Legal Pirlo
 
Nodejsconf 2012
Nodejsconf 2012Nodejsconf 2012
Nodejsconf 2012
 
Mockup, wireframe e visual: una breve introduzione
Mockup, wireframe e visual: una breve introduzioneMockup, wireframe e visual: una breve introduzione
Mockup, wireframe e visual: una breve introduzione
 
Djangoday lt 20120420
Djangoday lt 20120420Djangoday lt 20120420
Djangoday lt 20120420
 
Unbit djangoday 20120419
Unbit djangoday 20120419Unbit djangoday 20120419
Unbit djangoday 20120419
 
Geodjango
GeodjangoGeodjango
Geodjango
 
Django mongodb -djangoday_
Django mongodb -djangoday_Django mongodb -djangoday_
Django mongodb -djangoday_
 
Virtualenv
VirtualenvVirtualenv
Virtualenv
 
Fagungis
FagungisFagungis
Fagungis
 
Django e il Rap Elia Contini
Django e il Rap Elia ContiniDjango e il Rap Elia Contini
Django e il Rap Elia Contini
 
PepperTweet - Project presentation Startup Weekend Brescia
PepperTweet - Project presentation Startup Weekend BresciaPepperTweet - Project presentation Startup Weekend Brescia
PepperTweet - Project presentation Startup Weekend Brescia
 
Peppertweet - Presentazione Startup Weekend Brescia
Peppertweet - Presentazione Startup Weekend BresciaPeppertweet - Presentazione Startup Weekend Brescia
Peppertweet - Presentazione Startup Weekend Brescia
 
NoSql Day - Chiusura
NoSql Day - ChiusuraNoSql Day - Chiusura
NoSql Day - Chiusura
 
NoSql Day - Apertura
NoSql Day - AperturaNoSql Day - Apertura
NoSql Day - Apertura
 

Dernier

A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersNicole Novielli
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxLoriGlavin3
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersRaghuram Pandurangan
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionDilum Bandara
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 

Dernier (20)

A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
A Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software DevelopersA Journey Into the Emotions of Software Developers
A Journey Into the Emotions of Software Developers
 
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptxUse of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
Use of FIDO in the Payments and Identity Landscape: FIDO Paris Seminar.pptx
 
Generative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information DevelopersGenerative AI for Technical Writer or Information Developers
Generative AI for Technical Writer or Information Developers
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko"Debugging python applications inside k8s environment", Andrii Soldatenko
"Debugging python applications inside k8s environment", Andrii Soldatenko
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
Advanced Computer Architecture – An Introduction
Advanced Computer Architecture – An IntroductionAdvanced Computer Architecture – An Introduction
Advanced Computer Architecture – An Introduction
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 

IGA Workflow Management & Analysis

  • 2. Introduction IGA Workflow is a web-based tool created using the Django Framework. It is intended as a management tool for the IGA wet-lab. We use it to track the lifecycle of biological samples, from vial to file.
  • 3. Overview ● Laboratory management, from sample to flowcell ● Bioinformatic analyses ● Pipelines management ● Technology ● Other applications and future developments
  • 5. Technology - Basics Database server ● Postgresql ● Redis Workers ● Celery Web server ● nginx + uWSGI
  • 6. Overview - Lab ● SAMPLE: The basic unit in the lab ● LIBRARY: a treated sample with an attached chemical TAG. ● POOL: a set of libraries, ready to be placed on a flowcell's lane.
  • 8. Overview - Lab The main challenge was to replace notebooks with a tool that allows to: ● insert samples, libraries, pools, and edit them; ● create lanes and runs and the configuration files for the physical sequencer; ● collect the sequencer results and map them in a easy way Almost done!
  • 9. Overview - Lab We started using the basic Django admin BUT ● the page loading was slow ● due to the admin's nature we lacked flexibility ● we were forcing the lab people procedures ● management was cumbersome
  • 10.
  • 11.
  • 12. Overview - Analyses After the physical sequencing the raw data (basecalls) must be converted in FASTQ files. The FASTQ files are FASTA files with some embedded quality stats. They are the starting point for almost every genomic analysis.
  • 13. Overview - Analyses To optimize time and resources we use a cluster of Celery workers. ● we track the software packages used ● we track their parameters ● we create a set of useful stats
  • 14.
  • 15. Technology Additions: ● Informational celery tasks (list directories content, copy files between devices with dbus plugin and UDisks instead of ugly hacks, ...)
  • 16. Overview - Pipelines FASTQ files alignments and assemblies Each analysis use different software in sequence or in parallel. Using hundreds of samples, the analyses can't be handled manually.
  • 17. Pipelines The results of each pipelines (like previous analyses) must be tracked. Since CLI based software is not user-friendly, we develop a graphical pipeline builder. Users are able to choose and combine different softwares to perform their own analyses.
  • 18. Pipeline ● Workers have different queues in order to satisfy different tasks ● Worker's tasks talk each other with Redis to avoid inconsistencies and to improve performances
  • 19. Pipelines Different queues (stored in Redis dbs): ○ available ○ queued ○ active ○ completed ○ error
  • 20. Pipelines Video demonstration of a working pipeline with use of different kind of steps. (1 video: 1 minute)
  • 21. Under development ● simple interface that allow customers to: ○ insert their samples directly ○ watch the results of their pipeline in a genome browser - also made with Django (see below)! ● barcodes ● genome browser (like GMOD GBrowse, but with the greatness of Python instead of the confusion of Perl)
  • 22. Genome browser An application that allows browsing a genome's annotations (like genes, or where reads are aligned). Actually, the best web genome browser is GMOD Gbrowse.
  • 23. Genome browser The challenge is to develop a genome browser that has a set of basic features and could accept plugins for particular type of data - like GMOD Gbrowse. In addition, it must be quick and easy to manage - NOT like GMOD Gbrowse.
  • 24. Genome browser Video demonstration of the genome browser. (2 videos: 1 minute + 1 minute)
  • 25. Acknowledgments ● The wet-lab Teams that developed: ladies at IGA ● Django ● JQuery ● nginx ● WEBdeBS ● uWSGI ● Celery ● Redis ● pip and virtualenv ● PostgreSQL ● All the open source projects involved