"Petascale Genomics with Spark", Sean Owen,Director of Data Science at Cloudera
•Télécharger en tant que PPTX, PDF•
2 j'aime•1,193 vues
"Petascale Genomics with Spark", Sean Owen, Director of Data Science at Cloudera YouTube Link: https://www.youtube.com/watch?v=HY93FdK5i60 Watch more from Data Natives 2015 here: http://bit.ly/1OVkK2J Visit the conference website to learn more: www.datanatives.io Follow Data Natives: https://www.facebook.com/DataNatives https://twitter.com/DataNativesConf Stay Connected to Data Natives by Email: Subscribe to our newsletter to get the news first about Data Natives 2016: http://bit.ly/1WMJAqS About the Author: Sean is Director of Data Science at Cloudera, based in London. Before Cloudera, he founded Myrrix Ltd, a company commercializing large-scale real-time recommender systems on Apache Hadoop. He has been a primary committer and VP for Apache Mahout, and co-author of Mahout in Action. Previously, Sean was a senior engineer at Google. He holds and MBA from the London Business School and a BA in Computer Science from Harvard.
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à "Petascale Genomics with Spark", Sean Owen,Director of Data Science at Cloudera
Similaire à "Petascale Genomics with Spark", Sean Owen,Director of Data Science at Cloudera (20)
Plus de Dataconomy Media
Plus de Dataconomy Media (20)
Dernier
Dernier (20)
"Petascale Genomics with Spark", Sean Owen,Director of Data Science at Cloudera
- 1. 1© Cloudera, Inc. All rights reserved. The Redemptive Power of Hadoop Sean Owen | @sean_r_owen Scaling Up Genomics with Spark
- 2. 2© Cloudera, Inc. All rights reserved. What is genomics?
- 3. 3© Cloudera, Inc. All rights reserved. Organism Cell Genome
- 4. 4© Cloudera, Inc. All rights reserved.
- 5. 5© Cloudera, Inc. All rights reserved. Reference chromosome Location
- 6. 6© Cloudera, Inc. All rights reserved. “… decoding the Book of Life”
- 7. 7© Cloudera, Inc. All rights reserved.
- 8. 8© Cloudera, Inc. All rights reserved.
- 9. 9© Cloudera, Inc. All rights reserved.
- 10. 10© Cloudera, Inc. All rights reserved.
- 11. 11© Cloudera, Inc. All rights reserved.
- 12. 12© Cloudera, Inc. All rights reserved.
- 13. 13© Cloudera, Inc. All rights reserved. >read1 TTGGACATTTCGGGGTCTCAGATT >read2 AATGTTGTTAGAGATCCGGGATTT >read3 GGATTCCCCGCCGTTTGAGAGCCT >read4 AGGTTGGTACCGCGAAAAGCGCAT Bioinformatics!
- 14. 14© Cloudera, Inc. All rights reserved. Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation Pipelines!
- 15. 15© Cloudera, Inc. All rights reserved. C HPC (scheduler) POSIX filesystem Java HPC (Queue) POSIX filesystem C++ Single-node SQLite It’s file formats all the way down!
- 16. 16© Cloudera, Inc. All rights reserved. ##fileformat=VCFv4.1 ##fileDate=20090805 ##source=myImputationProgramV3.1 ##reference=file:///seq/references/1000GenomesPilot-NCBI36.fasta ##contig=<ID=20,length=62435964,assembly=B36,md5=f126cdf8a6e0c7f379d618ff66beb2da,species="Homo sapiens",taxonomy=x> ##phasing=partial ##INFO=<ID=NS,Number=1,Type=Integer,Description="Number of Samples With Data"> ##INFO=<ID=DP,Number=1,Type=Integer,Description="Total Depth"> ##INFO=<ID=AF,Number=A,Type=Float,Description="Allele Frequency"> ##INFO=<ID=AA,Number=1,Type=String,Description="Ancestral Allele"> ##INFO=<ID=DB,Number=0,Type=Flag,Description="dbSNP membership, build 129"> ##INFO=<ID=H2,Number=0,Type=Flag,Description="HapMap2 membership"> ##FILTER=<ID=q10,Description="Quality below 10"> ##FILTER=<ID=s50,Description="Less than 50% of samples have data"> ##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype"> ##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality"> ##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Read Depth"> ##FORMAT=<ID=HQ,Number=2,Type=Integer,Description="Haplotype Quality"> #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT NA00001 NA00002 NA00003 20 14370 rs6054257 G A 29 PASS NS=3;DP=14;AF=0.5;DB;H2 GT:GQ:DP:HQ 0|0:48:1:51,51 1|0:48:8:51,51 1/1:43:5:.,. 20 17330 . T A 3 q10 NS=3;DP=11;AF=0.017 GT:GQ:DP:HQ 0|0:49:3:58,50 0|1:3:5:65,3 0/0:41:3 20 1110696 rs6040355 A G,T 67 PASS NS=2;DP=10;AF=0.333,0.667 GT:GQ:DP:HQ 1|2:21:6:23,27 2|1:2:0:18,2 2/2:35:4 20 1230237 . T . 47 PASS NS=3;DP=13;AA=T GT:GQ:DP:HQ 0|0:54:7:56,60 0|0:48:4:51,51 0/0:61:2 Compressed text files (non-splittable) Semi-structured Poorly specified Global sort order
- 17. 17© Cloudera, Inc. All rights reserved. /** * Main work method. Reads the BAM file once and collects sorted information about * the 5' ends of both ends of each read (or just one end in the case of pairs). * Then makes a pass through those determining duplicates before re-reading the * input file and writing it out with duplication flags set correctly. */ protected int doWork() { // build some data structures buildSortedReadEndLists(useBarcodes); generateDuplicateIndexes(useBarcodes); final SAMFileWriter out = new SAMFileWriterFactory().makeSAMOrBAMWriter(outputHeader, true, OUTPUT); final CloseableIterator<SAMRecord> iterator = headerAndIterator.iterator; while (iterator.hasNext()) { final SAMRecord rec = iterator.next(); if (!rec.isSecondaryOrSupplementary()) { if (recordInFileIndex == nextDuplicateIndex) { rec.setDuplicateReadFlag(true); // Now try and figure out the next duplicate index if (this.duplicateIndexes.hasNext()) { nextDuplicateIndex = this.duplicateIndexes.next(); } else { // Only happens once we've marked all the duplicates nextDuplicateIndex = -1; } } else { Method Code
- 18. 18© Cloudera, Inc. All rights reserved. @Option(shortName = "MAX_FILE_HANDLES", doc = "Maximum number of file handles to keep open when spilling " + "read ends to disk. Set this number a little lower than the " + "per-process maximum number of file that may be open. This " + "number can be found by executing the 'ulimit -n' command on " + "a Unix system.") public int MAX_FILE_HANDLES_FOR_READ_ENDS_MAP = 8000; Dedup Method Code Platform
- 19. 19© Cloudera, Inc. All rights reserved. Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation
- 20. 20© Cloudera, Inc. All rights reserved. It’s pipelines all the way down! Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation Node 1 Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation Node 2 Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation Node 3
- 21. 21© Cloudera, Inc. All rights reserved.
- 22. 22© Cloudera, Inc. All rights reserved. Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation Alignment Dedup Recalibrate QC/Filter Alignment Dedup Recalibrate QC/Filter
- 23. 23© Cloudera, Inc. All rights reserved. Node 1 Alignment Dedup Recalibrate QC/Filter Variant Calling Variant Annotation Node 2 Node 3 Alignment Dedup Recalibrate QC/Filter Alignment Dedup Recalibrate QC/Filter Node 4
- 24. 24© Cloudera, Inc. All rights reserved. Node 1 Alignment Dedup QC/Filter Variant Calling Variant Annotation Node 2 Node 3 Alignment Dedup QC/Filter Alignment Dedup QC/Filter Node 4 Recalibrate
- 25. 25© Cloudera, Inc. All rights reserved. Why Are We Still Defining File Formats By Hand? • Instead of defining custom file formats for each data type and access pattern… • Parquet creates a compressed format for each Avro-defined data model • Improvements over existing formats • ~20% for BAM • ~90% for VCF
- 26. 26© Cloudera, Inc. All rights reserved. YARN-managed Hadoop cluster Spark executors 𝑗=1 𝑑 𝑖 𝑃(𝑏𝑖𝑗|𝑒𝑖𝑗, 𝑓𝑖) 𝑗=1 𝑑 𝑖 𝑃(𝑏𝑖𝑗|𝑒𝑖𝑗, 𝑓𝑖) 𝑗=1 𝑑 𝑖 𝑃(𝑏𝑖𝑗|𝑒𝑖𝑗, 𝑓𝑖)Partial sums 𝑖=1 𝑁 𝑗=1 𝑑 𝑖 𝑃(𝑏𝑖𝑗|𝑒𝑖𝑗, 𝑓𝑖) Driver Application code ContEst Algorithm
- 27. 27© Cloudera, Inc. All rights reserved. Hadoop provides layered abstractions for data processing HDFS (scalable, distributed storage) YARN (resource management) MapReduce Impala (SQL) Solr (search) Spark ADAMquince guacamole … bdg-formats(Avro/Parquet)
- 28. 28© Cloudera, Inc. All rights reserved. Executing query in Hadoop: interactive Spark shell (ADAM) def inDbSnp(g: Genotype): Boolean = true or false def isDeleterious(g: Genotype): Boolean = g.getPolyPhen val samples = sc.textFile("path/to/samples").map(parseJson(_)).collect() val dbsnp = sc.textFile("path/to/dbSNP").map(_.split(",")).collect() val dnaseRDD = sc.adamBEDFeatureLoad("path/to/dnase”) val genotypesRDD = sc.adamLoad("path/to/genotypes") val filteredRDD = genotypesRDD .filter(!inDbSnp(_)) .filter(isDeleterious(_)) .filter(isFramingham(_)) val joinedRDD = RegionJoin.partitionAndJoin(sc, filteredRDD, dnaseRDD) val maf = joinedRDD .keyBy(x => (x.getVariant, getPopulation(x))) .groupByKey() .map(computeMAF(_)) maf.saveAsNewAPIHadoopFile("path/to/output") apply predicates load data join data group-by aggregate (MAF) persist data
- 29. 29© Cloudera, Inc. All rights reserved. • Hosted at Berkeley and the AMPLab • Apache 2 License • Contributors from both research and commercial organizations • Core spatial primitives, variant calling • Avro and Parquet for data models and file formats Spark + Genomics = ADAM
- 30. 30© Cloudera, Inc. All rights reserved. Core Genomics Primitives: Spatial Join
- 31. 31© Cloudera, Inc. All rights reserved. ADAM preliminary performance
- 32. 32© Cloudera, Inc. All rights reserved.
- 33. 33© Cloudera, Inc. All rights reserved. Acknowledgements UCBerkeley Matt Massie Frank Nothaft Michael Heuer Tamr Timothy Danford MSSM Jeff Hammerbacher Ryan Williams Cloudera Uri Laserson Tom White Sandy Ryza
- 34. 34© Cloudera, Inc. All rights reserved. Thank you @sean_r_owen sowen@cloudera.com
Notes de l'éditeur
- Before we dive in, let me ask a couple of questions: Biologists? Spark experts? There are always at least three different constituencies in the room: * biologists * programmers * someone thinking about how to build a business around this Gonna tell you a lot of lies today. Won’t satisfy everyone. Where I skip over the truth, maybe there will be at least a breadcrumb of truth left over. This will not be a very technical talk.
- What even is genomics? Who here has heard the terms ‘chromosome’ and ‘gene’ before, and could explain the difference? So before we dive into the main part of the talk, I’m going to spend a few minutes discussing some of the basic biological concepts.
- But each of those cells has, ideally, an identical genome. The genome is a collection of 23 linear molecules. These are called ‘polymers,’ they’re built (like Legos) out of a small number of repeated interlocking parts – these are the A, T, G, and C you’ve probably heard about. The content of the genome is determined by the linear order in which these letters are arranged. (Linear is important!)
- Without losing much, assume that our genomes are contained on just a single chromosome. Now, not only do all the cells in your body have identical genomes… [ADVANCE]
- We can define a concept of ‘location’ across chromosomes. This is possibly the most important concept in genome informatics, the idea that DNA defines a common linear coordinate system. This also means that we can talk about differences between individuals in terms of diffs to a common reference genome. But where does this reference genome come from?
- Here is Bill Clinton (and Craig Venter and Francis Collins), announcing in June of 2000 the “rough draft” of the Human Genome – this is the Human Genome Project. Took >10 years and $2 billion What did this actually do?
- Anyone recognize this? Genome analogy: a text file a part of the linear sequence of ACGTs. Difficult to understand.
- Mapmakers work to add ANNOTATIONS to the map.
- And often, it’s only the annotations that are interesting, so mapmakers focus on *annotation* of the maps themselves. The core technologies are 2D planar and spherical geometry, geometric operations composed out of latitudes and longitudes. What does the annotated map of the genome look like?
- Chromosome on top. Highlighted red portion is what we’re zoomed in on. See the scale: total of about 600,000 bases (ACGTs) arranged from left to right. Multiple annotation “tracks” are overlaid on the genome sequence, marking functional elements, positions of observed human differences, similarity to other animals. In part it’s the product of numerous additional large biology annotation projects (e.g., HapMap project, 1000 Genomes, ENCODE). Lot's of bioinformatics is computing these elements, or evaluating models on top of the elements. How are these annotations actually generated? Shift gears and talk about the technology.
- DNA SEQUENCING If satellites provide images of the world for cartography, sequences are the microscopes that give you “images” of the genome. Over past decade, massive EXPONENTIAL increase in throughput (much faster than Moore’s law)
- Bioinformatics is the computational process to reconstruct the genomic information. But… [ADVANCE]
- Pipelines, of course. Example pipeline: raw sequencing data => a single individual’s “diff” from the reference. How are these typically structured? Each step is typically written as a standalone program – passing files from stage to stage These are written as part of a globally-distrbuted research program, by researchers and grad students around the world, who have to assume the lowest common denominator: command line and filesystem What does one of these files look like?
- Bioinformaticians LOVE hand-coded file formats. But only store several fundamental data types. Strong assumptions in the formats. Inconsistent implementations in multiple languages. Doesn’t allow different storage backends. OK, we discussed what the data/files are like that are passed around. What about the computation itself?
- Imposes severe constraint: global sort invariant. => Many impls depend on this, even if it’s not necessary or conducive to distributed computing.
- But what if we jump into one of these functions. You’ll find a dependence on… [ADVANCE]
- Most bioinformatics tools make strong assumptions about their environments, and also the structure of the data (e.g., global sort), when it shouldn’t be necessary. Ok, but that’s not all… [ADVANCE]
- We’ve looked at the data and a bit of code for one of these tools. But this runs the pipeline on a single individual. But of course, it’s never one pipeline… [ADVANCE]
- Scale out! Typically managed with a pretty low-level job scheduler.
- SCALE! New levels of ambition for large biology projects. 100k genomes at Genomics England in collaboration with National Health Service. Raw data for a single individual can be in the hundreds of GB But even before we hit that huge scale (which is soon)…
- We don’t want to analyze each sample separately. We want to use ALL THE DATA we generate. Well, these pipelines often include lots of aggregation, perhaps we can just… [ADVANCE]
- Do the easy thing! Not ideal, especially as the amount of data goes up (data transfer), number of files increases (we saw file handles). May start hitting the cracks. But even worse… [ADVANCE]
- God help you if you want to jointly use all the data in earlier part of the pipeline. 2 Problems: Large scale Using all data simultaneously
- Things like global sort order are overly restrictive and leads to algos relying on it when it’s not necessary.
- Example of an algo. Bioinformatics loves evaluating probabilistic models on the chromosomes. We can easily extract parallelism at different parts of our pipelines. Use higher level distributed computing primitives and let the system figure out all the platform issues for you: storage, job scheduling, fault tolerance, shuffles.
- Cheap scalable STORAGE at bottom Resource management middle EXECUTION engines that can run your code on the cluster and provide parallelism Consistent SERIALIZATION framework Scientist should NOT WORRY about lower levels (coordination, file formats, storage details, fault tolerance)
- Another computation for a statistical aggregate on genome variant data. Details not important. Spark data flow: Distributed data load High level joins/spatial computations that are parallelized as necessary. But really nice thing is because our data is stored using the Avro data model… [ADVANCE]
- We’ve implemented this vision with Spark, starting from the Amplab (same people that gave you Spark) into a project called ADAM The reason this works is that Spark naturally handles pipelines, and automatically performs shuffles when appropriate, but also…
- In addition to some of the standard pipeline transformations, implemented the core spatial join operations (analogous to a geospatial library).
- Single-node performance improvements. Free scalability: fixed price, significant wall-clock improvements See most recent SIGMOD.
- Not to be outdone, Craig Venter proposes 1 million genomes at Human Longevity Inc.
- Cloudera is hiring. Including the data science team.