My Hadoop World presentation

1. Hadoop for Bioinformatics
Deepak Singh
Amazon Web Services
Hadoop World, NYC

3. Via Reavel under a CC-BY-NC-ND license

4. By ~Prescott under a CC-BY-NC license

5. data sets

6. many data sets

7. PFAM PDB
GENBANK ENSEMBL
Many Others

8. manageable

9. Image: Matt Wood

10. Human
genom
e
Image: Matt Wood

13. Image: Matt Wood

14. ~100 TB/Week
Image: Matt Wood

15. ~100 TB/Week
>2 PB/Year
Image: Matt Wood

17. years

18. days

19. hours

20. gigabytes

21. terabytes

22. petabytes

23. really fast

25. typical informatics workflow

30. Via Christolakis under a CC-BY-NC-ND license

31. Via Argonne National Labs under a CC-BY-SA license

32. killer app
Via Argonne National Labs under a CC-BY-SA license

33. Via asklar under a CC-BY license

36. Image: Chris Dagdigian

38. rethink algorithms

39. rethink computing

40. rethink data management

41. rethink data sharing

42. operational mindset

43. scalability

44. we are data geeks not data center geeks

45. two key trends

49. develop applications

50. distribute applications

51. use applications

52. some work

53. filters
some work
^

54. High Throughput Sequence Analysis
Mike Schatz, University of Maryland

55. • Read Mapping
• Mapping & SNP Discovery
• De novo Genome Assembly

56. Short Read Mapping

57. Asian Individual Genome: 3.3 Billion 35bp, 104
GB (Wang et al., 2008)
African Individual Genome: 4.0 Billion 35bp, 144
GB (Bentley et al., 2008)

58. Alignment > 10000 CPU hrs

59. Seed & Extend
Good alignments must have significant
exact alignment
Minimal exact alignment length = l/(k+1)

60. Seed & Extend
Good alignments must have significant
exact alignment
Minimal exact alignment length = l/(k+1)
Expensive to scale

61. Seed & Extend
Good alignments must have significant
exact alignment
Minimal exact alignment length = l/(k+1)
Expensive to scale

62. Seed & Extend
Good alignments must have significant
exact alignment
Minimal exact alignment length = l/(k+1)
Expensive to scale
Need parallelization framework

63. CloudBurst
Catalog k-mers Collect seeds End-to-end alignment

64. http://cloudburst-bio.sourceforge.net; Bioinformatics 2009 25: 1363-1369

66. CloudBurst efficiently reports every k-difference
alignment of every read

67. many applications only need the best alignment

68. Bowtie: Ultrafast short read aligner
Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efficient alignment of short DNA sequences to the human
genome. Genome Biol 10 (3): R25.

69. SOAPSnp: Consensus alignment and SNP calling
Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efficient alignment of short DNA sequences to the human
genome. Genome Biol 10 (3): R25.

70. Crossbow: Rapid whole genome SNP analysis
Ben Langmead
Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efficient alignment of short DNA sequences to the human
genome. Genome Biol 10 (3): R25.

72. Preprocessed reads

73. Preprocessed reads
Map: Bowtie

74. Preprocessed reads
Map: Bowtie
Sort: Bin and partition

75. Preprocessed reads
Map: Bowtie
Sort: Bin and partition
Reduce: SoapSNP

76. Crossbow condenses over 1,000 hours
of resequencing computation into a few
hours without requiring the user to own
or operate a computer cluster

77. Comparing Genomes

78. Estimating relative evolutionary rates
from sequence comparisons:
Identification of probable orthologs
Admissible comparisons: A or B vs. D
C vs. E
Inadmissible comparisons: A or B vs. E
C vs. D
A B C D E species tree
gene tree
S. cerevisiae C. elegans

79. Estimating relative evolutionary rates
from sequence comparisons:
1. Orthologs found using the Reciprocal
smallest distance algorithm
2. Build alignment between two orthologs
>Sequence C
MSGRTILASTIAKPFQEEVTKAVKQLNFT-----PKLVGLLSNEDPAAKMYANWTGKTCESLGFKYEL-…
>Sequence E
MSGRTILASKVAETFNTEIINNVEEYKKTHNGQGPLLVGFLANNDPAAKMYATWTQKTSESMGFRYDL…
3. Estimate distance given a substitution
matrix
Phe Ala Pro Leu Thr
Phe
Ala µπ
Pro µπ µπ µπ
Leu µπ µπ µπ µπ
A B C D E species tree
gene tree
S. cerevisiae C. elegans

80. RSD algorithm summary
Genome I Genome J
Ib Jc
Align sequences &
Calculate distances L Orthologs:
Align sequences &
Calculate distances H
ib - jc D = 0.1
c
vs. D=1.2 vs. D=0.2
a b a
vs. D=0.1 vs. D=0.3
c b b b
vs. D=0.9 vs. D=0.1
c c b c

81. Prof. Dennis Wall
Harvard Medical School

82. Roundup is a database of orthologs
and their evolutionary distances.
To get started, click browse. Alternatively, you can
read our documentation here.
Good luck, researchers!

83. massive computational demand

84. 1000 genomes = 5,994,000 processes =
23,976,000 hours

85. 2737 years

86. periodic task

87. must scale up

88. not scalability gurus

89. hadoop streaming

91. compared 50+ genomes

92. what’s next?

93. de novo assembly

94. machine learning and statistics

95. protein structure prediction

96. docking

97. trajectory analysis

98. key driving factors?

99. the ecosystem

100. Pig

101. Cascading

102. Hive

103. RHIPE

104. domain specific libraries and tools

105. http://aws.amazon.com/publicdatasets/

107. http://aws.amazon.com/education/

109. Thank you!
deesingh@amazon.com; Twitter:@mndoci
Presentation ideas from @mza, @simon and @lessig