Clouds, Grids and Data
•Télécharger en tant que ODP, PDF•
1 j'aime•606 vues
The Next-Generation sequencing data-deluge requires storage and compute services to be provisioned at an ever-increasing rate. Can Cloud (and last decade's buzzword, Grid), help us? Talk given at the NHGRI Cloud computing workshop, 2010.
![Clouds, Grids and Data Guy Coates Wellcome Trust Sanger Institute [email_address]](https://image.slidesharecdn.com/nhgri-sanger-110301082935-phpapp02/85/Clouds-Grids-and-Data-1-320.jpg)
![[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Clouds, Grids and Data
Similaire à Clouds, Grids and Data (20)
Dernier
Dernier (20)
Clouds, Grids and Data
- 1. Clouds, Grids and Data Guy Coates Wellcome Trust Sanger Institute [email_address]
- 7. Past Collaborations Data Sequencing Centre + DCC Sequencing centre Sequencing centre Sequencing centre Sequencing centre
- 8. Future Collaborations Collaborations are short term: 18 months-3 years. Sequencing Centre 3 Sequencing Centre 1 Sequencing Centre 2A Sequencing Centre 2B Federated access
- 9. Genomics Data Unstructured data (flat files) Data size per Genome Structured data (databases) DAS, bioMART etc ? Intensities / raw data (2TB) Alignments (200 GB) Sequence + quality data (500 GB) Variation data (1GB) Individual features (3MB)
- 13. Some will have patient identifiable data.
- 14. Plan for it now.
- 16. iRODS ICAT Catalogue database Rule Engine Implements policies Irods Server Data on disk User interface WebDAV, icommands,fuse Irods Server Data in database Irods Server Data in S3
- 19. Allows user at institute A to seamlessly access data at institute B in a controlled manner.
- 22. Controlled data is hard:
- 23. Encrypt files and place on private FTP dropboxes.
- 25. Software knows about S3 storage layers.
- 27. Culture shock.
- 29. Single sign on?
- 31. Cloud Archives
- 33. Is data in an inaccessible archive really useful?
- 36. 3 month lead time.
- 37. ~$1.5M capex.
- 38. Elephant in the room
- 41. NCBI -> Sanger: 15 Mbyte/s (120 Mbit/s)
- 43. 20 days to pull down 100TB from Oxford.
- 47. Put VMs on compute that is “attached” to the data. Data CPU CPU CPU CPU Data CPU CPU CPU CPU VM
- 48. Proto-Example: Ssaha trace search Hash Table (320 GB) trace Database ~30TB 1. hash database CPU CPU CPU CPU hash hash hash hash 2 .Distribute hash across machines query 3. Run query in parallel
- 55. We have effectively tied ourselves to a single provider.
- 56. Compute architecture VS CPU CPU CPU Fat Network Posix Global filesystem CPU CPU CPU CPU thin network Local storage Local storage Local storage Local storage Batch schedular hadoop/S3 Data-store Data-store
- 59. ...then beowulf took over the world.
- 62. The challenge is computing across the data at scale.
- 63. Network infrastructure and cloud architectures still problematic.
- 66. Gen-Tao Chiang
- 67. Pete Clapham
- 70. John Teague
- 72. Backup
- 76. Common workload.
- 78. Ensembl / Annotation TCCTCTCTTTATTTTAGCTGGACCAGACCAATTTTGAGGAAAGGATACAGACAGCGCCTG GAATTGTCAGACATATACCAAATCCCTTCTGTTGATTCTGCTGACAATCTATCTGAAAAA TTGGAAAGGTATGTTCATGTACATTGTTTAGTTGAAGAGAGAAATTCATATTATTAATTA TTTAGAGAAGAGAAAGCAAACATATTATAAGTTTAATTCTTATATTTAAAAATAGGAGCC AAGTATGGTGGCTAATGCCTGTAATCCCAACTATTTGGGAGGCCAAGATGAGAGGATTGC TTGAGACCAGGAGTTTGATACCAGCCTGGGCAACATAGCAAGATGTTATCTCTACACAAA ATAAAAAAGTTAGCTGGGAATGGTAGTGCATGCTTGTATTCCCAGCTACTCAGGAGGCTG AAGCAGGAGGGTTACTTGAGCCCAGGAGTTTGAGGTTGCAGTGAGCTATGATTGTGCCAC TGCACTCCAGCTTGGGTGACACAGCAAAACCCTCTCTCTCTAAAAAAAAAAAAAAAAAGG AACATCTCATTTTCACACTGAAATGTTGACTGAAATCATTAAACAATAAAATCATAAAAG AAAAATAATCAGTTTCCTAAGAAATGATTTTTTTTCCTGAAAAATACACATTTGGTTTCA GAGAATTTGTCTTATTAGAGACCATGAGATGGATTTTGTGAAAACTAAAGTAACACCATT ATGAAGTAAATCGTGTATATTTGCTTTCAAAACCTTTATATTTGAATACAAATGTACTCC