How can public cloud and technologies like Docker and OpenStack help to deliver next generation scientific computing infrastructure? My talk for the UK/USA HPC workshop in July 2015, organized by HPC-SIG (UK) and CASC (USA).

1. Cloud for Research and Innovation
Martin Hamilton
Photo CC BY Flickr user atosorigin 1Cloud for Research and Innovation - UK USA HPC Workshop July 201510/07/2015

2. Cloud for research and innovation
1. Cloud adoption in enterprise IT
– Trends in UK higher education
– Bimodal IT versus bipolar institutions
2. Public clouds are starting to look more like HPC clusters
– Orchestration, Infiniband, GPU and FPGA availability (at a price)
3. Code re-use and app portability
– Will Docker and OpenStack be our Rosetta Stone(s)?
– What does a unified computing model look like?

3. 1. Cloud adoption
in enterprise IT
10/07/2015 Cloud for Research and Innovation - UK USA HPC Workshop July 2015 3

4. Cloud in UK higher education
Collaboration suites: Google Apps for Education
» The University of Westminster estimates that it saved £1m by moving to Google Apps for
Education for email and other online collaboration facilities. They also state that “an additional
benefit has been the reduced time spent in systems and user support with a minimal number of
calls for support for such a significant system”, and that moving to the cloud has “liberated staff and
students from the smaller storage limits of the previous in-house solution.”
http://services.google.com/fh/files/misc/univofwestminster.pdf
» Ealing, Hammersmith and West London College have also migrated to Google Apps.They observe
that “each year, 20,000 new accounts are created providing 600 terabytes (around 150,000
DVDs worth) of storage in less than five minutes.” Subsequently to this article being written,
Google moved to offering free unlimited storage.
http://www.wlc.ac.uk/college/news/article.asp?newsID=4228

5. Cloud in UK higher education
Collaboration suites: Microsoft Office365
» The University of theWest of London moved its 14,000 students to Office365, and observes that
once a student graduates “they can maintain that Office365 account through the University for
life.We can continue to supply them with information about the University, help them find their
second job, their third job and so on.This isn’t just about collaboration while studying, it’s about
creating a life-long connection between the students and the University.”
http://www.slideshare.net/Microsofteduk/university-of-west-london-case-study-office-365
» The University of Dundee migrated all of its 22,000 student accounts to Microsoft’s Office365 for
Education service over a week.They note that this was achieved with zero down time, and estimate
that Office365 “will save us at least £500,000 in infrastructure and staffing over five years.”
http://www.slideshare.net/Microsofteduk/university-of-dundee-case-study-office-365

6. Cloud in UK higher education
Jisc Amazon cloud portal:
› Monthly invoicing - credit cards are no longer
required for payment
› Itemized billing - consolidated across
users/departments
› Billing inGBP, not dollars
› Setting of budget limits for individual user
accounts or departments
› The retrieval of service usage information within
own areas of responsibility
› Volume-discounts through aggregation across
multiple educational institutions

7. Cloud in UK higher education
Leveraging the Janet network:

8. Cloud in UK higher education
Private peerings:

9. Cloud in UK higher education
Private peerings:

10. Cloud in UK higher education
From www.jasmin.ac.uk

11. Cloud in UK higher education
Dogfooding:
› We moved
our own core
infrastructure
to AWS
› Estimate that
this saved
£100K p.a.
› Cloud first /
mobile first
for new stuff
› NB we are not
an HPC
provider

12. 2. Public clouds are
starting to look like HPC clusters
10/07/2015 Cloud for Research and Innovation - UK USA HPC Workshop July 2015 12

13. Case study: Azure HPC

14. Case study: Azure HPC

15. Case study: Azure HPC

16. Case study: Azure HPC
http://azure.microsoft.com/blog/2015/07/09/azure-linux-rdma-hpc-available/

17. Case study: Azure HPC

18. Case study: Azure HPC

19. Case study: Azure HPC

20. Case study: Azure HPC

21. Case study: Google Compute Engine

22. Case study: AWS

23. Case study: AWS

24. Caveat Emptor: Magellan
http://science.energy.gov/~/media/ascr/pdf/program-
documents/docs/Magellan_Final_Report.pdf

25. Caveat Emptor: Magellan
http://science.energy.gov/~/media/ascr/pdf/program-
documents/docs/Magellan_Final_Report.pdf

26. 3. Code re-use &
application portability
10/07/2015 Cloud for Research and Innovation - UK USA HPC Workshop July 2015 26

27. Re-use & portability: OpenStack
http://www.datacenterknowledge.com/archives/2014/11/07/ce
rns-openstack-cloud-to-reach-150000-cores-by-2015/

28. Re-use & portability: OpenStack
http://www.datacenterknowledge.com/archives/2014/11/07/ce
rns-openstack-cloud-to-reach-150000-cores-by-2015/
http://www.datanami.com/2013/09/23/cern_turns_to_google_
for_datacenter_direction/

29. Re-use & portability: OpenStack

30. Re-use & portability: OpenStack

31. Re-use & portability: Containers

32. Re-use & portability: Containers
https://wiki.openstack.org/wiki/Docker

33. Re-use & portability: Performance
http://en.community.dell.com/techcenter/high-performance-computing/b/general_hpc/archive/2014/11/04/containers-docker-virtual-machines-and-hpc

34. Re-use & portability: Open Container Project

35. Re-use & app portability: Kubernetes & Heat
http://kubernetes.io http://techs.enovance.com/7104/multi-tenant-docker-with-openstack-heat

36. Re-use & app portability: A unified model?
This is one model. Other models are available 
› Run OpenStack on your in-house facility [Q: Still need one?]
– This creates your own private/community cloud
– Machine images can be programmatically converted to/from QCOW2
› Package your app with Docker and you can run it anywhere
– Embraced by public cloud providers, can run in-house too as open source
– Could be bare metal or on top of e.g. OpenStack [Q: Performance?]
› Lots of options for orchestration from general purpose uses
– Kubernetes scheduling particularly interesting from an HPC perspective
› Open questions from an e-Infrastructure perspective:
– Integrating identity management, authentication, authorization, accounting
– User driven (instant gratification) versus controlled environment
– Funding model for use of public and community clouds, potential for national deals
– Lots of new stuff to learn about! (for developer, sysadmin, …)
– That’s all fine for the compute, but what about the data?
Photo CC-BY Flickr
user codep8

37. That’s all folks…
37
Except where otherwise noted, this
work is licensed under CC-BY
Martin Hamilton
Futurist, Jisc, London
@martin_hamilton
martin.hamilton@jisc.ac.uk
Cloud for Research and Innovation - UK USA HPC Workshop July 201510/07/2015