Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à 20140509 cern open_stack_linuxtag_v3
Similaire à 20140509 cern open_stack_linuxtag_v3 (20)
Plus de Tim Bell
Plus de Tim Bell (20)
Dernier
Dernier (20)
20140509 cern open_stack_linuxtag_v3
- 3. 09/05/2014 LinuxTag 2014 3 CERN was founded 1954: 12 European States “Science for Peace” Today: 21 Member States Member States: Austria, Belgium, Bulgaria, the Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Israel, Italy, the Netherlands, Norway, Poland, Portugal, Slovakia, Spain, Sweden, Switzerland and the United Kingdom Candidate for Accession: Romania Associate Members in Pre-Stage to Membership: Serbia Applicant States for Membership or Associate Membership: Brazil, Cyprus (awaiting ratification), Pakistan, Russia, Slovenia, Turkey, Ukraine Observers to Council: India, Japan, Russia, Turkey, United States of America; European Commission and UNESCO ~ 2,300 staff ~ 1,000 other paid personnel > 11,000 users Budget (2013) ~1,000 MCHF
- 8. A Big Data Challenge 09/05/2014 LinuxTag 2014 8 In 2014, • 100PB archive with additional 35PB/year • 10,000 servers • 75,000 disk drives • 45,000 tapes In 2015, • Run 2 of LHC expected to double data rates • But many limits and limitations…
- 9. The CERN Meyrin Data Centre 09/05/2014 LinuxTag 2014 9
- 10. 09/05/2014 LinuxTag 2014 10
- 11. 09/05/2014 Bamboo Koji, Mock AIMS/PXE Foreman Yum repo Pulp Puppet-DB mcollective, yum JIRA Lemon / Hadoop / LogStash / Kibana git OpenStack Nova Hardware database Puppet Active Directory / LDAP LinuxTag 2014 11
- 12. Status • Multi-data centre cloud in production since July 2013 (Geneva and Budapest) • Currently running OpenStack Havana • KVM and Hyper-V deployed • All configured automatically with Puppet • 65,000 cores in CERN IT Private Cloud • 3PB Ceph pool available for volumes, images and other physics storage 09/05/2014 LinuxTag 2014 12
- 13. 09/05/2014 LinuxTag 2014 13 Microsoft Active Directory CERN DB on Demand CERN Network Database Account mgmt system Horizon Keystone Glance Network Compute Scheduler Cinder Nova Block Storage Ceph & NetApp CERN Accounting Ceilometer
- 14. Monitoring - Flume, Elastic Search, Kibana 14 HDFS Flume gateway elasticsearch Kibana OpenStack infrastructure
- 15. compute-nodescontrollers compute-nodes Scaling Architecture Overview 15 Child Cell Geneva, Switzerland Child Cell Budapest, Hungary Top Cell - controllers Geneva, Switzerland Load Balancer Geneva, Switzerland controllers
- 16. Architecture Components 16 rabbitmq - Keystone - Nova api - Nova conductor - Nova scheduler - Nova network - Nova cells - Glance api - Ceilometer agent-central - Ceilometer collector Controller - Flume - Nova compute - Ceilometer agent-compute Compute node - Flume - HDFS - Elastic Search - Kibana - MySQL - MongoDB - Glance api - Glance registry - Keystone - Nova api - Nova consoleauth - Nova novncproxy - Nova cells - Horizon - Ceilometer api - Cinder api - Cinder volume - Cinder scheduler rabbitmq Controller Top Cell Children Cells - Stacktach - Ceph - Flume
- 17. Some Caution on Cells • Single cell limits around 1,000 hypervisors • Can be adapted using Bluehost alternative approach with MySQL replication • Significant function gap being worked on • Flavors, Availability zones, Scheduling, Ceilometer need workarounds • Tested in the OpenStack gate • Not blocking so local QA environment needed 09/05/2014 LinuxTag 2014 17
- 18. Scheduling at Scale • CERN users want more sophisticated scheduling: • Processor architecture • Private network subnets • Varying memory/core/disk ratios • Hardware with more redundancy • Servers should be used fully • Tetris-like problem to find the matches • Packing is more difficult the nearer to 100% used • Cells scheduler is rather simple currently • Try Cell X, if not match, try Cell Y… 09/05/2014 LinuxTag 2014 18
- 19. Upgrade Strategy • Surely “OpenStack can‟t be upgraded” • Our Essex, Folsom and Grizzly clouds were „tear-down‟ migrations • Puppet managed VMs are typical Cattle cases – re-create • User VMs snapshot, download image and upload to new instance • One month window to migrate • Users of production services expect more • Physicists accept not creating/changing VMs for a short period • Running VMs must not be affected 09/05/2014 LinuxTag 2014 19
- 20. Phased Migration • Migrated by Component • Choose an approach (online with load balancer, offline) • Spin up „teststack‟ instance with production software • Clone production databases to test environment • Run through upgrade process • Validate existing functions, Puppet configuration and monitoring • Order by complexity and need • Ceilometer, Glance, Keystone • Cinder, Client CLIs, Horizon • Nova 09/05/2014 LinuxTag 2014 20
- 21. Upgrade Experience • No significant outage of the cloud • During upgrade window, creation not possible • Small incidents (see blog for details) • Puppet can be enthusiastic! - we told it to be • Community response has been great • Bugs fixed and points are in Juno design summit • Rolling upgrades in Icehouse will make it easier 09/05/2014 LinuxTag 2014 21
- 22. OpenStack Federation • OpenStack clouds in many high energy physics sites • 2 more clouds at CERN in experiment areas (>20K cores each) • Many collaborating sites adopting OpenStack • Rackspace collaboration in Openlab • Aim for seamless cloud resources (CERN, sites, public) • All code to be included as open source in core OpenStack • Federation building blocks (authentication, images, compute) • Authentication included in Icehouse • More to come… 09/05/2014 LinuxTag 2014 22
- 23. Next Steps • Scaling to >100,000 cores by 2015 • Around 100 hypervisors per week with fixed staff • Deploying and configurimg the latest features • Kerberos / X.509 certificate authentication • Delegated quota management • Orchestration • Database as a Service • Cells scaling and scheduling • Federation 09/05/2014 LinuxTag 2014 23
- 24. Summary • OpenStack at CERN is in production for thousands of physicists to analyse the results of the LHC • Rapid innovation around OpenStack gives new function at an incredible rate • Upgrades already done at scale and are approaching transparent in future • Collaboration around vibrant open source communities has delivered production quality services 09/05/2014 LinuxTag 2014 24
- 25. Questions ? 09/05/2014 LinuxTag 2014 25 • Details at http://openstack-in- production.blogspot.fr • CERN User guide at http://information- technology.web.cern.ch/boo k/cern-private-cloud-user- guide • Previous presentations at http://information- technology.web.cern.ch/boo k/cern-private-cloud-user- guide/openstack-information
- 26. 09/05/2014 LinuxTag 2014 26
- 27. Service Models 09/05/2014 LinuxTag 2014 27 • Pets are given names like pussinboots.cern.ch • They are unique, lovingly hand raised and cared for • When they get ill, you nurse them back to health • Cattle are given numbers like vm0042.cern.ch • They are almost identical to other cattle • When they get ill, you get another one
- 28. 09/05/2014 LinuxTag 2014 28
- 29. 09/05/2014 LinuxTag 2014 29 http://www.eucalyptus.com/blog/2013/04/02/cy13-q1-community-analysis-%E2%80%94-openstack-vs-opennebula-vs-eucalyptus-vs- cloudstack
- 30. 09/05/2014 LinuxTag 2014 30
- 31. 09/05/2014 LinuxTag 2014 31 Tier-1 (11 centres): •Permanent storage •Re-processing •Analysis Tier-0 (CERN): •Data recording •Initial data reconstruction •Data distribution Tier-2 (~200 centres): • Simulation • End-user analysis • Data is recorded at CERN and Tier-1s and analysed in the Worldwide LHC Computing Grid • In a normal day, the grid provides 100,000 CPU days executing over 2 million jobs
- 32. 09/05/2014 LinuxTag 2014 32
- 33. Training for Newcomers 09/05/2014 LinuxTag 2014 33 Buy the book rather than guru mentoring
- 34. What are the Origins of Mass ? 09/05/2014 LinuxTag 2014 34
- 35. Matter/Anti Matter Symmetric? 09/05/2014 LinuxTag 2014 35
- 36. Where is 95% of the Universe? 09/05/2014 LinuxTag 2014 36
- 37. New Data Centre in Budapest 09/05/2014 LinuxTag 2014 37
- 40. Metering at Scale • Ceilometer provides metering functions for OpenStack • Requires careful configuration for cells 09/05/2014 LinuxTag 2014 40
- 41. I/O at Scale • Most hypervisors are recycled servers • Most are 2 SATA disks 1-2 TBs • Some SSD but limited capacity • IOPS limited with local storage • Some guest tuning e.g. Linux scheduler • General approach to use remote storage • Ceph storage • Network protocols such as webdav 09/05/2014 LinuxTag 2014 41
Notes de l'éditeur
- Over 1,600 magnets lowered down shafts and cooled to -271 C to become superconducting. Two beam pipes, vacuum 10 times less than the moon
- These collisions produce data, lots of it. Over 100PB currently 45,000 tapes… data rates of up to 35 PB/year currently and expected to significantly increase in the next run in 2015. The data must be kept at least 20 years so we’re expecting exabytes….
- Recording and analysing the data takes a lot of computing power.The CERN computer centre was built in the 1970s for mainframes and crays. Now running at 3.5MW of power, it houses 11,000 servers but is at the limit of cooling and electrical power. It is also a tourist attraction with over 80,000 visitors last year!As you can see, racks are only partially empty in view of the limits on cooling.
- We adopted a Google toolchain approach. The majority of home written software was replaced by open source projects. Commercial tools which were already working well such as JIRA and Active Directory were maintained. The approach was to select a tool, prototype, fail early and then refine requirements (following the we are not special approach)Key technologies were Puppet for configuration management and OpenStack for the private cloud.
- Already 3 independent clouds – federation is now being studiedRackspace inside CERN openlabHelix Nebula as discussed later
- Account Management AutomationCERN legacy network databaseNo Neutron yet
- HA Proxy load balancers to ensure high availabilityRedundant controllers for compute nodesCells used by the largest sites such as Rackspace and NeCTAR – more than 1000 hypervisors is the recommended configuration
- Child cells have their own keystone in view of load from ceilometerRequires care to set up and test
- The Worldwide LHC Computing grid is used to record and analyse this data. The grid currently runs over 2 million jobs/day, less than 10% of the work is done at CERN. There is an agreed set of protocols for running jobs, data distribution and accounting between all the sites which co-operate in order to support the physicists across the globe.
- We asked our 20 member states to make us an offer for server hosting using public procurement. 27 proposals and Wigner centre in Budapest, Hungary was chosen. This allows us to envisage sufficient computing and online storage for the run from 2015.