HPC in a Box - Docker Workshop at ISC 2015
•
1 j'aime•1,846 vues
In this video from the Docker Workshop at ISC 2015, Christian Kniep from QNIB Solutions shows how he uses Docker in his efforts to provide a HPC software stack in a box, encapsulating each layer in the HPC stack within a Linux Container. Watch the video presentation: http://wp.me/p3RLHQ-eos Learn more: http://qnib.org/about/
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à HPC in a Box - Docker Workshop at ISC 2015
Similaire à HPC in a Box - Docker Workshop at ISC 2015 (20)
Plus de inside-BigData.com
Plus de inside-BigData.com (20)
Dernier
Dernier (20)
HPC in a Box - Docker Workshop at ISC 2015
- 2. • HPC in a Box • Cluster Use-Case QNIBInventory Dashboards • Development Workflow w/ Containers • Microservices 2 Agenda This Workshop was recorded: https://youtu.be/L9SyY9TZyY4
- 3. HPC in a Box
- 4. 4 QNIB
- 5. 5 Mock-Up a Cluster Stack • ‘Hello World’ everything ibsim, SLURM cluster, monitoring • First attempt using VirtualBox Predefined resource use Gave up after starting a handful of VMs • Stumbled upon Docker in 2013 Applied it to my problem
- 6. 6 QNIBMonitoring srv backendconsul • CONTAINERISE all the things Service discovery, health checks, clustering CONTAINER RELEVANT SERVICE
- 7. 7 Consul
- 8. 8 Consul
- 9. 9 QNIBMonitoring srv backendconsul carboncarbon graphite-apigraphite-api Performance grafanagrafana • CONTAINERISE all the things Service discovery, health checks, clustering metrics engine CONTAINER RELEVANT SERVICE
- 11. 11 QNIBMonitoring elasticsearch srv backendconsul carboncarbon graphite-apigraphite-api Performance grafanagrafana Log/Events elasticsearch logger logstash kibana3 kibana3 kopf es-kopf • CONTAINERISE all the things Service discovery, health checks, clustering metrics engine log event framework CONTAINER RELEVANT SERVICE
- 12. 12 Logs (ELK) $ echo "Hello World"|nc -w1 192.168.99.100 5514
- 14. Cluster Use-Case
- 15. docker host elasticsearch • Small SLURM cluster 7 compute nodes, 2 spine-, 2 leaf-SW simple workload (ib_write_bw) 15 Cluster Use-case srv backend consul opensmopensm Cluster carboncarbon graphite-apigraphite-api Performance grafanagrafana Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf neo4j neo4j Inventory inventory QINBInv clusterinfo ibinfo slurminfo compute0 compute1 compute2 compute3 compute4 compute5 compute6 IB ETH
- 16. QNIBInventory
- 17. • InfiniBand topology is reflected in GraphDB 17 QNIBInventory
- 18. • InfiniBand topology is reflected in GraphDB Routing information 18 QNIBInventory
- 19. • InfiniBand topology is reflected in GraphDB Routing information • SLURM information 19 QNIBInventory
- 21. • Enrich Log/Events • Build up history 21 QNIBInventory
- 22. Dashboards
- 24. 24 IB Dashboard
- 25. • SLURM job overview 25 Autogenerated Dashboards
- 26. • SLURM job overview • Individual Job Dashboards 26 Autogenerated Dashboards
- 28. • New dashboard cubism? 28 Rapid Prototyping elasticsearch srv backend consul opensmopensm Cluster carboncarbon graphite-apigraphite-api Performance grafanagrafana Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf neo4j neo4j Inventory inventory QINBInv clusterinfo ibinfo slurminfo
- 29. • New dashboard cubism? 29 Rapid Prototyping elasticsearch srv backend consul opensmopensm Cluster carboncarbon graphite-apigraphite-api Performance grafanagrafana Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf neo4j neo4j Inventory inventory QINBInv clusterinfo ibinfo slurminfo cubismcubism.js
- 30. • New backend for graphite: InfluxDB written in go, explicit TS-database, nice API, SQL-queries compatible input “key val ts” and integrates with Graphite-API 30 Rapid Prototyping #2 elasticsearch srv backend consul opensmopensm Cluster carboncarbon graphite-apigraphite-api Performance grafanagrafana Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf neo4j neo4j Inventory inventory QINBInv clusterinfo ibinfo slurminfo carbon.service.consul
- 31. • New backend for graphite: InfluxDB written in go, explicit TS-database, nice API, SQL-queries compatible input “key val ts” and integrates with Graphite-API 31 Rapid Prototyping #2 elasticsearch srv backend consul opensmopensm Cluster carboncarbon graphite-apigraphite-api Performance grafanagrafana Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf neo4j neo4j Inventory inventory QINBInv clusterinfo ibinfo slurminfo influxdb influxdb graphite-api’ graphite-api grafana’ grafana carbon-relaycarbon carbon.service.consul
- 32. Workstation ~/dev/inventory/ • Write local, execute within container Reproducible, reliable development environment 32 Development Environment Container /opt/inventory/
- 33. Workstation ~/dev/inventory/~/prod/inventory/ • Write local, execute within container Reproducible, reliable development environment 33 Development Environment Container /opt/inventory/
- 34. docker host elasticsearch 34 Iterate on Log-Patterns srv backend consul opensmopensm Cluster carboncarbon graphite-apigraphite-api Performance grafanagrafana Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf neo4j neo4j Inventory inventory QINBInv clusterinfo ibinfo slurminfo compute0 compute1 compute2 compute3 compute4 compute5 compute6 IB ETH
- 35. • GROK test • GROK pattern • Logstash NEW_PORT Hello World 35 Iterate on Log-Patterns NEW_PORT: compare: "%{NEW_PORT}" input: "Creating new port object with GUID 0x0002c90300ee1b81" result: { "src_port_guid": "2c90300ee1b81" } if [osm_func] == "ni_rcv_process_existing_ca_or_router" { grok { patterns_dir => "/etc/grok/patterns/" match => [ "message", "%{NEW_PORT}" ] } } NEW_PORT Creatings+news+ports+objects+withs+GUIDs+%{SRC_PORT_GUID}
- 36. MacBook elasticsearch 36 Iterate on Log-Patterns srv backend consul Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf • Start minimal stack on workstation
- 37. Microservices
- 38. 38 Microservice Definition –Adrian Cockcroft „Loosely coupled service oriented architecture with bounded contexts“
- 39. MacBook elasticsearch 39 bounded context srv backend consul Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf • Iterate within the context • Rely on stable version of (loosely coupled) dependencies Thinkpad elasticsearch srv backend consul Log/Events elasticsearch neo4j neo4j Inventory inventory QINBInv
- 40. 40 µServices: Break down Silos Super User Prod Mgr Sys Arch Dev QA Sys Adm Net Adm HPC Snow- flakes Product Team Using Monolithic Delivery Product Team Using Monolithic Delivery Product Team Using Microservices A P I Srv Team Srv TeamProduct Team Using Microservices
- 41. • Linux Containers do not add much overhead • Lighting fast development iteration, since boot-up is plain ‘fork()’ • build, distribute and run extremely powerful private/public registries • image hierarchy to add new service in minutes 41 Conclusion
- 42. Q&A