Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

HPC in a Box - Docker Workshop at ISC 2015

1 595 vues

Publié le

In this video from the Docker Workshop at ISC 2015, Christian Kniep from QNIB Solutions shows how he uses Docker in his efforts to provide a HPC software stack in a box, encapsulating each layer in the HPC stack within a Linux Container.

Watch the video presentation: http://wp.me/p3RLHQ-eos

Learn more: http://qnib.org/about/

Publié dans : Technologie
  • Soyez le premier à commenter

HPC in a Box - Docker Workshop at ISC 2015

  1. 1. ssSystem Service Disruption
  2. 2. • HPC in a Box • Cluster Use-Case QNIBInventory Dashboards • Development Workflow w/ Containers • Microservices 2 Agenda This Workshop was recorded:
 https://youtu.be/L9SyY9TZyY4
  3. 3. HPC in a Box
  4. 4. 4 QNIB
  5. 5. 5 Mock-Up a Cluster Stack • ‘Hello World’ everything ibsim, SLURM cluster, monitoring • First attempt using VirtualBox Predefined resource use Gave up after starting a handful of VMs • Stumbled upon Docker in 2013 Applied it to my problem
  6. 6. 6 QNIBMonitoring srv backendconsul • CONTAINERISE all the things Service discovery, health checks, clustering CONTAINER RELEVANT SERVICE
  7. 7. 7 Consul
  8. 8. 8 Consul
  9. 9. 9 QNIBMonitoring srv backendconsul carboncarbon graphite-apigraphite-api Performance grafanagrafana • CONTAINERISE all the things Service discovery, health checks, clustering metrics engine CONTAINER RELEVANT SERVICE
  10. 10. 10 Performance (graphite)
  11. 11. 11 QNIBMonitoring elasticsearch srv backendconsul carboncarbon graphite-apigraphite-api Performance grafanagrafana Log/Events elasticsearch logger logstash kibana3 kibana3 kopf es-kopf • CONTAINERISE all the things Service discovery, health checks, clustering metrics engine log event framework CONTAINER RELEVANT SERVICE
  12. 12. 12 Logs (ELK) $ echo "Hello World"|nc -w1 192.168.99.100 5514
  13. 13. 13 Event/Metric Correlation
  14. 14. Cluster Use-Case
  15. 15. docker host elasticsearch • Small SLURM cluster 7 compute nodes, 2 spine-, 2 leaf-SW simple workload (ib_write_bw) 15 Cluster Use-case srv backend consul opensmopensm Cluster carboncarbon graphite-apigraphite-api Performance grafanagrafana Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf neo4j neo4j Inventory inventory QINBInv clusterinfo ibinfo slurminfo compute0 compute1 compute2 compute3 compute4 compute5 compute6 IB ETH
  16. 16. QNIBInventory
  17. 17. • InfiniBand topology is reflected in GraphDB 17 QNIBInventory
  18. 18. • InfiniBand topology is reflected in GraphDB Routing information 18 QNIBInventory
  19. 19. • InfiniBand topology is reflected in GraphDB Routing information • SLURM information 19 QNIBInventory
  20. 20. • Enrich Log/Events 20 QNIBInventory before after
  21. 21. • Enrich Log/Events • Build up history 21 QNIBInventory
  22. 22. Dashboards
  23. 23. 23 Static Dashboards
  24. 24. 24 IB Dashboard
  25. 25. • SLURM job overview 25 Autogenerated Dashboards
  26. 26. • SLURM job overview • Individual Job Dashboards 26 Autogenerated Dashboards
  27. 27. Benefits / Implications
  28. 28. • New dashboard cubism? 28 Rapid Prototyping elasticsearch srv backend consul opensmopensm Cluster carboncarbon graphite-apigraphite-api Performance grafanagrafana Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf neo4j neo4j Inventory inventory QINBInv clusterinfo ibinfo slurminfo
  29. 29. • New dashboard cubism? 29 Rapid Prototyping elasticsearch srv backend consul opensmopensm Cluster carboncarbon graphite-apigraphite-api Performance grafanagrafana Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf neo4j neo4j Inventory inventory QINBInv clusterinfo ibinfo slurminfo cubismcubism.js
  30. 30. • New backend for graphite: InfluxDB written in go, explicit TS-database, nice API, SQL-queries compatible input “key val ts” and integrates with Graphite-API 30 Rapid Prototyping #2 elasticsearch srv backend consul opensmopensm Cluster carboncarbon graphite-apigraphite-api Performance grafanagrafana Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf neo4j neo4j Inventory inventory QINBInv clusterinfo ibinfo slurminfo carbon.service.consul
  31. 31. • New backend for graphite: InfluxDB written in go, explicit TS-database, nice API, SQL-queries compatible input “key val ts” and integrates with Graphite-API 31 Rapid Prototyping #2 elasticsearch srv backend consul opensmopensm Cluster carboncarbon graphite-apigraphite-api Performance grafanagrafana Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf neo4j neo4j Inventory inventory QINBInv clusterinfo ibinfo slurminfo influxdb influxdb graphite-api’ graphite-api grafana’ grafana carbon-relaycarbon carbon.service.consul
  32. 32. Workstation ~/dev/inventory/ • Write local, execute within container Reproducible, reliable development environment 32 Development Environment Container /opt/inventory/
  33. 33. Workstation ~/dev/inventory/~/prod/inventory/ • Write local, execute within container Reproducible, reliable development environment 33 Development Environment Container /opt/inventory/
  34. 34. docker host elasticsearch 34 Iterate on Log-Patterns srv backend consul opensmopensm Cluster carboncarbon graphite-apigraphite-api Performance grafanagrafana Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf neo4j neo4j Inventory inventory QINBInv clusterinfo ibinfo slurminfo compute0 compute1 compute2 compute3 compute4 compute5 compute6 IB ETH
  35. 35. • GROK test 
 
 
 
 • GROK pattern • Logstash NEW_PORT Hello World 35 Iterate on Log-Patterns NEW_PORT: compare: "%{NEW_PORT}" input: "Creating new port object with GUID 0x0002c90300ee1b81" result: { "src_port_guid": "2c90300ee1b81" } if [osm_func] == "ni_rcv_process_existing_ca_or_router" { grok { patterns_dir => "/etc/grok/patterns/" match => [ "message", "%{NEW_PORT}" ] } } NEW_PORT Creatings+news+ports+objects+withs+GUIDs+%{SRC_PORT_GUID}
  36. 36. MacBook elasticsearch 36 Iterate on Log-Patterns srv backend consul Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf • Start minimal stack on workstation
  37. 37. Microservices
  38. 38. 38 Microservice Definition –Adrian Cockcroft „Loosely coupled service oriented architecture with bounded contexts“
  39. 39. MacBook elasticsearch 39 bounded context srv backend consul Log/Events elasticsearch logger logstash kibana kiabana kopf es-kopf • Iterate within the context • Rely on stable version of (loosely coupled) dependencies Thinkpad elasticsearch srv backend consul   Log/Events elasticsearch neo4j neo4j Inventory inventory QINBInv
  40. 40. 40 µServices: Break down Silos Super User Prod Mgr Sys Arch Dev QA Sys Adm Net Adm HPC Snow- flakes Product Team Using Monolithic Delivery Product Team Using Monolithic Delivery Product Team Using Microservices A
 P
 I Srv Team Srv TeamProduct Team Using Microservices
  41. 41. • Linux Containers do not add much overhead • Lighting fast development iteration, 
 since boot-up is plain ‘fork()’ • build, distribute and run extremely powerful private/public registries • image hierarchy to add new service in minutes 41 Conclusion
  42. 42. Q&A

×