Contenu connexe
Similaire à Hp cmu – easy to use cluster management utility @ hpcday 2012 kiev
Similaire à Hp cmu – easy to use cluster management utility @ hpcday 2012 kiev (20)
Plus de Volodymyr Saviak (12)
Hp cmu – easy to use cluster management utility @ hpcday 2012 kiev
- 1. HP Insight CMU
Cluster Management Utility Tour
Sébastien Cabaniols, CMU WW Team lead / EMEA HPC Presales consultant
7th HPC Day conference, Kiev, Ukraine
October 2012
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 2. Agenda
HP Insight CMU Introduction & Review
Introduction, History / Customers
Product mindset
Insight CMU v7.0 tour
Provisioning (Cloning / Autoinstall / Diskless)
Monitoring ( TimeView / Collectl / GPGPUs…)
Scalable/Frictionless administration ( cmudiff…)
Custom GUI & partners integration
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 3. Insight CMU
Introduction & History
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 4. Insight CMU introduction
CMU = Cluster Management Utility
‚CMU optimizes the TCO of compute farms‛
CMU scaling specification: 4k nodes
CMU has lots of industrial clusters in production with 2k/3k+ nodes
CMU has a strong presence in the TOP500 (www.top500.org)
CMU at customer site since 2000
4 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 5. Insight CMU major milestones
2000: initial implementation for Tru64 Unix (Alphaserver)
2001: port to Alpha Linux, 1600 servers commercial cluster
2002: port to x86 & IA64 Linux / HPUX Itanium
2004: port to x86_64 Linux. (only port maintained*)
2007: Swedish gov, 6th @ TOP500
2010: Tsubame 2, HP first public 1+ PFlop cluster, 5th @ TOP500
*ARM port in progress...
5 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 6. Worldwide CMU Deployments
HP ships >2 CMU clusters per week WW
UNIVERSITIES ENGINEERING
GOVERNMENT and RESEARCH LABS ENERGY
6
6
April 2009
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 7. Insight CMU project mindset
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 8. Insight CMU project mindset
CMU provides the core functionalities for a farm/cloud
runs any HP* server (even mix) / any Linux distribution (even mix)
independent of many architectural aspects of the system:
interconnects / GPGPUs / IO accelerators...
network topology (open cluster, guarded cluster, WAN…)
batch/job schedulers, MPI stacks, math libraries, compilers....
CMU is not a ‘predefined’ SW supercomputer appliance
>90% systems delivered as ‚turn-key solutions‛
CMU can also be purchased standalone with support and manuals
8 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 9. Insight CMU v7.0 tour
CMU functionalities / typical CMU implementation
CMU Provisioning
CMU Monitoring
CMU Scalable / Frictionless administration
CMU Custom GUI & partners integration
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 10. CMU starts here: typical { farm / HPC cluster } implementation
{ high speed Interconnect }
{ Highly Avail.}Head node
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 11. Insight Cluster Management Utility Basics
CMU is a single package running on the head node (upgrade is trivial)
CMU mgt node can be an HA cluster (HP service guard,Redhat Cluster, SLES HA)
Provides a full fledged interactive CLI
Provides cmu_* commands as an API (for scripting)
For integration with other software or command-line activity (see partner’s integration)
Provides GUI client for single dashboard control
launch from a web page served from head node (JAVA© webstart technology)
run on a local laptop/desktop
user mode for monitoring
admin mode for remote administrationcontained herein is subject to change without notice.
11 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information
- 12. The three pillars of HP Insight CMU
Provision Monitor Control
• Simplified discovery, • ‘At a glance’ view of • GUI & CLI & API
firmware audits entire system / • Easy GUI, friction-
• Fast & scalable partition less control of
cloning • Customizable remote servers
• Legacy support of • Lightweight • Scalable pdsh with
Kickstart/Autoyast/D • Instant 2D view cmudiff analyser
ebian Preseed • TimeView, 3D live
• Diskless support history
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 13. Insight CMU v7.0 Tour
CMU Provisioning
Scalable Cloning
Legacy/Compatible Autoinstall (Kickstart/Autoyast/Preseed)
Diskless
Firmware audit
Bare metal netboot low level tools (hpacucli / hponcf / ipmitool)
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 14. CMU provisioning engine
backup/cloning (up to 4k nodes)
RHEL (& clones) / SLES ( & clones) / Debian / Ubuntu (only compute nodes)
performance only depends on the image size & harddrive speed
no architectural dependence on trunked/IB/10gig networks, 1Gig ethernet is sufficient.
22 minutes to clone 1000 nodes with SAS drives and 10 GB image
continuous cloning for reprovisionning from batch schedulers
autoinstall CMU bridge to legacy/standard tools
:
Redhat Kickstart / SLES Autoyast / Debian Preseed
do not use above 100 nodes
diskless (advised if improving the density of the solution and/or data security)
statefull diskless engine (hybrid NFS ro + rw personalities)
14 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 15. CMU firmware / bare metal netboot tools
firmware hooks:
firmware version checks (HP conrep based currently / HP rcu soon)
firmware settings audit (HP conrep + cmudiff)
firmware flashing engine (to feed with SCEXE HP files)
bare metal netboot tools (available from ‚pre_reconf ‚/ ‚reconf‛ )
hpacucli : configure HP smartarray controllers
locfg.pl/ hponcfg: configure HP ILO from the CMU netboot environment
ipmitools: configure IPMI capable BMC from the CMU netboot environment
19 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 16. Insight CMU v7.0 Tour
CMU Monitoring
scalable / ‘HPC aware’ monitoring engine (collectl, GPGPUS)
2D Instant View / 3D Time View (Live History)
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 17. CMU monitoring
Backend: ‚HPC aware‛ monitoring since years
Scalable monitoring ( proven on 4k nodes system )
Non intrusive (leverage collectl + ‚HPC synchro‛ mode)
Programmable (monitor anything you can script )
Nvidia & AMD GPGPUs monitoring tool
Extended Monitoring to inject arbitrary monitoring data
Alerting system & CMU Reactions
Frontend GUI (JAVA client/server) / CLI
GUI: Instant view 2D / TimeView 3D (Live History)
cmu_dynamic_user groups (see later in presentation)
21 CLI/API: cmu_monstat and flat human readable files
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 18. ‚Instant View*‛ CMU Display
* renamed « Instant View » since CMU v7.0
22 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 19. 23 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 20. TimeView (Live History)
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 21. Existing ‘well known’ CMU Display since 2004
25 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 22. 3D Display of Sensor Histories
readability, efficiency, precision
26
2
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 23. 3D Display of Sensor Histories
global job overview
27 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 24. 28 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 25. 29 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 26. 30 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 27. 31 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 28. GPGPUs monitoring
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 29. CMU GPGPU Support
CMU provides a tool for extracting GPU metric data
from GPU driver
‚cmu_get_nvidia_gpu‛ monitors:
load, mem_util, mem_alloc, power_state, and ECC_double_bit
alerts by default
Power_usage, various clock speeds, fan speeds, and
temperature also configured but commented out by default
33 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 30. 37 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
F
3ooter goes here
- 32. CMU Extended Monitoring
Inject monitoring data from another source into CMU
Extended metrics will be used for:
Server hardware metrics (ILO4 out-of-band & agentless monitoring)
• Temperatures, fan speeds, power usage
• Gathered out-of-band
• OS-neutral
Cluster peripherals
• MCS temperatures, switch status
Workload schedulers
39 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 33. CMU alerts & reactions
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 34. CMU alerts & CMU reactions
CMU monitoring engine can trigger alerts
CMU alerts can trigger scripts as reactions to alerts
reaction examples:
• SNMP traps (send all alerts to an SNMP sink such as HPSIM)
• Send an email
• Remove a compute node from a batch scheduler…
44 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 35. Insight CMU v7.0 Tour
Scalable / Frictionless monitoring
Interactive command broadcast (ssh, BMC interfaces)
cmudiff non interactive scalable command output analyzer
GUI accelerators (power off / UID leds/ three clicks ‘en masse’ cloning….)
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 36. Insight CMU GUI basics
Cluster mgmt
panel
displays all
nodes in
selected
groupings: by
switch
location; by
image; or by
custom
grouping
node states
display
current state
of each node CMU Main Alerts displayed
47
Display Panel
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
along the bottom
- 37. CMU GUI Basics
– Right-click to select sensors to
display
– CMU pre-configured with standard
sensors: CPU and memory usage,
and disk and network I/O
– Simple to add any sensor or alert
– CMU provides simple support for
monitoring GPU temp and ECC
errors
– Three clicks to clone compute nodes
!
48 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 38. Friction-less remote control of target nodes
Selected Power
nodes Broadcast
commands
commands
Provisioning
commands
User-defined
commands
49 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 39. CMU remote management commands
• Multi-window broadcast command (access OS or console)
51
type here…
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
...and see it there
- 40. cmudiff
scaling the command line.
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 41. Compare node outputs with Scalable Text Analyser (cmudiff)
Single-window pdsh with
cmu_diff example
One command
executed across a
set of selected
nodes…
…finds one node
running with an old
BIOS version!
53 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 42. dshbak vs cmudiff: round #1…. ‘date’ on five hosts
cmudiff
dshbak
57 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 43. dshbak vs cmudiff: round #2..‘ifconfig’ on 3 hosts
cmudiff
dshbak
58 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
58 HP Confidential
- 44. Partners software integration
& Custom menu GUI
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 45. 62 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 46. 63 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 47. CMU Custom Menu Support
/opt/cmu/etc/cmu_custom_menu
64 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 49. Insight CMU as a (job) power monitor
70 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 50. Insight CMU Partner Integrations
Moab – Dynamic Provisioning
PBS Pro – Green Scheduling & OS Provisioning
LSF – Platform HPC
ScaleMP – create large virtual SMP nodes
StackIQ – CMU part of HP ‚roll‛
HP Matrix CMU CloudMap
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 51. Thank you for your interest in
HP Insight CMU
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.