PuppetConf 2012 - Everyone hears about automation of OS installs and configuration of services on top of the OS yet no one talks about automating the configuration of firmware. There are a number of venders who provide tools that allow the intrepid systems administrator to never again have to arrow through BIOS menus to toggle some less known but needed setting. Gone are the days of manually entering disk configurations into a RAID controller. This session is for SysAdmins or hardware geeks that want to free up their time to enjoy the finer things in life. You will be presented with the knowledge gained from two years of experience managing firmware configuration as a part of hardware discovery and provisioning.

1. APOLLO GROUP
Automated Configuration of Firmware
Michael Arnold
Principal Systems Engineer
27 September 2012

2. Agenda
Who
Tools
Example Workflow
Example Code
Links
Questions
APOLLO GROUP © 2012 Apollo Group 2

3. APOLLO GROUP
Who
APOLLO GROUP Apollo Group
© 2012 3

4. Who is Apollo?
Apollo Group is a leading provider of higher
education programs for working adults.
APOLLO GROUP © 2012 Apollo Group 4

5. Who is Michael Arnold?
Systems Administrator
Automation geek
13 years in IT
Hadoop Cluster Admin
I deal with:
–Server hardware specification/configuration
–Server firmware
–Server operating system
–Hadoop Cluster performance and availability
APOLLO GROUP © 2012 Apollo Group 5

6. How does firmware tie in with Puppet?
It has nothing to do with Puppet.
–Unless someone writes a module to program
firmware from the running OS.
• Which could work for RAID controllers.
APOLLO GROUP © 2012 Apollo Group 6

7. APOLLO GROUP
Tools
APOLLO GROUP Apollo Group
© 2012 7

8. Hardware Toolkits
HP SmartStart Scripting Toolkit
Dell OpenManage Deployment Toolkit
IBM ServerGuide Scripting Toolkit
Fujitsu ServerView Scripting Toolkit
APOLLO GROUP © 2012 Apollo Group 8

9. What do the tools provide?
Configuration of
–BIOS
–BMC (DRAC/iLO/RSA/etc.)
–Hardware RAID
–FC HBA
APOLLO GROUP © 2012 Apollo Group 9

10. How does the toolkit work?
Manually configure reference hardware
Capture the firmware configuration
Clone the configuration to multiple machines
APOLLO GROUP © 2012 Apollo Group 10

11. Toolkit Components
Kernel & Firmware
Toolkit
Initrd Tools
APOLLO GROUP © 2012 Apollo Group 11

12. Toolkit Requirements
TFTP
DHCP NFS
Toolkit
APOLLO GROUP © 2012 Apollo Group 12

13. APOLLO GROUP
Example Workflow
APOLLO GROUP Apollo Group
© 2012 13

14. Assumptions
Hardware is racked and cabled
Switch ports are up
Server is manually powered on
Firmware defaults are to boot from network if no
other boot device is found.
–No other boot devices are found
APOLLO GROUP © 2012 Apollo Group 14

15. Constraints
All server hardware is from the same vendor (HP)
APOLLO GROUP © 2012 Apollo Group 15

16. The Process
Firmware Hardware Firmware Buildsystem
Update Discovery Configuration Registration
APOLLO GROUP © 2012 Apollo Group 16

17. Screencast
APOLLO GROUP © 2012 Apollo Group 17

18. Workflow
Unknown system netboots
APOLLO GROUP © 2012 Apollo Group 18

19. Workflow
Unknown system netboots
Boot into HP SmartStart Scripting Toolkit
APOLLO GROUP © 2012 Apollo Group 19

20. Workflow
Unknown system netboots
Boot into HP SmartStart Scripting Toolkit
Gather hardware information (CPU/RAM/disk/NIC)
via hpdiscovery
APOLLO GROUP © 2012 Apollo Group 20

21. Workflow
Unknown system netboots
Boot into HP SmartStart Scripting Toolkit
Gather hardware information (CPU/RAM/disk/NIC)
via hpdiscovery
If this is HP hardware, then
–Update firmware
–Configure BIOS
–Configure iLO
–Configure RAID
APOLLO GROUP © 2012 Apollo Group 21

22. Workflow
Unknown system netboots
Boot into HP SmartStart Scripting Toolkit
Gather hardware information (CPU/RAM/disk/NIC)
via hpdiscovery
If this is HP hardware, then
–Update firmware
–Configure BIOS
–Configure iLO
–Configure RAID
cobbler-register to the hardware profile
APOLLO GROUP © 2012 Apollo Group 22

23. Workflow
Unknown system netboots
Boot into HP SmartStart Scripting Toolkit
Gather hardware information (CPU/RAM/disk/NIC) via
hpdiscovery
If this is HP hardware, then
–Update firmware
–Configure BIOS
–Configure iLO
–Configure RAID
cobbler-register to the hardware profile
System reboots
APOLLO GROUP © 2012 Apollo Group 23

24. Workflow
Modify Cobbler system profile, giving it an identity
(hostname, IP, OS metadata)
APOLLO GROUP © 2012 Apollo Group 24

25. Workflow
Modify Cobbler system profile, giving it an identity
(hostname, IP, OS metadata)
Known system netboots
APOLLO GROUP © 2012 Apollo Group 25

26. Workflow
Modify Cobbler system profile, giving it an identity
(hostname, IP, OS metadata)
Known system netboots
Boot into OS installer
APOLLO GROUP © 2012 Apollo Group 26

27. Workflow
Modify Cobbler system profile, giving it an identity
(hostname, IP, OS metadata)
Known system netboots
Boot into OS installer
Install OS
APOLLO GROUP © 2012 Apollo Group 27

28. Workflow
Modify Cobbler system profile, giving it an identity
(hostname, IP, OS metadata)
Known system netboots
Boot into OS installer
Install OS
Install Puppet
APOLLO GROUP © 2012 Apollo Group 28

29. Workflow
Modify Cobbler system profile, giving it an identity
(hostname, IP, OS metadata)
Known system netboots
Boot into OS installer
Install OS
Install Puppet
…
APOLLO GROUP © 2012 Apollo Group 29

30. Workflow
Modify Cobbler system profile, giving it an identity
(hostname, IP, OS metadata)
Known system netboots
Boot into OS installer
Install OS
Install Puppet
…
Profit!
APOLLO GROUP © 2012 Apollo Group 30

31. APOLLO GROUP
Example Code
APOLLO GROUP Apollo Group
© 2012 31

32. Dell TFTP Config
LABEL dtk-install_hadoop
kernel /images/dtk-3.5.1/SA.1
append initrd=/images/dtk-3.5.1/SA.2
ramdisk_size=156482 Stage3_type=cdrom quiet DEBUG=0
share_location=192.168.1.1:/srv/nfs/ro/dtk-3.5.1
share_type=nfs selinux=0 share_opts=ro,nolock
share_script=install_hadoop.sh cprofile=Hadoop
ipappend 2
APOLLO GROUP © 2012 Apollo Group 32

33. HP TFTP Config
LABEL sstk-install_hadoop
kernel /images/sstk-8.50/vmlinuz
append initrd=/images/sstk-8.50/initrd.img
root=/dev/ram0 rw ramdisk_size=184248 ide=nodma
ide=noraid pnpbios=off network=1
sstk_mount=192.168.1.1:/srv/nfs/ro/sstk-8.50
sstk_mount_type=nfs sstk_mount_options=ro,nolock
sstk_script=install_hadoop.sh cprofile=Hadoop
ipappend 2
APOLLO GROUP © 2012 Apollo Group 33

34. Dell Config Capture
cd /opt/dell/toolkit/template/scripts
/opt/dell/toolkit/template/scripts/syscap.sh /tmp/syscap
/opt/dell/toolkit/template/scripts/raidcap.sh /tmp/raidcap
/opt/dell/toolkit/template/scripts/raccap.sh /tmp/raccap
# MegaCLI does not come with the DTK so "install" it.
cp -a /opt/dell/toolkit/systems/MegaCLI/opt/MegaRAID
/opt/MegaRAID
/opt/MegaRAID/MegaCli/MegaCli -CfgSave -a0 -f /tmp/mega.0
APOLLO GROUP © 2012 Apollo Group 34

35. HP Config Capture
cd /TOOLKIT
./conrep -s -f /tmp/conrep.dat # BIOS
./hpacuscripting -c /tmp/cpqacuxe.dat # RAID
./hplpcfg /s /tmp/hplpcfg.dat # Emulex HBA
./hpqlarep /S /tmp/hpqlarep.dat # Qlogic HBA
if ./ifhw /tmp/hw_discovery.dat allboards.xml
"PCI:Integrated Lights-Out" 2> /dev/null ; then
./hponcfg -w /tmp/hponcfg.dat # iLO 2/3
else
./lo100cfg -o /tmp/lo100cfg.dat # LO100i
fi
APOLLO GROUP © 2012 Apollo Group 35

36. Platform Detection
*** Detecting system type ***
System Information:
Name : ProLiant DL360 G5
Serial : USE123D456
Arch : x86_64
LNAME : ProLiantDL360G5
SNAME : DL360 G5
Asset Tag : 01234567
APOLLO GROUP © 2012 Apollo Group 36

37. Some Available Variables
 HP SSSTK has a function library that sets many variables.
– $PRODUCT_NAME =
• "ProLiant SL170z G6"
• "ProLiant DL360 G6"
– $SERIAL_ID = hardware serial number
– $PRODUCT_LNAME =
• "ProLiantSL170zG6"
• "ProLiantDL360G6"
– $PRODUCT_SNAME =
• "SL170z G6"
• "DL360 G6"
– $TOOLKIT_MNTPNT = /mnt/main
– $TOOLKIT_SERVER_IP = NFS server IP
– $HWDISC_FILE = /TOOLKIT/hpdiscovery.xml
APOLLO GROUP © 2012 Apollo Group 37

38. Inside install_hadoop.sh
 Set Variables Based on Hardware Type
export "`./hwquery ${HWDISC_FILE} allboards.xml SERVERNAME=SystemName`"
case "${SERVERNAME}" in
"ProLiant DL160 G6"* )
CONREPDAT=${TOOLKIT_MNTPNT}/data_files/dl160_conrep.dat
LO100CFGDAT=${TOOLKIT_MNTPNT}/data_files/dl160_lo100cfg.dat
;;
"ProLiant SL170z G6"* )
CONREPDAT="${TOOLKIT_MNTPNT}/data_files/sl170z_conrep.dat -x
${TOOLKIT_MNTPNT}/conrep_xml/conrep_SL170zg6_20100401.xml"
LO100CFGDAT=${TOOLKIT_MNTPNT}/data_files/sl170z_lo100cfg.dat
;;
#...
APOLLO GROUP © 2012 Apollo Group 38

39. Inside install_hadoop.sh
#...
"ProLiant DL360 G5"* )
CONREPDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_conrep.dat
if ./ifhw ${HWDISC_FILE} allboards.xml "PCI:Smart Array E200i Controller"
2> /dev/null ; then
HPACUDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_saE200i_cpqacuxe.dat
fi
if ./ifhw ${HWDISC_FILE} allboards.xml "PCI:Smart Array P400i Controller"
2> /dev/null ; then
HPACUDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_saP400i_cpqacuxe.dat
fi
HPONCFGDAT=${TOOLKIT_MNTPNT}/data_files/dl360g5_hponcfg.dat
;;
esac
APOLLO GROUP © 2012 Apollo Group 39

40. Inside install_hadoop.sh
if [ -n "$CONREPDAT" ] ; then
echo "** Applying BIOS Configuration"
./conrep -l -f ${CONREPDAT}
fi
if [ -n "$HPACUDAT" ] ; then
echo "** Clearing RAID Configuration"
./hpacucli/hpacuscripting -i ${HPACUDAT}_clear; sleep 5
echo "** Applying RAID Configuration"
./hpacucli/hpacuscripting -i ${HPACUDAT}
fi
if [ -n "$LO100CFGDAT" ] ; then
echo "** Applying BMC Configuration"; ./lo100cfg -i ${LO100CFGDAT}
elif [ -n "$HPONCFGDAT" ] ; then
echo "** Applying iLO Configuration"; ./hponcfg -f ${HPONCFGDAT}
fi
APOLLO GROUP © 2012 Apollo Group 40

41. Register with Buildsystem
echo "** Registering with cobbler"
# Koan does not come with SSSTK, so "install" it.
${TOOLKIT_MNTPNT}/koan/install
# cprofile is set via kernel boot parameter and points at part of a
# Cobbler profile name.
# SERIAL_ID and PRODUCT_LNAME are set via SSSTK.
# PRODUCT_LNAME = ProLiantSL170zG6, ProLiantDL360G6, etc.
# SERIAL_ID = hardware serial number
echo "** Registering to profile ${cprofile}-${PRODUCT_LNAME}"
cobbler-register --server=${TOOLKIT_SERVER_IP} --fqdn=${SERIAL_ID}
--profile=${cprofile}-${PRODUCT_LNAME} --batch
APOLLO GROUP © 2012 Apollo Group 41

42. Clear the HP RAID Controller
$ cat dl360g5_saE200i_cpqacuxe.dat_clear
; Version: 8.50.6.0
; http://h30499.www3.hp.com/t5/ProLiant-Servers-ML-DL-
SL/CPQACUXE-reset-utility-within-WinPE/td-p/3939475
Action= Reconfigure
Method= Custom
Controller= SLOT 0
ClearConfigurationWithDataLoss= Yes
APOLLO GROUP © 2012 Apollo Group 42

43. Configure the HP RAID Controller
$ cat dl360g5_saE200i_cpqacuxe.dat ; Array Specifications
; Version: 8.50.6.0 Array= A
Action= Configure ; Array Drive Type is SAS
Method= Custom ; 1I:1:1 (72 GB), 1I:1:2 (72 GB)
; Controller Specifications Drive= 1I:1:1, 1I:1:2
; Controller HP Smart Array E200i OnlineSpare= No
; Firmware Version 1.82 ; Logical Drive Specifications
Controller= SLOT 0 LogicalDrive= 1
ReadCache= 100 RAID= 1
WriteCache= 0 Size= 69973
RebuildPriority= Medium Sectors= 32
ExpandPriority= Medium StripeSize= 128
SurfaceScanDelay= 15 ArrayAccelerator= Enabled
DriveWriteCache= Disabled
APOLLO GROUP © 2012 Apollo Group 43

44. Error Handling
You may have to test for any failures when
applying configurations.
When reusing HP hardware, adding a user to the
BMC that already exists will cause a failure to
apply all of the configuration.
Newer firmware can have different config options
that will cause an older capture to fail.
APOLLO GROUP © 2012 Apollo Group 44

45. Summary
Firmware Hardware Firmware Buildsystem
Update Discovery Configuration Registration
APOLLO GROUP © 2012 Apollo Group 45

46. Links
 HP SmartStart Scripting Toolkit
http://h18013.www1.hp.com/products/servers/management/toolkit/index.html
 Dell OpenManage Deployment Toolkit
http://en.community.dell.com/techcenter/systems-management/w/wiki/1772.dell-
openmanage-deployment-toolkit.aspx
 IBM ServerGuide Scripting Toolkit
http://www-03.ibm.com/systems/be/management/sgstk/
 Fujitsu ServerView Scripting Toolkit
http://www.fujitsu.com/fts/products/computing/servers/primergy/management/de
ploy/
 Cobbler - Linux Installation Server
http://cobbler.github.com/
 LSI MegaRAID MegaCLI
http://www.lsi.com/downloads/Public/MegaRAID%20Common%20Files/8.02.24_
MegaCLI.zip
APOLLO GROUP © 2012 Apollo Group 46

47. Contact
 https://intelligentsysadmin.wordpress.com/
 https://github.com/razorsedge
 http://forge.puppetlabs.com/razorsedge
APOLLO GROUP © 2012 Apollo Group 47

48. APOLLO GROUP
Questions?
APOLLO GROUP Apollo Group
© 2012 48