John Brimmer and Tony Hansmann's talk on Data Centers for http://www.meetup.com/SF-Bay-Area-Large-Scale-Production-Engineering/events/15481153/

1. Efficient Data Center Installations John Brimmer, Data Center Manager Tony Hansmann, Storage Manager Complete Genomics

2. Why Focus on Installations? Our goal is to create an installation process to operationalize the management of our data centers Most organizations have “self service data centers”, where your DBA or Unix admins are also data center managers No standards- wild west of DC installation Not efficient - poor power and cooling management Error prone - Circuit overload, unmarked cables Long lead time – good gear sits on the shelf Not cost efficient – Data Center providers typically charge per circuit so under utilized circuits and racks cost $$$ 2

3. Why we looked for a better way A small IT team can’t afford to have installs be a distraction The IT support team may have a limited knowledge of power and data center equipment Wiring and labeling standards are hard to maintain Vendor Installs were taking to long (worst was 2 weeks) Disorganized on install day Vendors don’t know our environment Incorrect plug types of networking cables, leads to delays Problems with orders were dragging out for weeks After a bad install that dragged out for weeks, our vendor told us they could have fixed all the issues on install day 3

4. Understanding Requirements Hardware requirements (physical space, power) Network requirements (MAC address, IP address, switch ports) Finance requirements (serial number, asset management) Support requirements (Service Tag number, remote management) Data Center requirements (host names, documentation) 4

5. Efficiency = Growth Using 3rd party rack stack vendors (RSV) can significantly help in maintaining installation policy, and reduce deployment time. Typically RSVs cost much less than having a full time IT admin installing hardware Efficient Data Center installation policy helps maintain: Infrastructure standards- cable plan management, rack elevations, power management Simplified manageability – resources are easy to locate and support Fast installation process – plan and execute quickly Stable data center operations – reduces accidents, overloads Gather and report data – PDU’s , UPS, temp/humidity monitors, BMS all provide data Cost Efficiency- Get the most out of your big $ investments 5

6. Installation Day Technical Installation Team needs to be trained Request “dedicated” install team from RSV Train installation team on standards (naming standards, labeling standards, documentation) Provide rack elevation diagrams pre-installation Create a host list with label names Outline work timeline and completion time Validate installation Using standards we were able to install 1PB storage cluster in a day! 6

7. Managing Vendors Vendors need to be trained to our standards Hardware specific data to be provided pre-install (MAC, Serial Number, Support Number) Onsite hardware verification Vendors unpack hardware to ensure shipment accuracy Vendors checks for damaged goods Vendor provides installation overview 7

8. Defined what an install should look like Moved to having our RSV do install by default Asked our vendors to sell us gear integrated into our environment Stopped accepting best effort Conference calls to do dry runs of the install Install-day Schedules are to the 30 min We sense it’s a bit torturous to our vendors to start, but they have no excuse if they don’t cover something They still miss stuff 8

9. What we’ve gained Installs can be run by John alone It works better if there’s only one staff member there All questions are answered promptly, no miscommunications because there’s only one person to ask The vendor does what their best at, we do what we’re best at They are specialist at navigating their org, we let them We have issues, but they are not problems for us, they are something they managed for us We have schedules we can count on! Much more Fire and Forget Spend time on install tasks your vendor can’t do for us Our wiring/racking standards stay strong A real sense of teamwork and everyone is covering their zone 9

10. Lessons Learned Have to get over feeling like a grade-school teacher We learned to say ‘because’ a lot Because we have a quality initiative Because our purchasing department says cut install costs Have to spell out: the only people lifting/installing/wiring is your RSV Otherwise vendor tries to ‘help’ Shoo them off as soon as possible The vendors end up appreciating it after they go through the process a few times 10

11. Thank You! 11