Slide presentation at http://www.open-ix.org/ on April 8th, 2015 presented by Adam Rothschild from Packet (www.packet.net) Synopsis: Choosing the right datacenter is a key component for a successful infrastructure deployment, and something often overlooked, with disastrous consequences. Only when the purchaser and provider are truly aligned, can success be achieved. This presentation will provide “tricks of the trade” for dealing with the RFP process, and subsequent facility and TCO analysis, leaving attendees with a sample document and financial model they can tweak to suit their own needs. About Adam Rothschild Adam Rothschild is Packet’s Senior Vice President of Network and Datacenter Infrastructure; he is responsible for all facets of the company’s next-generation infrastructure strategy and engineering. Adam was previously a member of Voxel dot Net’s founding technical team, where he served as Vice President of Network Architecture. Following Voxel’s sale to Internap in 2011, he held several leadership positions at Internap, and was most recently Senior Manager of Network Architecture and Product Development. - See more at: https://www.packet.net/about/team/adam-rothschild

1. Datacenter
Selection
Adam Rothschild <asr@packet.net>
Open-IX - Americas Interconnection Summit
San Diego, CA - April 2015

2. DISCLAIMERS
○ Personal experiences and opinions only, not those of the Open-IX Association, working
groups, or any employer.
○ Not intended as legal advice. Datacenter contracts are extremely complex and require
review by counsel.
○ Datacenters presented are completely fictitious. Any resemblances, living or dead, are
coincidental.

3. WHY DATACENTERS
MATTER○ Significant operational expenditure, often exceeding cost of IP transit or
interconnection.
○ Often selected by the “network person” or “vendor negotiator” (hence, this audience)
○ Can be a major source of downtime if planned poorly.
○ Power, Plumbing, and Ping is not an “attractive” topic like “cloud” or “SDN”, though it
underpins it; therefore not often discussed around these parts.
○ Lack of best-practice sharing among operators. Not a germane topic for a NANOG,
etc.

4. WHEN YOU’RE NOT
ALIGNED
CENSORED

5. WHEN YOU’RE NOT
ALIGNED: SANDY EDITION
Force majeure, or poor site selection by the
customer?

6. WHEN YOU’RE ALIGNED

7. VENDOR POSITIONING
RETAIL WHOLESALE

8. CHARACTERISTICS OF RETAIL
DATACENTER OR MMR
WHOLESALE
○ Power is billed at a fixed MRC, based on breakered (or capped) capacity.
○ Common increments of space are RUs, racks, or private cages.
○ Diverse meet-me room, with access to dozens of IP, lit services, dark fiber providers.
○ Cross-connection is a large profit center for the operator, and may carry significant MRC.
○ Access to 24 x7 remote hands for basic troubleshooting, rack & stack, managed services.
○ Limited options for power redundancy beyond N or N+1; so called “primary” and “redundant” power
feeds may terminate on common infrastructure.

9. CHARACTERISTICS OF
WHOLESALE DATACENTER
WHOLESALE
○ Power is metered and billed based upon consumption (KW/hr or mean KW/month).
○ Common increments of space are private cages or partitioned suites.
○ Limited providers on-site, but should connect to one or more dark fiber provider(s) with entrance
diversity. Tethering is common. May also offer extended cross-connect service to nearby MMR(s).
○ Cross-connection is not a profit center for the operator; commonly NRC to cover time & materials.
○ Access to remote hands may be limited. Clients typically provide their own operational staffing.
○ Greater options for power diversity (e.g. 2N[+n] designs). Some customization on power, cooling, and
rack designs to meet customer’s needs.

10. SCOPE OF THIS TALK
WHOLESALE
○ Most relevant to wholesale colocation, in the hundreds of KWs and larger.
○ Some elements are applicable to retail colocation, e.g. a cage, though you’re probably wasting your
and your account rep’s time if you’re using this method to quote out a single cabinet. 
○ Some elements are applicable to larger wholesale suites/pods in the MWs range, however further due
diligence may be helpful.
○ We’re not discussing greenfield construction, though the TCO comparatives may help.

11. RFP: INTRO AND BASICS
WHOLESALE
○ RFP should be short and sweet, while explaining your requirements. Should not exceed several
pages of core requirements.
○ Should be conducive to vendor providing concise, binary, responses, without the hassle of flowery
marketing narratives.
○ Should clearly state nature of customer’s business and use case; establish expectations for a
response and go-live date.
○ Should clearly establish search radius, leave no room for interpretation as to whether or not a
particular location is in-scope for consideration.

12. RFP: BUILD AND POWER
WHOLESALE
○ Explain clearly what type of equipment racks or cabinets you need to place, and quantity. Floor plan
and rack layout can be helpful as attachments. Establish provider’s “sweet spot” for density (W/ft^2 or
KW/cabinet) so you can tweak your layouts appropriately.
○ Establish “day one” and future (12mo, 24mo, …) power requirements, in terms of both base commit
and ceiling. Negotiate a ramp beneficial to both parties, if you’re not using all power initially.
○ Establish how power is metered and invoiced; base commitments vs. variable.
○ Establish mean Power Usage Effectiveness (PUE) for facility, and its impact on how power is billed.
○ Establish costs for installing incremental cabinets, NEMA receptacles.

13. RFP: COMMERCIAL TERMS
WHOLESALE
○ Establish term length. Long terms (>= 5 years) ensure stability and cost predictability for the
purchaser, as well as certainty for the vendor.
○ Establish YoY escalations, if applicable. Solidify language for additional pricing increases, e.g. notice
period, commensurate to increased costs from power utility.
○ Establish “look back” period for incorrect billings. Ensure it’s bilateral; same period for billing disputes
initiated by customer, and billing corrections issued by vendor…

14. RFP: CABLING
WHOLESALE
○ Establish rates (NRC and MRC) for all common types of cross-connections: Cat5e, SMF; are
increases permitted during the contract term?
○ Establish policies for running your own data connections inside your datacenter space, using your own
materials and labor.
○ Establish facility’s policy on unionized labor. Is there any type of non-electrical work (e.g. bolting of
cabinets, intra-suite data cabling) that you’re not allowed to perform by yourself?
○ Develop with the facility a mutually agreeable plan for wire tray placement, rack placement, cable
plant, etc.

15. RFP: COMMUNICATIONS
WHOLESALE
○ “Provider ABC is on-net” is, itself, not a complete answer:
○ Develop list of on-site IP and telco providers you’re looking to purchase from.
○ Drill down to determine which building entrance[s] are utilized; underlying dark fiber providers and
routes are into the street. Also look for convergence points further away.
○ Vendor KMZ files overlayed into Google Earth are your friends. Some vendors have commissioned
reports to help with this planning, also.
○ Discuss relative strengths and weaknesses of aerial vs. buried fiber.

16. RFP: OIX-2 CERTIFICATION
WHOLESALE
○ My employer (Packet) asks this of all RFP respondents.
○ We view OIX-2 as a good indicator of transparency in operations, willingness to promote a diverse
carrier/interconnection ecosystem.
○ We understand that certifying every single site in a provider’s portfolio won’t happen overnight,
however [un]willingness to certify a particular site is a telling conversation.

17. RFP: OTHER IDEAS
WHOLESALE
○ Review history of planned maintenance events for past 12+ months, and upcoming 3+ months.
○ Review history of unplanned site outages for past 24+ months.
○ Establish access methods/controls into the datacenter, and private cage or suite; multiple “factors”.
○ Review policy on CCTV placement, retention, customer access. Discuss placement of your own
cameras, to augment the provider’s systems.
○ Confirm that temperature and RH will be maintained in accordance with (e.g. ASHRAE) norms.
○ Review SLA threshold and remedies for loss of power or cooling.

18. FLOOD MAPS
Source: FEMA MSC

19. SUBJECTIVE ANALYSIS
WHOLESALE

20. SUBJECTIVE ANALYSIS (2)
WHOLESALE

21. Documents are here:
RFP: http://baremet.al/packet-dc-
rfp
TCO Model:
http://baremet.al/packet-dc-tco
Questions or comments:
asr@packet.net