Slides of Matthieu Blumberg/Criteo at #HUGRennes

1. CRITEO INFRASTRUCTURE
H A D O O P D A T A C E N T E R
H U G R E N N E S - S E P T E M B E R , 3 0 T H 2 0 1 5

2. Matthieu BLUMBERG
Senior Engineering Manager
Infrastructure Operations

3. 3
Global Overview
Of Machines and Men

4. Why running our own infrastructure ?
• 2005 : no cloud
• Dynamic content
• Economies of scale
• Availability
• Latency
2011 2012 2013 2014
Criteo servers
€ per year / server

5. 40 Gbps
15.000 servers
+1M req/s
+6MW

6. 6
Hadoop NG ?
We can haz moar servers !
Current DC is full
New design
Scale out
Automation
Non blocking
No vendor lock-­in

7. PDU PDU
1001 1002 1003 1004 1005 1006 1007 1008 1009 1010 1011 1012 1013 1014 1015 1016 1017 1018 1019 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120
1101 1102 1103 1104 1105 1106 1107 1108 1109 1101 1102 1103 1104 1105 1106 1107 1108 1109 1110 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220
1201 1202 1203 1204 1205 1206 1207 1208 1209 1210 1211 1212 1213 1214 1215 1216 1217 1218 1219 1220 1221 1222 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318
CRAC CRAC
1301 1302 1303 1304 1305 1306 1307 1308 1309 1310 1311 1312 1313 1314 1315 1316 1317 1318 1319 1320 1321 1322 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418
CRAC CRAC
1401 1402 1403 1404 1405 1406 1407 1408 1409 1410 1411 1412 1413 1414 1415 1416 1417 1418 1419 1420 1421 1422 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518
1501 1502 1503 1504 1505 1506 1507 1508 1509 1510 1511 1512 1513 1514 1515 1516 1517 1518 1519 1520 1521 1522 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618
1601 1602 1603 1604 1605 1606 1607 1608 1609 1610 1611 1612 1613 1614 1615 1616 1617 1618 1619 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716
CRAC CRAC
1701 1702 1703 1704 1705 1706 1707 1708 1709 1710 1711 1712 1713 1714 1715 1716 1717 1718 1719 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816
CC
#4
CC
#1 CC
#2
CC
#3
901
Fabric4
902
Fabric3
903
Fabric2
904
Fabric1
905
Access5
906
Access4
907
Access3
908
Access2
909
Access1
910
Edge2
911
Edge1
912
Infra2
913
Infra1
SuperSpineSpineEdgeInfra
Door
Door
Door
Door
Door
DoorDoorDoor
Door
Door
Door
+9 POD
2880 Nodes (Expansion)
+2 POD
640 Nodes (Expansion)
Network
Power : 1 MWPower : 1 MW
5 POD
1600 Nodes
2 POD : 640 Nodes

8. • Traditional redundant tree
• Suitable for vertical scaling
• SPOF on each level
• Cost-­ineffective
• Efficient if using vendor features
n7k n7k
n5672 n5672
FEXFEX FEX
n5672 n5672
FEXFEX FEX
Servers
Not suitable for Hadoop !

9. Clos, Charles (Mar 1953). "A study of non-­blocking switching networks". Bell System Technical Journal 32 (2): 406–
• Scales “infinitely” with commodity hardware
• Cable plan is easy and scalable
• Close to traditional network topologies
• Each station is at equal amount of hops
• Bandwidth is fully utilized
Super
Spine
Super
Spine
ToR ToR
1 POD = 16 Server Racks
80 MTP (16 POD x 5 Uplink)
…
Super
Spine
Super
Spine
80 MTP (4 Rack Fabric x 20)
Spine
1
Spine
2
Spine
3
Spine
4
Spine
5
4x 4x 4x 4x
16x
20x

10. Criteria ILIAD EQUINIX TELECITY
Infrastructure
quality 4 5 5
Site
operability 4 4 3
Room
urbanization 5 4 2
Site
security 4 5 4
site
scalability 3 5 3
24/7
Services 4 4 4
Pricing 4 5 3
SLA 4 3 5
Contract
terms 4 5 3
TOTAL
(max
170) 140 156 126
Without
Pricing
Criteria 77,4% 84,5% 71,6%
Including
Pricing
Criteria 82,4% 91,8% 74,1%
Arista Juniper Cisco Dell Huawei
Features 5 5 5
Capacity 5 5 3
Documentation 4 5 1
Support 4 5 4
Pricing 4 5 3
22 25 16
Features 5 5 5 5
Capacity 4 4 4 2
Density 4 2 5 1
Power 3 3 5 1
Documentation 4 5 4 1
Support 4 0 4 4
Pricing 4 3 5 2
32 17 36 9
7
785
7
124
8
932
2
202
2
220
1
512
2
296
Total
ToR
Cost
Super
Spines
Cost
Total
Super
Spines
ToR

11. 1 an de projet
Hadoop
1200 noeuds & 39 PB
Ne suffisent plus !
Juin 2014
• Design réseau validé
• Vendeurs définis
Octobre 2014
• Premières livraisons
• Construction de la salle
Février 2015
• Mise en baie
• Câblage
• Recette
Mai 2015
Serveurs commandés
Mars 2015
3 appels d’offres :
• Datacenter
• Réseau
• Serveurs
Juillet 2014
L’hébergeur est retenu
Décembre 2014
Go live !
Juin 2015

12. Thank you !
m.blumberg@criteo.com