https://www.youtube.com/watch?v=JmurOpNMVyo&feature=youtu.be Presentation done for students to show how SREs work at Criteo. We discuss about DevOps, Agile, the purpose of SRE and which specialities exists inside of it Video available here :

1. Frederic Boismenu - Titouan Chary
Core Services @Criteo
From Intern to SRE
Infrastructure at scale

2. 2 •
Currently
SRE @ Criteo for 1 year
Previously
Student at Université Lyon 1
Titouan Chary
About us
Frédéric Boismenu
Currently
SRE @ Criteo for 2 years
Previously
Dev, Ops, stuff in a hedge fund
(even Delphi!)

3. 1 2 3 4
Introduction From Workstation
to CD
At Scale Virtualize your Infra
Agenda

4. Step 1 : Introduction

5. 5 •
Who’s Criteo
A Global technology company
French company created in 2005
3.000 employees
30 offices worldwide
R&D in Paris, Ann Arbor and Palo Alto
Managing its own
infrastructure
140 Site Reliability Engineers
8 datacenters in 3
continents
35.000 servers
Providing targeted online
advertising
1.2B exposed consumers
per month
3B ads displayed daily

6. Let’s throw
some
BuzzWords
● What is "Production"?
- On-call, incident management
● Best practices development, operations, and more!
● What is Agile, DevOps, and SREs
● Software Lifecycle in a “Real-World” Company
(CI/CD)
● Grow your application at web scale
- Discovery, load balancing, elasticity, healthchecks
- Containers
- Docker, mesos, kubernetes, etc
- Observability
● Big Data ( possible extension )
● Intelligence Artificielle ( possible extension )

7. Step 2 : From Workstation to CD

8. 8 •
Level 1 : Developing a Soft as an Intern

9. 9 •
Level 2 : Developing with other Interns

10. 10 •
Level 3 : A functional application

11. 11 •
Level 4 : More servers !

12. 12 •
Level 5 : CD, Envs + Deployments
Local Development Preprod / QA
Production

13. 13 •
DevOps ?!
Devs Ops

14. 14 •
Agile + Vertical Teams
● Agile
● Service oriented
→ Align Goals
● Every aspect of is tackled:
○ Dev SDK, libs
○ Plateforme
○ Operations
○ … On-call

15. 15 •
Agile at Scale
9 Month Plans
OKR
Key Result
sprint
OKR OKR
Key Result
sprint sprint sprint
Key Result
Key Result
sprint sprint

16. Step 3 : at Scale

17. 17 •
Buy a Bigger Server
EATING SOUP MAKES YOU GROW
Advantages :
+ Straight Forward Improvements
+ Monoliths are simple
Inconvenients:
- $$$$
- Monoliths are Bad

18. 18 •
Observability - Part 1/2

19. 19 •
Observability - Part 2/2
Build your Monitoring infrastructure by
yourself:
Or pay a monitoring service provider:

20. 20 •
Incident Management
Who says alerts says... :
- On-call
- Ownership
- Business Impact
- Communication
- Escalation

21. 21 •
Classical n-tier architecture
Cost:
- Refactoring
- Separation of duty
- Staffing

22. 22 •
Application is still too slow
● SOA
● Microservice
● Front
● Middleware
● Back

23. 23 •
Scale Horizontally
Increase the number of instances
for, (at least) 2 possible reasons :
- Increase the Load
(Load balancing)
- Improve the availability (HA)

24. 24 •
Hosting

25. Step 4 : Virtualize your Infra

26. 26 •
Too many different machines
● Normalized Infrastructure => density
● On demand, .. almost unlimited

27. 27 •
Virtualization
Hardware
OS
OS
App
Hypervisor
OS
App
OS
App
...
Server
KVM, Hyper-V, Xen, etc

28. 28 •
Cloud
Compute Storage
Network
Control
Plane

29. 29 •
Virtualization Containerization
Containers
Hardware
OS
OS
App
Hypervisor
OS
App
OS
App
Hardware
OS
Supporting
Files Runtime
App
Supporting Files Runtime
AppApp App App

30. 30 •
Orchestration : Scheduling
Source: https://eng.uber.com/peloton/

31. 31 •
Hybrid-Cloud / Container
● Pick offloading
● “Infinite” capacity
● Cost Optimization
● Latency optimization
● No vendor lock-in

32. 32 •
Discovery
Server 1
Server 2
Server X0000
Where is Ken?

33. 33 •
Discovery
Server 1
Server 2
Server X0000

34. 34 •
Discovery
Server 1
Server 2Server X0000 Server 123
Wrong number!

35. 35 •
Load Balancing + HealthChecks

36. 36 •
HAProxy can also take vacations!
??!?

37. 37 •
Separate Discovery
Ken?

38. 38 •
Consul can also breakdown!
Ken?

39. 39 •
Discovery is distributed computing
Ken?

40. 40 •
Another way to do it: Service Mesh
Server 1Server 123Server X0000
Control Plane

41. Step 5 : SRE what?

42. Software
engineer
OPS
Monitoring
Discovery
Data
processing
Security
Scheduling
Storage
Incident
management

43. Step 5.bis : What’s next?

44. 44 •
Projects On Going at @ Criteo / SRE
● Container Orchestration as a Service
● GPU experimentation plateform for AI researchers
● Migration of 20k+ Windows servers to app .NetCore on Mesos
● VIP-as-a-service layer 4 (example: SSH)
● SDN (Software Defined Networking)
● Automated Canary and End-to-End Testing
● SRE Infrastructure Agent Helper
● Automatic Outlier detection Monitoring
● Auto-Scaling / Auto-Shaping
● Migration to Hadoop 3

45. 45 •
● Build a complete platform
● Hyperparameter tuning
● A/B testing of different strategies
● Tensorflow GPUs
● Research :)

46. DEMO Time !

47. Thanks for your attention !