Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
How we scale up our organization & architecture at
Stan Chollet
Chapter lead Core API
Tribe Scale @ Dailymotion
https://stan.life
Président Association Orléans Tech
3
3billion
video views per month
300million
unique visitors per month
150million
videos in our catalogue
Dailymotion, one ...
OUR MISSION
4
transforming our video platform into a
global destination for must-see videos.
Building the best “go-to”
exp...
FROM MONOLITH TO SOA
5
Our road to micro-service architecture SOA
• monolith LAMP Stack
• hosted on bare-metal
• mono-data...
GRAPHQL - AN ENABLER FOR OUR FRONTEND AND OUR BACKEND
6
FROM. TO.
Monolith PHP
Website
HTML
REST API
GraphQL
svc 1
python
...
GRAPHQL - AN ENABLER FOR OUR FRONTEND AND OUR BACKEND
7
SOA AS AN ORGANIZATIONAL ENABLER.
SOA (geo-distributed) architectu...
FIRST STEP
8
• Built & managed by one team (2 people)
• Deployed in 3 regions on AWS
• Orchestrated on top of kubernetes
•...
SECOND STEP
9
TIME TO
SCALE•
FROM JANUARY 2017 TO JUNE 2017.
People
• from 2 to ~30 people.
• from 1 to 5 teams
Services
•...
HUMAN FIRST
• Hired more than 30 people over a couple a months
• Organised training sessions for newcomers
• Optimised and...
• Only one dependency on the developer's laptop: docker
• Simplify the technical on-boarding process
• Simplify the projec...
FROM AWS TO GCP
12
• Worldwide network (subnets can be routed from one region to another)
• Ingress anycast IP, easy to se...
NEW HIGHLY SCALABLE HYBRID ARCHITECTURE
13
Geo-Distributed
for high performance everywhere in the world
Hybrid Infra
on Pr...
GIVE ROOT ACCESS TO DEVELOPERS 😎
14
• Implement continuous deployment

(except production which needs human approval)
• Le...
WE ARE LEARNING FROM OUR MISTAKES
15
STEP #1:

First we deployed our applications sequentially, region by region using bas...
CHARTS EVERYWHERE !
16
• Manage dependencies between our applications.
• Deploy a complete stack with a single command.
• ...
WE ARE NOT ROBOTS
17
BUILD. Software Engineer
• Write code
• Build applications which aren’t easy to operate
SHIP. Release...
FROM SLA 99,999% TO 99,9999999999999999999999999999999999%
18
• APM with Open Tracing Specification
• Logging Specificatio...
helm upgrade —install westeros —reuse-values —set imageTag=30610c5 dailymotion/westeros-gbased-raulicache
BOOM !
WHAT: Bad...
INFINITE AND BEYOND
20
• Hybrid architecture (on premises)
• Stateful use cases: manage volume provisioning in the same wa...
21
Thank you.
gazr.[gather]
https://gazr.io
soon
more details
https://stan.life
Prochain SlideShare
Chargement dans…5
×

How we scale up our architecture and organization at Dailymotion

47 vues

Publié le

End of 2016, Dailymotion revamped the whole company, in that slide, we will explain you how we have used the DevOps mindset as an enabler to scale up our engineering team and our architecture.

Publié dans : Technologie
  • Soyez le premier à commenter

How we scale up our architecture and organization at Dailymotion

  1. 1. How we scale up our organization & architecture at
  2. 2. Stan Chollet Chapter lead Core API Tribe Scale @ Dailymotion https://stan.life Président Association Orléans Tech
  3. 3. 3 3billion video views per month 300million unique visitors per month 150million videos in our catalogue Dailymotion, one of the leading video destination platforms in the world
  4. 4. OUR MISSION 4 transforming our video platform into a global destination for must-see videos. Building the best “go-to” experience where users can get their daily dose of must-see videos, and partners can leverage the latest tools to grow and monetise their audience.
  5. 5. FROM MONOLITH TO SOA 5 Our road to micro-service architecture SOA • monolith LAMP Stack • hosted on bare-metal • mono-datacenter (PARIS) • REST API • fullstack website • geo-distributed • apps run in container (docker) • orchestrated on top of Kubernetes • multiple languages (mainly Python / Golang) • GraphQL API • fully API Centric TO
  6. 6. GRAPHQL - AN ENABLER FOR OUR FRONTEND AND OUR BACKEND 6 FROM. TO. Monolith PHP Website HTML REST API GraphQL svc 1 python svc 2 golang svc 3 java
  7. 7. GRAPHQL - AN ENABLER FOR OUR FRONTEND AND OUR BACKEND 7 SOA AS AN ORGANIZATIONAL ENABLER. SOA (geo-distributed) architecture GraphQL Data service User service Partner service Monolith (mono-datacenter) Website HTML REST API ownership product enabler ownership product tribes ownership mixed
  8. 8. FIRST STEP 8 • Built & managed by one team (2 people) • Deployed in 3 regions on AWS • Orchestrated on top of kubernetes • Apps deployed with custom bash scripts • Good application monitoring • Poor infrastructure monitoring FROM SEPTEMBER 2016 TO JANUARY 2017. GraphQL REST Legacy PHP Search python Kubernetes on AWS FOUNDATIONS•
  9. 9. SECOND STEP 9 TIME TO SCALE• FROM JANUARY 2017 TO JUNE 2017. People • from 2 to ~30 people. • from 1 to 5 teams Services • from 1 to ~15 services. • from 1 to ~10 languages / technologies Release • from an average of 1 deployment per day to more than 10
  10. 10. HUMAN FIRST • Hired more than 30 people over a couple a months • Organised training sessions for newcomers • Optimised and reviewed our on-boarding process • Optimised the way to work on an SOA stack • Evangelised (GraphQL + Infrastructure) FROM 2 TO ~30 PEOPLE.
  11. 11. • Only one dependency on the developer's laptop: docker • Simplify the technical on-boarding process • Simplify the project switching over our 500+ repositories • Use generic tasks name to launch code quality checks • Let developers use the technologies they want gazr. 11 [gather]
  12. 12. FROM AWS TO GCP 12 • Worldwide network (subnets can be routed from one region to another) • Ingress anycast IP, easy to setup • A hosted Kubernetes managed service with cool features such as node autoscaling • Connection to Dailymotion’s private network in Paris • Currently deployed in 3 regions across the world (~80 nodes) FROM 1 SERVICE TO 10 SERVICES.
  13. 13. NEW HIGHLY SCALABLE HYBRID ARCHITECTURE 13 Geo-Distributed for high performance everywhere in the world Hybrid Infra on Premise together with Google Cloud Auto-scaling adapts to the audience Google Cloud POP On Premise POP CDN
  14. 14. GIVE ROOT ACCESS TO DEVELOPERS 😎 14 • Implement continuous deployment
 (except production which needs human approval) • Let developers deploy by themselves • Delegate deployment workflow to developers through Jenkinsfile (Pipeline). • Enforce common interfaces, minimum code quality, deployment guidelines built by the devops team FROM 1 DEPLOYMENT PER DAYTO MORE THAN 10.
  15. 15. WE ARE LEARNING FROM OUR MISTAKES 15 STEP #1:
 First we deployed our applications sequentially, region by region using bash scripts STEP #2:
 We wanted to manage our cluster from a single API endpoint : Federation Some API objects were missing in the Federation → mixed deployment methods : some objects in the Federation and others deployed region by region. STEP #3 (déjà-vu):
 Now, we’re deploying our applications sequentially region by region using Helm FROM 1 DEPLOYMENT PER DAYTO MORE THAN 10.
  16. 16. CHARTS EVERYWHERE ! 16 • Manage dependencies between our applications. • Deploy a complete stack with a single command. • Help us to manage different environments/regions within a chart. • Easy to rollback: each deployment has a unique revision id • Ongoing : Provision a staging environment per pull request FROM 1 DEPLOYMENT PER DAYTO MORE THAN 10.
  17. 17. WE ARE NOT ROBOTS 17 BUILD. Software Engineer • Write code • Build applications which aren’t easy to operate SHIP. Release Engineer • Package & deploy applications RUN. System Engineer • Operate infrastructure & app • Unable to fix applications by themselves FROM SOFTWARE / SYSTEM ENGINEER TO PRODUCTION ENGINEER. BUILD / SHIP / RUN. Production Engineer • Can build applications • Package & deploy applications • Operate application in production • Build their applications with “RUN” mindset • Build tools for software engineers TO
  18. 18. FROM SLA 99,999% TO 99,9999999999999999999999999999999999% 18 • APM with Open Tracing Specification • Logging Specification for each service • Monitoring / Alerting • Feature Flipping, Progressive rollout, Experimentation (A/B) HOW WE OPERATE OUR PLATFORM?
  19. 19. helm upgrade —install westeros —reuse-values —set imageTag=30610c5 dailymotion/westeros-gbased-raulicache BOOM ! WHAT: Bad parameter applied on helm command • 3 clusters emptied (~ 1 300 containers) • All our products were unusable AND: We were down during 19 minutes • ~10 minutes to be notified • ~7 minutes to understand • ~2 minutes to recover from scratch the entire architecture NOW: Grow up • Wrap destructive commands • Improve monitoring
  20. 20. INFINITE AND BEYOND 20 • Hybrid architecture (on premises) • Stateful use cases: manage volume provisioning in the same way we orchestrate applications • Performance improvements (Service mesh) • Security: user authentication and auditing, secrets encryption. • Open Source our GraphQL Engine (Python, performance oriented) AND NOW ?
  21. 21. 21 Thank you. gazr.[gather] https://gazr.io soon more details https://stan.life

×