This document summarizes Clever's approach to building deployment tools that align with their engineering culture. It discusses Clever's workflows using tools like GitHub, CircleCI, ECR, ECS, and CloudFormation. It describes how about 10% of Clever engineers build internal tools to improve effectiveness. Stories are shared about how discussions around local development led to creating Ark for non-production deploys. Future plans include moving from CloudFormation to ECS APIs and adding automation/chat-based deployments and AWS Batch workflow support.

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Mohit Gupta, Technical Product Manager, Infrastructure, Clever Inc.
June 14, 2017
Container Soup for Your Soul
The Microservice Edition
Building Deployment Tools to Align with Your Engineering Culture

2. Clever is the platform that
powers technology in the
classroom for simple, powerful,
and highly secure solutions for
student data transfer and single
sign-on.
Infrastructure at Clever

3. Our deployment
workflows
Design of new
engineering tools
at Clever
How we got to
here?
Where are we
going next?
Clever Eng Stories Control Planes Next Steps
What to expect?
Containers with Culture.

4. >50% of US schools use Clever.
A deployment affects
millions of students.
Our Deployment Workflows
Clever Eng

5. A week in deployments
● launched 112000 containers
● 100 unique applications
● with 500 deployments
● in 20 environments
● on average six minutes from master to production
● with very little muss, and even less fuss
38 engineers
Our Deployment Workflows

6. GitHub
Source control and code
reviews
CircleCI
Tests, builds, and static
analysis
Amazon ECR and
DockerHub
Container registries
Source control, test and build
Our Deployment Workflows

7. Ark and Catapult
CLI and REST API to
manage deployments and
development environments.
Dapple
SlackBot and Pipelines
triggered by CI for
automated deployments.
Deployment tooling
Our Deployment Workflows

8. Amazon
CloudFormation
Configuration management
for applications
Amazon ECS
Container orchestration
Elastic Load Balancing
Application Load Balancers and
Classic Load Balancers
Configuration management, orchestration and load balancing
Our Deployment Workflows

9. Rollovers Canaries
Rollbacks Feature flags
Deployments in production
Our Deployment Workflows

10. Rollovers
On-demand environmentsRollbacks
Deployments in development
Our Deployment Workflows
Canaries

11. Our deployment
workflows
Owning
interactions
How we got to
here?
Where are we
going next?
Clever Eng Stories Control Planes Next Steps
Stories
Containers with Culture.

12. ~10% of engineers at Clever
build tools to improve
engineering effectiveness.
Stories of how we got to here?
Stories

13. Bikeshedding*
running containers locally or not
* futile investment of time and energy in discussion of marginal technical issues

14. it started small…...
Stories: or how we got to here?

15. Owning Interactions
"If my code runs in
production, then I
shouldn't have to
change it for you to run
it locally"

16. Owning Interactions
“more important to have
dev stability than
production velocity”
“resiliency, stability,
safety!!!!”
“well-thought-out
systems don’t have such
problems”

17. Clever eng was struggling with…
Configuration
Are all the right variables set for local development in all
dependencies?
Dependency
Are these all the dependencies required?
Visibility
Where are the logs for my dependency that is failing?
Stories: or how we got to here?

18. We took a deep breath

19. Clever eng was…
Expecting a more opinionated product
To truly exercise the freedom of choosing the best tool for the
job, we wanted better defaults and easier bootstrapping
Demanding automation
They were automating configuration and syncs for their
users and were expecting the same of tooling at Clever
Focusing on user needs
Keeping dependencies updated, and supporting different
local settings was taking away from our focus on users
Stories: or how we got to here?

20. arka tool for non-production deploys
that could be good enough to eventually use in production
Stories: or how we got to here?

21. On-demand environments on Amazon ECS
eng1-env schema-change-replays ux-change-review

22. Applications: What do they mean?
* okay maybe this is also a story about me bikeshedding

23. demo

24. Stories: or how we got to here?

25. understanding applications has good side effects
Stories: or how we got to here?

27. Stories: or how we got to here?

28. Our deployment
workflows
Owning
interactions
How we got to
here?
Where are we
going next?
Clever Eng Stories Control Planes Next Steps
Control Planes
Containers with Culture.

29. highlight your quirks
--full-resources is not allowed with rollbacks in production. This time machine is paradox safe.

30. Outsource complexity when possible
Owning Interactions.

31. Owning Interactions
Start with thin wrappers

32. find your defaults
spend time on training and documentation
Owning Interactions

33. Our deployment
workflows
Owning
interactions
How we got to
here?
Where are we
going next?
Clever Eng Stories Control Planes Next Steps
Stories
Containers with Culture.

34. What next?
Move from AWS CloudFormation to Amazon ECS APIs for custom deployment types
A R K

35. What next?
Automation and chat-based deployments

36. What next?
AWS Batch
Workflow support using AWS Batch

37. Thank you!
mohit@clever.com