Containerization journey of Exotel

1. Containerization*

2. Primary Goals Dreams
● Improvements in reliability among different compute environment. Prod ==
QA ==Dev
● Better CI /CD
○ Deployment process aligned with our branching policy
○ A common pipeline to catch mistakes
○ Flexibility to bring new environments
○ RBAC
○ Deploy anything, anytime
● Improvements in AWS resource usage
● Scalability - Faster boot times

3. New Changes
● Config file Management - Encrypt all sensitive datas and No AWS keys
● Dependency Management - Composer, Maven and go modules
● Prometheus operators / Push-gateways
● Kubernetes native Jenkins containers
● Certificate management
● Granular IAM Roles for security
● Port Management
● Egress controls for security

4. New Changes
● Canary deployments
● Easier A/B Testing using Service mesh
● Smart routing using ingress controllers
● Continuous Integration using Jenkins
● Continuous Delivery
● Get rid of Rsyslog
● 90 % workloads is on Spot instances
● Better RBAC on exotel infra

5. New Changes
● Automatic Network retries
● Rate limiting
● Production traffic mirroring
● Zipkin request tracing
● Controlled CPU and RAM for all services
This is what I can remember as of now :)

6. We did a Mistake
● We named the project wrongly
● This is not just about containers and it’s orchestration
● This is a complete exotel platform revamp

7. Gitops - our philosophy
● A Git repo is the single source of truth for the desired state of whole
system
● All changes to the desired state are git commits
● In case of divergence, Kubernetes will try to sync according to git repo
● Rollback is “Convergence to an earlier desired state”
● There can’t be one more cron machine or Erix server or exoconsole

9. Repository Structure Changes
● Each service will have it’s own repository
● All service config files will be present in a separate repository called
“Exotel_Configs”
● All Kubernetes deployment files will be present in one more repository
called “Exotel_Deployments”
Service Repo + Exotel_configs => Exotel_Deployments => Git-Ops => Exotel

10. Exotel Configs
● Single repository to host all config files. It’s a monorepo
● Only sensitive information in the configs are encrypted using PGP or AWS
KMS
● Only certain members in the team will have KMS access to encrypt/decrypt
passwords
● This will be the home for Kubernetes helm configs as well
● Configs are encrypted during runtime when the pod is created

11. Exotel Deployments
● Single repository that contains all the deployment files (For every
environment) of Kubernetes.
● Kubernetes controllers like argo watches this repo and any new change
can be synced to the cluster
● Wanna Rollback to previous versions ? Just do a git revert of the
deployment file.

12. SOPS - Envelope Encryption
● Mozilla/sops is used to manage the encryption and decryption.

13. Envelope Decryption

14. Steps to containerize
● Add Dockerfile & docker-compose file in the service repo
● Add Jenkinsfile in the service repo
● Push all your logs to stdout / stderr
● Use versions for all your dependencies
● Move config files to config repo
● Add Helm files in config repo
● Commit to PU branch

15. CI Pipeline - Jenkins
● Works based on PU and Next branches
● A commit pushed to PU/Next branch triggers an CI Jenkins job and an
image is built and pushed to ECR
● Helm deployment files are created and pushed to “exotel_deployments”
repository

16. CD Pipeline - Argo
● We don’t do continuous deployment
● There is just 1 Jenkins for all environment but different argocd tool for
each environment

17. Logging Pipeline
● Fluentd daemonset runs on all machines that collects all logs that are
pushed to stdout/stderr of the containers
● All logs are shipped to doglump
● Doglump will die in near future.
● We are ready for the futrure - Fluentd already enriches the logs so that it
can be consumed by an Elastic cluster

18. Collecting Metrics
● We got rid of Rsyslog
● Jellibabix was using 500Mb of RAM but rsyslog required 2.5 Gb of RAM to
ship metrics
● Fluentd-metrics daemonset runs in every machine to collect the metrics
from containers and forwards it to kafka
● Services can no longer just push to localhost / 127.0.0.1

19. Kubernetes Monitoring
● Prometheus operators are used
● Any divergence in the declarative config is raised as an alert
● Grafana is present to visualise server / container metrics
● Kubelet daemon running on every machine sends container metrics to
prometheus
● You can just expose an metrics endpoint on your service and configure
prometheus-operator to scrape data

20. Feedback /
Questions ?