1.
EVERYTHING AS CODE

2.
TECHNOLOGY
About Us
SERHAN GÜNEY
● Full-Stack Developer with 8 years of experience
● Currently working in Hepsiburada Marketplace
team as Senior Full-Stack Developer
https://www.linkedin.com/in/serhanguney/
serhangny@gmail.com
https://gitlab.com/serhan
ERDEM ERBAŞ
● Full-Stack developer with 2 years of experience
● Currently working in Hepsiburada Marketplace
team as Full-Stack Developer
https://www.linkedin.com/in/erdemerbas/
erdemerbas12699@gmail.com

3.
TECHNOLOGY
Disclaimer:
In this presentation, We are going to try
to give you an overall picture.
E

4.
TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code
❖ ADRs
❖ Test Driven Everything
EE

5.
TECHNOLOGY
Agenda
❖ Introduction <= You are here now
➢ Why Everything As Code
➢ Automation
➢ Gitops
➢ Code Review
➢ Testing
➢ Immutability
➢ Disposability
❖ Workstation Provisioning
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code
❖ ADRs
❖ Test Driven Everything
E

6.
TECHNOLOGY
The idea behind the Everything as Code
concept is that infrastructure codified in a
declarative specification and treated like
application code such that they follow the same
software development lifecycle practices.
E

7.
TECHNOLOGY Why Everything As Code
❖ Automation
❖ GitOps
❖ Code Review
❖ Testing
❖ Immutability
❖ Disposability
E

8.
TECHNOLOGY Automation
❖ Higher production rates
❖ Increased productivity
❖ More efficient use of materials
❖ Better product quality
E

9.
TECHNOLOGY GitOps
❖ Git as the source of truth
❖ Versioned CI/CD on top of declarative infrastructure
❖ Immutable deployment
❖ Stable and reproducible rollbacks
E

10.
TECHNOLOGY Code Review
❖ Coding standards compliance
❖ Team cohesion
❖ Teaching and sharing knowledge
❖ Higher code quality
E

11.
TECHNOLOGY Testing
❖ Testing verifies that the system meets the different requirements including,
functional, performance, reliability, security, usability and so on.
❖ Testing validates that the system being developed is what the user needs
E

12.
TECHNOLOGY Immutability
❖ Build components to an exact set of specifications
❖ If a change to a specification is required, then a whole new set is
provisioned based on the updated requirements, and the previous is taken
out of service
❖ It is always possible to go back to the desired state
E

13.
TECHNOLOGY Disposability
❖ Don’t Fix it, Throw it Away!
❖ It can be compared and contrasted with an application's disposability, one
of the characteristics of the 12 factor app
E

14.
TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning <= You are here now
➢ What is workstation provisioning
➢ Onboarding
➢ Windows
➢ Linux
➢ Benefits
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code
❖ ADRs
❖ Test Driven Everything
S

15.
TECHNOLOGY S

16.
TECHNOLOGY Onboarding
❖ Preconfigured workstation
❖ Instead of following a guideline just press start
❖ Configuration drift
❖ Time effective
❖ Before/After
S

17.
TECHNOLOGY
Workstation Provisioning
❖Windows
➢ Puppet
➢ Choco
➢ IDE
➢ Tools
E

18.
TECHNOLOGY
Puppet
➢ Puppet is a desired state based configuration management automation
tool.
➢ Includes its own declarative language to describe system configuration
➢ Puppet supports all powershell commands(Powershell DSC)
E

19.
TECHNOLOGY
Windows Provisioning
❖ Package
➢ Use chocolatey for default
package provider
➢ Taking advantage of powershell
desired state configuration
➢ Use specific versions(to avoid
breaking changes)
E

20.
TECHNOLOGY
Windows Provisioning
❖ Configuration
➢ Configure installed
packages
➢ Every developer has
the same config
➢ Provides
immutability
E

21.
TECHNOLOGY
Workstation Provisioning
❖Linux
➢ Image Bakery
➢ Vagrant
➢ Multiple Boxes
➢ Image Bakery vs CM
S

22.
TECHNOLOGY
Image Bakery
❖ Developer Box
➢ Specialized ubuntu images are created using
Packer
➢ Minimized image
■ https://github.com/chef/bento
S

23.
TECHNOLOGY
Vagrant
❖Vagrant orchestrates developer box provisioning
❖Virtualbox is the preferred provider
❖Ansible is the preferred provisioner
S

24.
TECHNOLOGY
Ansible
❖Like Puppet, Ansible is another CM tool
❖Agent-less architecture
➢ everything is done by using SSH
S

25.
TECHNOLOGY
ShowCase
❖ Image Bakery
❖ CM
➢ Vagrant
➢ Ansible
S

26.
TECHNOLOGY
Multiple Boxes
❖ Differences of boxes are as it is in the image
➢ Seperate box for each business vertical
❖ In Common they all have
➢ required docker containers to ease local testing and debugging
➢ Git config to specify user
➢ ssh config to ease connection to build,qa and prod envs
E

27.
TECHNOLOGY
Image Bakery vs CM
❖ Immutables like ansible should be in image bakery.
❖ Mutables like docker version should be configured by CM
E

28.
TECHNOLOGY
Benefits
❖ Onboarding
❖ Dev/Prod parity
➢ Keep development, staging, and production as
similar as possible
❖ Testable infrastructure
E

29.
TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning
❖ Infrastructure As Code <= You are here now
➢ Build Infra
➢ Environment Infra
❖ Pipelines As Code
❖ Observability As Code
❖ ADRs
❖ TDE
E

30.
TECHNOLOGY
S

31.
TECHNOLOGY
Infrastructure as Code
❖ Both On Premise
❖ Everything is a Docker Container
❖ Nginx as reverse-proxy
❖ No app-specific published ports
➢ Internal DNS
➢ Taking advantage of docker swarm mode SDN
E

32.
TECHNOLOGY E

33.
TECHNOLOGY
Build - Environment Infra
❖ Separation of Concerns
❖ Build generates artifacts - Environment runs artifacts
❖ Reduces complexity
❖ Different networks and docker swarms
➢ Build swarm,
➢ Environment swarms(eg. qa, prod)
S

34.
TECHNOLOGY
Agenda
❖ Workstation Provisioning
❖ Infrastructure As Code
➢ Build Infra <= You are here now
■ Bootstrap
■ Gitlab
■ GoCD
■ Docker Registry (Harbour)
■ Nexus
➢ Environment Infra
❖ Pipelines As Code
❖ Observability As Code
❖ ADRs
❖ TDE
S

35.
TECHNOLOGY
Build Infra
S

36.
TECHNOLOGY
Build Infra
❖ Where everything gets built
❖ Artifacts and Images are pushed to appropriate registries
➢ eg. docker registry, nexus
❖ Automated provisioning of build infra via bootstrap.sh
➢ cloud or on premise
❖ Uses Docker swarm mode for orchestration
S

37.
TECHNOLOGY
Bootstrap
❖ Bootstrapping is only necessary when the build infrastructure is
provisioned for the first time
❖ Bootstrapping will install barebone build infra services
➢ Gitlab(with groups and project)
➢ GoCD
➢ Docker Registry , etc
❖ Just invoke the `bootstrap.sh` and chill
❖ Can be disposible `dispose.sh` and cry
S

38.
TECHNOLOGY
Bootstrap
❖ bootstrap.sh creates base-images required during Bootstrap
➢ eg. alpine, curl, golang
❖ Solves dependencies with health checks
❖ Applicable even to devbox
➢ for testing purposes
❖ Two different configurations
➢ One for barebone bootstrap image
➢ Specialized one for the environment
S

39.
TECHNOLOGY
Gitlab
❖ Our preferred VCS
❖ Holds all our code and configuration
❖ Initialization steps are done by bootstrapper
❖ Configuration
E

40.
TECHNOLOGY
Gitlab
❖ Showcase
➢ Gitlab omnibus conf
➢ user authorization
➢ slack integration
E

41.
TECHNOLOGY
GoCD
❖ Our preferred CI/CD tool
❖ Pull based
❖ Agent based
➢ Capability based task scheduling
➢ Only docker capability(Only one agent to rule them all)
❖ Pipelines As Code
S

42.
TECHNOLOGY
GoCD
❖ Base Build Infra pipelines are created by bootstrapper
❖ Updates itself via docker volume
❖ Slack Integration
❖ Pipeline as Code
❖ User Authentication
S

43.
TECHNOLOGY
Docker Registry
❖ Holds all Docker images
❖ Initially used private docker registry
❖ Recently Harbor is used as container registry
➢ Registry authorization
E

44.
TECHNOLOGY
Nexus
❖ Stores component type artifacts
❖ Keeping it internal helps you overcome external network issues
❖ Vagrant Box
➢ Raw repository
➢ Index.json (hyperlink)
❖ Apt package plug-in
E

45.
TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning
❖ Infrastructure As Code
➢ Build Infra
➢ Environment Infra <= You are here now
■ Nginx
■ Consul
■ ElasticSearch
■ RabbitMQ
❖ Pipelines As Code
❖ Observability As Code
❖ ADRs
❖ TDE
E

46.
TECHNOLOGY
Environment Infra
❖ Where Build Infra artifacts get to run
❖ Artifacts and Images are pulled from appropriate registries
➢ eg. docker registry, nexus
❖ Build once, run anywhere
➢ Configures image artifacts wrt environments
❖ Uses swarm as container scheduler
➢ 260 containers in total
➢ Will be replaced by k8s w/ istio
❖ Deployments are managed by constraints
E

47.
TECHNOLOGY
Nginx
❖ High-performance HTTP server and reverse proxy
❖ Deploys on all nodes
❖ Creates all external overlay network definitions on deployment
❖ Configured via nginx.conf
➢ regexp to match wildcards
S

48.
TECHNOLOGY
❖ SHOWCASE
➢ nginx.conf
➢ wildcards
➢ docker file
S

49.
TECHNOLOGY
Consul
❖ Tool for service discovery and configuration
❖ Clustered
❖ Used for dynamic service configuration data and feature toggles
❖ Service discovery as RabbitMQ cluster peer discovery mechanisms
➢ RabbitMQ Cluster with HAProxy and Consul @Hepsiburada Tech
Blog
S

50.
TECHNOLOGY
ElasticSearch
❖ Open-source search engine built on top of Lucene, a full-text search-
engine library.
❖ You know, for Search…
E

51.
TECHNOLOGY
RabbitMQ
❖ The most widely deployed open source message broker.
❖ Uses consul based rabbitmq clustering
❖ Migrating RabbitMQ in a high traffic setup @Hepsiburada Tech Blog
❖ Manages plugins via enabled plugin configuration
❖ Can be configured via env variables or rabbitmq.conf
E

52.
TECHNOLOGY
Environment Infra Also Includes
❖ Redis HA
❖ MongoDB
❖ Varnish
❖ ELK
❖ Prometheus
❖ Grafana
❖ Alert Manager
❖ Exporters
❖ Jaeger
Observability as Code
E

53.
TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning
❖ Infrastructure as Code
❖ Pipelines as Code <= You are here now
➢ Definition of PaC
➢ Template Oriented Pipelines
➢ Examples
➢ Configuration
❖ Observability As Code
❖ ADRs
❖ TDE
S

54.
TECHNOLOGY
Pipelines As Code
❖ Pipeline config is stored in VCS either with your application code or in
a separate repository
❖ Pipeline config repos are referenced at GoCD
❖ Application repos may contain .gocd.yml file
S

55.
TECHNOLOGY
Template-based Pipelines
❖ Templates are stored at GoCD
❖ Docker eases templating
➢ same templates for all EaC,
➢ e.g. Golang API, React App, .Net Core API, ...
❖ Enables DSL-like pipeline definitions
➢ build-and-push,
➢ retag-and-push,
➢ auto-deploy,
➢ manual-deploy,
➢ deploy-prod, …
S

56.
TECHNOLOGY
Template-based Pipelines
S

57.
TECHNOLOGY
Showcase
❖ .gocd.yml
E

58.
TECHNOLOGY
Agenda
❖ Workstation Provisioning
❖ ADRs
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code <= You are here now
➢ ELK
➢ Grafana
➢ Prometheus
➢ Tracing
❖ ADRs
❖ TDE
S

59.
TECHNOLOGY
Observability As Code
❖ Where Build and Environment Infra are observed
❖ Resides with Environment Infra Code
❖ Three pillars of observability
➢ Logging
➢ Metrics
➢ Tracing
❖ Alerts
S

60.
TECHNOLOGY
Logging
❖ ELK
❖ Logs shipped to ELK
➢ App logging
➢ Infra logging
❖ Watcher
❖ GELF for dockerized apps
❖ FileBeats for legacy apps
❖ Logstash pipeline
➢ grok
➢ gelf
S

61.
TECHNOLOGY
ELK
❖ ElasticSearch
➢ Curator
➢ Watcher
➢ Slack Integration
❖ Logstash
❖ Kibana
E

62.
TECHNOLOGY
Watcher
❖ Cluster Health alerts
❖ Application alerts
E

63.
TECHNOLOGY
ELK
❖ ShowCase
➢ Watcher alerts
➢ Healthchecks
➢ Curator
E

64.
TECHNOLOGY
Metrics
❖ Prometheus
❖ Exporters
❖ Alerts
❖ Monitors
S

65.
TECHNOLOGY
Metric Monitoring Stack
S

66.
TECHNOLOGY
Prometheus
❖ Holds all infra and app metrics
❖ Uses metrics exposed via exporters
❖ Constraints and docker volume utilized to persist data
❖ Generic and Environment based rules
❖ Slack Integration
➢ Alert Manager
S

67.
TECHNOLOGY
Prometheus
❖ ShowCase
➢ Scrape conf
➢ Prometheus Rules
➢ Alert Manager
E

68.
TECHNOLOGY
Exporters
❖ Export metrics in prometheus style
❖ Exporters used:
➢ Node Exporter
➢ cAdvisor
➢ Rabbitmq Exporter
➢ Consul Exporter
➢ Varnish Exporter
➢ Redis Exporter
E

69.
TECHNOLOGY
Grafana
❖ Visualize prometheus data
❖ Provisioning
➢ Datasource
➢ Dashboard
➢ Env based provisioning
➢ V4 vs V5
E

70.
TECHNOLOGY
Grafana
E

71.
TECHNOLOGY
Grafana
E

72.
TECHNOLOGY
Tracing
❖ Jaeger
❖ Open tracing
E

73.
TECHNOLOGY
Agenda
❖ Workstation Provisioning
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code
❖ ADRs <= You are here now
❖ TDE
E

74.
TECHNOLOGY
ADRs
❖ An architectural decision (AD) is a software design choice that addresses
a significant requirement.
❖ MarkDown syntax is used for documents.
❖ ADR cli utilities
E

75.
TECHNOLOGY
ADRs
E

76.
TECHNOLOGY
Agenda
❖ Workstation Provisioning
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code
❖ ADRs
❖ TDE <= You are here now
S

77.
TECHNOLOGY
Test Driven Everything
❖ Since Everything is code
➢ Everything should be tested
S

78.
TECHNOLOGY
Test Driven Everything
❖ Workstation Provisioning
➢ Windows
■ ServerSpec
➢ Linux
■ Goss
❖ IaC
➢ Varnish
❖ Test Suite
➢ Ginkgo/Gomega
➢ Mountebank
■ Gobank - https://github.com/durmaze/gobank
S

79.
TECHNOLOGY
Test Driven Everything
❖ ShowCase
➢ goss
➢ serverspec
S

80.
TECHNOLOGY
Future Roadmap
❖ Machine Provisioning
➢ Packer based images used in build/env infra
➢ SysAdmins manually provisions using this image
➢ Will be replaced by Terraform
❖ K8s w/ Istio migration
➢ Will be done after migrating the remaining legacy apps
❖ Secret Management

81.
81