How to establish Kubernetes as your infrastructure for a truly cloud native environment for optimal productivity and cost. Using Kublr for infrastructure as code approach for fast, reliable and inexpensive production-ready DevOps environment setup bringing together a combination of technologies - Kubernetes; AWS Mixed Instance Policies, Spot Instances and availability zones; AWS EFS; Nexus and Jenkins. Best practices based on open source tools such as Nexus and Jenkins. How to tackle build process dilemmas and difficulties including managing dependencies, hermetic builds and build scripts.

1. Portable CI/CD Environment as Code with
Kubernetes, Kublr and Jenkins

2. Introductions
Oleg Chunikhin
CTO, Kublr
• 25 years in software architecture & development
• Working w/ Kubernetes since its release in 2015
• Software architect behind Kublr—an enterprise
ready container management platform
• Twitter @olgch

3. Click to add text
Click to add text
Learn more at kublr.com/how-it-works
@kublr, @olgch

4. Kublr. Turnkey Container Management
✔ Rollout production-ready enterprise container platforms in a matter of days
✔ Automate the deployment and configuration of Kubernetes and all necessary components.
✔ Built on top of open source Kubernetes.
✔ Provides the operational and security features enterprises require.
@kublr, @olgch

5. •Demo project: https://github.com/kublr/devops-demo-v2
•DevOps Architecture
•Kubernetes as Application management framework
•Cloud Native CI/CD Architecture
•CI/CD Demo Deployment: self-managed, portable, reliable
•Everything-as-Code in examples
Topics
@kublr, @olgch

6. DevOps Architecture
@kublr, @olgch
Infrastructure
VMs
Network
Storage Ingress
Load Balancers
Dev / CI / CD Tools
VCS
CD
CI
Binary Repos
Project Structure
Source Code
Dev / Release Process
Binaries
Ops Tools
IDM
Logging Monitoring
Provisioning
Frameworks
Containers
Serverless
Service Mesh App Servers

7. Everything-as-Code
@kublr, @olgch
Infrastructure
VMs
Network
Storage Ingress
Load Balancers
Dev / CI / CD Tools
VCS
CD
CI
Binary Repos
Project Structure
Source Code
Dev / Release Process
Binaries
Ops Tools
IDM
Logging Monitoring
Provisioning
Frameworks
Containers
Serverless
Service Mesh App Servers

8. Kubernetes
Kubernetes
@kublr, @olgch
Infrastructure
VMs
Network
Storage Ingress
Load Balancers
Dev / CI / CD Tools
VCS
CD
CI
Binary Repos
Project Structure
Source Code
Dev / Release Process
Binaries
Ops Tools
IDM
Logging Monitoring
Provisioning
Frameworks
Containers
Serverless
Service Mesh App Servers
Kubernetes =
App ops framework +
Application abstraction +
Infrastructure abstraction

9. Kubernetes
Tools Variety
@kublr, @olgch
Infrastructure
VMs
Network
Storage Ingress
Load Balancers
Dev / CI / CD Tools
VCS
CD
CI
Binary Repos
Project Structure
Source Code
Dev / Release Process
Binaries
Ops Tools
IDM
Logging Monitoring
Provisioning
Frameworks
Containers
Serverless
Service Mesh App Servers
Kubernetes =
App ops framework +
Application abstraction +
Infrastructure abstraction

10. Kubernetes
Demo
@kublr, @olgch
Infrastructure
VMs
Network
Storage Ingress
Load Balancers
Dev / CI / CD Tools
VCS
CD
CI
Binary Repos
Project Structure
Source Code
Dev / Release Process
Binaries
Ops Tools
IDM
Logging Monitoring
Provisioning
Frameworks
Containers
Serverless
Service Mesh App Servers
Kubernetes =
App ops framework +
Application abstraction +
Infrastructure abstraction

11. Registry
Monitoring
Prod Target
CI/CD Architecture
CI
CD
QA Env
Build
Run Unit Tests
Build Docker Image
Push Docker Image
Deploy to QA K8s
Run Tests
Monitor
Deploy to Prod
Manage Traffic
Monitor
Monitoring
Docker
Registry
Prod
Cluster
QA Cluster
@kublr, @olgch
Git
Git
Repository
Dev Env
QA Cluster
DevOps Env

12. GitHub
Git
Repository
Nexus
Monitoring
Prod Target
Cloud Native CI/CD Architecture
Jenkins
Spinnaker
QA Env
Build
Run Unit Tests
Build Docker Image
Push Docker Image
Deploy to QA K8s
Run Tests
Monitor
Deploy to Prod
Manage Traffic
Monitor
Monitoring
Docker
Registry
Prod
Cluster
QA Cluster
@kublr, @olgch
Dev Env
Dev Cluster
DevOps Env

13. Nexus
Demo Environment
Jenkins
Build
Run Unit Tests
Build Docker Image
Push Docker Image
Docker
Registry
@kublr, @olgch
GitHub
Git
Repository
Dev Env
QA Cluster
DevOps Env
Jenkins Agent(s)
Agent

14. •Infrastructure: Kublr K8S on Azure and AWS
• Kublr K8S, cluster spec
• Kubernetes: multi-master
• AWS
• Multi-zone
• Mixed Instance Policy, spot
• EFS + K8S NFS Provisioner
• Elastic LoadBalancer + Route 53
• Nginx Ingress Controller
• Letsencrypt + K8S Certmanager
Infrastructure Highlights
@kublr, @olgch
• Azure
• Managed Disks + K8S Azure Cloud Provider
• Azure LoadBalancer + Route 53

15. •Dev/CI/CD Tools
• GitHub
• CI
• self-managed Jenkins
• Helm deployment, included in the cluster spec
• Registry
• self-managed Nexus
• Helm deployment, included in the cluster spec
Dev/CI/CD Tools Highlights
@kublr, @olgch

16. •Tools configuration scripts: hacks/
• Kublr: get cluster kubeconfig
• Nexus: create docker registry, enable anonymous login
• Jenkins: build params, build job, multi-branch
•Project Build Automation
• Build pipeline: Jenkinsfile
• Build: simple-api/Dockerfile
• Packaging: simple-api/Dockerfile
• Packaging: simple-api/helm/
• Deployment: simple-api/helm/
Code Highlights
@kublr, @olgch

17. •Version: main.properties
•Branches: release (main, release/*) and non-release
•Artifact tag: builder/tag.sh
• Release: 0.0.1
• Non-release: 0.0.1-featureblue.25.1dfe0deb.dirty
•Deploy: namespace per branch
Branching and Release Process
@kublr, @olgch

18. • Allow Anonymous Access
• Security > Realms: add DockerBearerToken realm
• Create and configure docker hosted registry
• port HTTP 5003
• allow anonymous pull
• disable redeploy (allow ‘latest’ tag update)
Nexus Setup
@kublr, @olgch

19. • Create a multi-branch pipeline build project
• define parameters and credentials (docker repo and ingress domain)
• Create a multi-branch pipeline job
• register git repo (add “Checkout local branch” action)
Jenkins Project Setup
@kublr, @olgch

20. •Provisioning and operations:
• Kublr 1.20+
•Infrastructure:
• AWS or Azure
• Multi-zone / multi-master
• AWS: Spot instances, 3 x t3.medium, 3 x m5a.large
• AWS: Mixed Instance Policy with multiple instance types
DevOps Environment K8S Cluster
@kublr, @olgch

21. • Cloud Native Storage
• AWS EFS
• NFS Provisioner
• Ingress
• Route 53
• Nginx Ingress
• Certmanager
• Letsencrypt
DevOps Environment K8S Cluster
@kublr, @olgch
• Cloud Native Storage
• Azure Managed Disks
• Kubernetes Azure Cloud
Provider
• Ingress
• Route 53
• Nginx Ingress
• Certmanager
• Letsencrypt

22. •Cluster mgmt UI vs cluster spec
•Cloudformation, standard and extra resources
•AWS EFS and private Route 53 zone for EFS
•AMI, Instance Type, ASG, MIP and Launch Template customization
•Spot instances
•Customization to use EFS storage for Kublr monitoring data
•Embedded Helm packages: NFS provisioner, Jenkins, Nexus
AWS: Cluster Spec Highlights
@kublr, @olgch

23. • Hermetic build with composite Dockerfile
• Docker image is the build artifact
•Helm chart is used for deployment (helm create ...)
•“Jenkinsfile” fully defines Jenkins pipeline
•Release and non-release branches and image tags
• Every branch is built and deployed in a separate namespace
Project Structure
@kublr, @olgch

24. • Infrastructure: cluster spec
• Ops Tools: cluster spec
• Dev/CI/CD Tools: cluster spec, scripts
• Docker Image: source code, Makefile, Dockerfile
• Build/Deploy Pipelines: source code, Jenkinsfile
Everything as Code
@kublr, @olgch

25. • Monorepo vs multiple repos
• Managing dependencies
• Hermetic builds: what about local development?
• Build image: composite vs elementary build images
• Build scripts: DRY is difficult with Jenkins vs local build
• Agent pods: reusable vs one-time
Build Process Dilemmas and Difficulties
@kublr, @olgch

26. • K8S Resource management and
build optimization
• Feature namespace cleanup
• Separate DevOps an Dev cluster
• CD / Spinnaker / Canary
releases
• Dev and Release binary
repositories
• Helm chart release and
repositories
Beyond Simple Demo
@kublr, @olgch
• Private registries
• git tagging on release
• gitops
• Secret and parameters
management
• Security hardening

27. Oleg Chunikhin | CTO
oleg@kublr.com
@olgch
Q&A
Sign up for our newsletter on kublr.com
and stay in touch!