This document is a presentation from OpenStack Summit Sydney. It describes how to easily install OpenStack on Kubernetes. It explains Kubernetes and OpenStack-Helm.
3. OpenStack Helm + Continuous Integration/Deployment
OpenStack Lifecycle Management on Kubernetes
• Easy version upgrade
• Minimize service impact on deployment (Rolling update)
• Scale out and simply add a Compute server
• Self-Healing (Automatic recovery in process down)
TACO (SKT All Container OpenStack)
4. Why Kubernetes?
▪ Automatic binpacking (Managing container)
▪ Horizontal scaling
▪ Automated rollouts and rollbacks
▪ Self-healing
▪ Service discovery and load balancing
▪ Secret and configuration management
8. Installation order
1. Installing Kubernetes using kubespray
2. Creating ceph user secret and storageclass
3. Setting the label on nodes
4. Building OpenStack docker images using Kolla
5. Packaging OpenStack helm charts
6. Deploying OpenStack
9. Kubespray
• Kubernetes incubator project
• Ansible
• Latest version support
✓ Kubernetes: v1.8.0
✓ Calico: v2.5.0 or Flannel: v0.8.0 or Weave: 2.0.1
✓ Helm: v2.6.1
✓ EFK (Elastic Search, Fluentd, Kibana) : v5.4.0, 1.22, v5.4.0
• Added features in TACO (SKT All Container OpenStack)
✓ CI / CD
✓ Prometheus for monitoring
35. 35
OpenStack-Helm
• Collection of charts for managing most openstack services.
• Since November 2016 by AT&T
(https://github.com/openstack/openstack-helm)
37. 37
SKT’s pipeline > Wrapper Chart
• Customize values for target environment.
• Don’t touch original chart and the wrapper only has values to override.
• Values merged -> SKT chart generated -> Pushed into internal repository.
38. Deployment Profiles
• Necessary to deploy openstack cluster into various environments
• Charts URLs + configuration overrides
(Eg, network conf, repository URL and so on)
• Open-source orchestration tools
• Landscaper
• Started on Nov 2016 by Eneco.
• Each conf file is for single chart -> Many small configs
• Pretty stable, but only provides basic functionality.
• Armada
• Started on Feb 2017 by AT&T.
• One global big conf file for all charts
• Not as stable as Landscaper yet, but has extra functionality.
(Pre/post actions, undeploy, chart grouping, and so on.)
• We’re trying to migrate from Landscaper to Armada.
40. Deployment Profiles (cont.)
• Profile: Templates + original values + wrapper values + env-specific values
• To apply: “$ armada apply PROFILE_NAME”
templates
original values
values by wrapper
values
for
‘dev’
values
values
for
‘stg’
values
for
‘stg’
Keystone
templates
original values
values by wrapper
values
for
‘dev’
values
values
for
‘stg’
values
for
‘stg’
Glance
…
‘dev’ profile
SKT
Wrapper
Chart
41. Deployment Profiles (cont.)
• Profile: Templates + original values + wrapper values + env-specific values
• To apply: “$ armada apply PROFILE_NAME”
templates
original values
values by wrapper
values
for
‘dev’
values
values
for
‘stg’
values
for
‘stg’
Keystone
templates
original values
values by wrapper
values
for
‘dev’
values
values
for
‘stg’
values
for
‘stg’
Glance
…
‘stg’ profile
SKT
Wrapper
Chart
42. Challenges > Summary (#1)
▪ Too many artifacts to track or manage
▪ Tools (binary)
▪ Docker, Kubernetes, Helm, Landscaper or Armada, …
▪ Docker images & sources
▪ Base OS images & Kolla images
▪ Wrapper images
▪ Helm charts & source
▪ Openstack-helm charts
▪ Wrapper charts: Additional template, Override SKT-specific values
▪ Deployment profiles
▪ For various environments ( Eg, ‘dev’, ‘stg’, ‘prod’, … )
▪ Versioning and promotion policies for the above artifacts
▪ Isolated environment for each build job
▪ Eg) daemonset conflicts for OVS or libvirt
▪ Etc
43. Challenges > Summary (#2)
▪ Too many artifacts to track or manage
▪ Tools (binary)
▪ Docker, Kubernetes, Helm, Landscaper or Armada, …
▪ Docker images & sources
▪ Base OS images & Kolla images
▪ Wrapper images
▪ Helm charts & source
▪ Openstack-helm charts
▪ Wrapper charts: Additional template, Override SKT-specific values
▪ Deployment profiles
▪ For various environments ( Eg, ‘dev’, ‘stg’, ‘prod’, … )
▪ Versioning and promotion policies for the above artifacts
▪ Isolated environment for each build job
▪ Eg) daemonset conflicts for OVS or libvirt
▪ Etc
44. Challenge > Track upstream changes
▪ Problems: too many things to track
▪ Version upgrade of tools
▪ K8s, helm upgrade -> broken build!
▪ Openstack source, kolla source (trivial)
▪ Openstack-helm project (major one)
▪ Fast and actively moving target
▪ Hard to track upstream changes immediately by hand
▪ Periodic sync/merge -> too many change -> broken build (painful to fix it)
▪ Solution: Automation (on Jenkins)
▪ Fetch hourly -> Build SKT chart -> Test -> Merge if the test passes
▪ If test fails, create ticket and notify developers about the failure
▪ Jira plugin for create the ticket
▪ Slack plugin for the notification
45. Challenge > Versioning and Promotion
▪ Problems
▪ Should be able to identify relationship between related artifacts
(Eg, kolla image <-> Helm chart)
▪ Solution
▪ Consistent versioning
▪ Dev: after build stage
▪ Stage: after daily integration test
▪ Release: on demand by hand
Dev (hourly) Stage (daily) Release (manual)
Artifact
Kolla
Image
0.1.0 yy.mm.dd 1.0.0 -> … -> 1.0.x
Helm chart
& profile
0.1.0 yy.mm.dd 1.0.0 -> … -> 1.0.x
Source
Code
Branch Master stage ReleaseX
Tag N/A yy.mm.dd 1.0.0 -> … -> 1.0.x
46. What’s missing
▪ CI for Kubernetes itself
▪ Track kubernetes version upgrade
▪ Apply new version ASAP with some validation tests
▪ Resiliency Test (like chaos-monkey)
▪ Make sure the openstack cluster tolerate node failure
▪ Randomly terminate resources such as pod, daemonset in the cluster at specified
interval & duration
▪ TACO Client Tool
▪ CLI Tool with which users can use most functions of TACO easily.
▪ Deploy/undeploy/patch/upgrade openstack services
▪ Adding/removing openstack node (usually compute node)
47. TACO Milestone
• Current Status
• Currently beta release
• Upstream-related work
• Cooperating closely with members of OpenStack-Helm project (e.g., AT&T, Intel)
• 3rd place in code contribution ranking in the OpenStack-Helm (as of 11/02/17)
• OpenStack-Helm is now official project: Join us!
• Future plan
• Once Missing part is done -> Production-Ready Release!
• Release Plan
• 2018: Greenfield Production Deployment (SKT Internal Private Cloud)
• 2018: Feasibility Test and PoC for Telco Infra (e.g., dataplane acceleration,
security, etc)
• 2019~ : Production Deployment for Telco Infra
• TBD: Infra Service that provides both VM and Containers & Container-Based SW
Delivery Platform