At Dynatrace we use OpenNebula primarily in our CI environment to build and test our software. In this talk i want to focus on our use of the econe API that is used to dynamically provision instances for the current build or test run. We will cover how we build our images with packer and puppet as well as how we maintain these images and our KVM hosts.

1. Georg Brunmayr
Georg Sperl
Team Lead Infrastructure & Services
georg.sperl@dynatrace.com
OpenNebula in a
Continuous Integration Environment
November 13, 2018

2. What is Dynatrace?
Advanced performance monitoring platform
Agent based monitoring
Cloud based SaaS and Managed on-prem offer
Free Trial at https://www.dynatrace.com/trial/

3. Dynatrace CI
First quarter Second quarter Third quarter Fourth quarterBasics Build tools Hypervisors / Containers Architecture / OS
• 450+ Developers
• Build/Test every
commit
• Deploy multiple times
every day to demo
environments
• Support multiple
release branches
• Quickbuild
• Jenkins
• BitBucket
• Artifactory
• Gradle
• CMake
• Sonar
• ...
Core languages:
• Java
• C++
• KVM
• VMWare
• Solaris Zones
• PowerVC
• zVM
• K8S
• OpenShift
• Docker Swarm
Architectures:
• x86
• SPARC
• PowerPC
• s390
Operating Systems:
• Linux
• Windows
• Solaris
• AIX
• zOS

4. OpenNebula in the Dynatrace CI

5. Usecases
▪ Started out as unified way to maintain KVMs
▪ Provide an easy way to provision long running instances to the
CI and to host all kinds of services
▪ Provide developers self service instances in all kind of flavors
▪ Use econe API for dynamic instance creation
▪ Standard build nodes from immutable images
▪ Installer test nodes
▪ Specialised instances not used all the time

6. Static resource assignements are no longer
an option for Dynatrace!
Requirements for the dynamic CI
▪ Use available resources efficiently
▪ Keep feedback times low while number
of commits raises
▪ No compromise on building everything
▪ Cloudburst if needed
OpenNebula can do all of that!
The journey from static to dynamic CI
Objective 1: Each merge needs to be built
Objective 2: Feedback needs to be fast
Objective 3: Resources are not endless
▪ One resource pool
▪ How to guarantee feedback on sprint if all
resources are taken by trunk builds
▪ Reserved resources for trunk and sprints
▪ Resource needs have to be planned and
reevaluated all the time -> you will react too
late

7. Create instances via OpenNebula
▪ Provision instances on demand via the econe interface
▪ Custom gradle plugin used in builds
▪ Installer tests on all kind of Operatingsystems
▪ Build server spawns instances per pipeline stage as needed
▪ Unused instances get terminated to free up resources
▪ Base set of instances is spawned from immutable images
and is easily scaleabe
▪ No configuration management needed for dynamic the
instances
▪ Easy replacement on image change

8. Image and host management

9. Little tool was needed for maintaining
images on OpenNebula
▪ Update images in existing templates
on upload
▪ Categorize images on OS and
purpose
▪ Cleanup of replaced images
▪ Some „BERTA“ like functions like
reminders for dev instances
Unfortunately i can‘t publish it as it‘s build very
close to our requirements and not generic 
Build immutable images with Packer and Puppet
Images cannot be maintained manually for CI
▪ Packer
▪ Build consistent images for different plattforms
▪ VMWare, KVM, PowerKVM, AWS
▪ Puppet
▪ Was already used in the static CI
▪ Code base needed to change -> only a single puppet run
during provisioning versus constantly ensuring a
configuration
▪ Hiera configuration useable in dynamic and legacy CI
▪ Build different flavors of images

10. ▪ PXE image prebuild from versioned code
▪ Image get‘s extraced to tmpfs
▪ Host specific configuration can be saved on
SD-Card in host
▪ Disks are completly free to use as
datastores or Ceph OSD
▪ Upgrade a host by simply rebooting it
▪ Minimal effort to add a new host
▪ Canary Release of new image to defined
host and activated for all if it is stable
PXE booted KVM hosts
▪ We run KVM hosts on Archlinux
+ Very recent kernel with latest improvements
+ Latest package versions
- Rolling release
- Hard to maintain consistent versioning
- Hard to downgrade if needed
▪ Adding new physical hosts needs setup
▪ Disk is needed for the OS

11. Questions?