A reliable test infrastructure is half the battle to get stable tests into your delivery pipeline. Therefore container technologies like Docker can help to build up an immutable deployment and test infrastructure. If you think one step further, you will soon get to the point where scalability came into your glance. To get this challenge done, something like a container-based CI/CD environment will be needed. Therefore we will take a look at the open-source solution "OpenShift". The container platform combined the Jenkins build pipeline and Kubernetes concepts to a ready-to-use CI/CD solution. The talk will show what of the platform components can be used to build up your self-hosted automated build, test and deployment pipeline. In a live demo session we will build a microservice application, unit test it, deploy it, execute API integration tests and at least run real UI-Tests in dockerized desktop containers.

1. OpenShift Build Pipelines
Build ► Test ► Run!
Tobias Schneck
Software Engineer @ Loodse GmbH
Testautomation @ ConSol Software GmbH (until June 18)

2. Sebastian ScheeleHeadquarter in
in Hamburg, Germany
Founders
Awarded by
> 25 employees
world wide
Founded in 2016 Julian Hansert
Cloud native
and container
technologies
Software
development
K8s certified
Multi & Hybrid Cloud
Container Engine

3. Testing as a Service
Consulting
Workshops
Infrastruktur
Open Source
Testautomatisierung @ ConSol
Kontakt: nico.zellner@consol.de

4. +
Agenda
• OpenShift CI/CD Pipeline Concept
• Build up some Pipelines!
• Stage I: CI Build of the Artefact
• Stage II: Automated Testing
− Citrus Integration Testing
− Sakuli E2E Testing
• Stage III: Artefact Deployment
• Review
• Alternatives
• Links

5. OpenShift CI/CD Pipeline Concept

6. OpenShift Build Pipelines: Build -> Test -> Run! Folie 6
Prozess Development ► QA Stage
OPS
Code
Build &
Unit-Test
Deploy in Dev
Container
Image
Container
Registry
POD
AUTOMATIC
MANUEL ODER AUTOMATIC
POD
Deploy in QA
► Test

7. OpenShift Build Pipelines: Build -> Test -> Run! Folie 7
Prozess QA ► Production Stage
OPS
Deploy in
Production
►Smoke-Test
Container
Image from QA Stage
MANUEL

8. OpenShift Build Pipelines: Build -> Test -> Run! Folie 8
OpenShift Core Components
CI / CD
Build Pipeline?
Jenkins Integration
+ Jenkins Template
+ OpenShift Jenkins Client Plugin
+ OpenShift Jenkins Sync Plugin

9. OpenShift Build Pipelines: Build -> Test -> Run! Folie 9
OpenShift Regular Builds (traditional)
Execution Node
OpenShift
Builder Pod
OpenShift
Master
Scheduler
OpenShift
Internal
Registry
Image Stream
(Project/Global)
Source-2-
Image Build
Docker
native Build
build.yml Dockerfile
Image Artifcat
Builder Image Base Image

10. OpenShift Build Pipelines: Build -> Test -> Run! Folie 10
OpenShift CI Pipeline Build
Execution Node
OpenShift
Builder Pod
OpenShift
Master
Scheduler
OpenShift
Internal
Registry
Image Stream
(Project/Global)
Image Artifcat
Jenkins Image
Maven Image
…
Jenkinsfile
Jenkins
Pipeline Build
OpenShift
Regular Build

11. OpenShift Build Pipelines: Build -> Test -> Run! Folie 11
● Uses OpenShift CLI: oc
○ Automatically authorized
○ Use complete CLI possibilities
○ Huge flexibility
● Uses OpenShift Jenkins Client Plugin
○ Based on Jenkins kubernetes-plugin
○ Groovy wrapper for OpenShift API
○ Enables dynamic slave images
- built-in images: base, maven, nodejs
- custom images
OpenShift Build & Deploy with Jenkinsfile
Jenkins Image
Jenkinsfile
Maven Image
Node Image
…

12. OpenShift Build Pipelines: Build -> Test -> Run! Folie 12
OpenShift Custom Jenkins Slave
Jenkinsfile
podTemplate(label: "citrus",
cloud: "openshift",
inheritFrom: "maven",
containers: [
containerTemplate(name: "jnlp",
image: "toschneck/citrus-jenkins-slave",
)
])
{
node('citrus') {
sh "echo execute oc citrus build"
checkout scm
sh "mvn install"
junit 'citrus-tests/target/citrus-reports/**/*.xml'
archiveArtifacts "citrus-tests/target/citrus-*/**/*"
}
}
Used by
citrusframework/citrus-docker-images PR #3
FROM consol/citrus:2.7.5
### sourced are copied from:
# https://github.com/openshift/jenkins/tree/master/slave-base
# https://github.com/openshift/jenkins/tree/master/slave-maven
# Copy the jenkins-slave entrypoint
ADD contrib/bin/* /usr/local/bin/
# Run the Jenkins JNLP client
ENTRYPOINT ["/usr/local/bin/run-jnlp-client"]

13. Build up a Build Pipeline!

14. OpenShift Build Pipelines: Build -> Test -> Run! Folie 14
Bakery App

15. OpenShift Build Pipelines: Build -> Test -> Run! Folie 15
Bakery App

16. OpenShift Build Pipelines: Build -> Test -> Run! Folie 16
Transition to OpenShift
Goals:
• Don’t change the Sourcecode of the Apps
• Also don’t change the Testcode
• Fully Automated Pipeline until Production
• All in OpenShift (CI, Build, Test, Deployment)
• Jenkins v1 -> v2

17. Stage I: CI-Build of the Artefact

18. OpenShift Build Pipelines: Build -> Test -> Run! Folie 18
CI Build ► Docker Image
build.pipeline.yml Jenkinsfile.dev
Maven Image
Dockerfiles
OpenShift
Regular Build
Docker Image
Artifact

19. DEMO
toschneck/openshift-example-bakery-ci-pipeline

20. Stage II: Automated Testing

21. Citrus Integration Testing

22. OpenShift Build Pipelines: Build -> Test -> Run! Folie 22
Mission ► Integration Tests
System under Test
System
Under Test
Verträge
Stammdaten
Dienstleister
Shipping
JM
S
XM
L
FTP
CSV
Webclient
Kunde
Agent / Techniker
Mobile, PDA
Http
REST
Http
SOAP

23. OpenShift Build Pipelines: Build -> Test -> Run! Folie 23
Mission ► Integration Tests
System
Under Test
Stammdaten
Dienstleister
JM
S
XM
L
FTP
CSV
Kunde
Agent / Techniker Http
REST
Http
SOAP
Automated API Interface Tests

24. OpenShift Build Pipelines: Build -> Test -> Run! Folie 24
Use Case: Testing of Middleware Integration / ESB / API
Citrus Features
• Ready to use components
− Client & Server Implementation
− Application Server Deployments
− Container Images
• Message transports
− Http REST, SOAP, JMS, FTP, TCP/IP, Mail, RMI, SSH, ...
• Data formats
− XML, Json, PlainText, CSV, XSD, WSDL
• Test DSL
− XML & Java
citrusframework/citrus

25. OpenShift Build Pipelines: Build -> Test -> Run! Folie 25
more Citrus
• Citrus Admin UI
• Easy Testsuite Management
• Reporting
• Citrus Simulator
• Complex Messaging Workflow Simulation
• Predefined Interface Packages: WBCI, WITA

26. OpenShift Build Pipelines: Build -> Test -> Run! Folie 26
QA ► Citrus API Integration Tests
Citrus Slave Image
build.pipeline.yml Jenkinsfile.qa

27. DEMO
toschneck/openshift-example-bakery-ci-pipeline

28. E2E Testing

29. OpenShift Build Pipelines: Build -> Test -> Run! Folie 29
Sakuli End-to-End Use Cases

30. OpenShift Build Pipelines: Build -> Test -> Run! Folie 30
Motivation
• Founded February 2014 Open Source (Apache)
• Goals:
− Combine a web and screen-based testing in ONE tool
− Use test results in CI- and monitoring systems
− Platform independent: (Linux/Windows/MacOS)
• Application tests from a End User Perspective
− Functionality (business critical path)
− Performance (e2e loading time)

31. OpenShift Build Pipelines: Build -> Test -> Run! Folie 31
Component 1: Sahi / Selenium
Web testing tool (sahi.co.in, seleniumhq.org)
method based DOM access:
_assertContainsText ("Logged in as: Sakuli", _div("user_field"));
_click(_span("Loaded Run Tabels"));
_assertExists(_table("cross_table_fixed"));
_assertExists(_cell(“testing allowed", _rightOf(_span("Name")), _under(_cell("Action")));
<table>
<tbody>
<tr>
<td>Alice</td>
<td>Bob</td>
</tr>
</tbody>
</table>
Webserverlocalhost:9999
{js}

32. OpenShift Build Pipelines: Build -> Test -> Run! Folie 32
Component 2: Sikuli
Visual automation tool (sikuli.org)
image identification, mouse & keyboard interaction:
screen.find("sap_ok").click();
screen.find("sap_ok").right(40).click().type("2223");
var bubble = new Region().waitForImage("bubble.png", 20);
bubble.dragAndDropTo(bubble.left(35)).highlight();

33. OpenShift Build Pipelines: Build -> Test -> Run! Folie 33
Benefits
• Integrated API for both Tools - one TestScope
• Step by Step Code (JavaScript, Java)
• Automatic Screenshots on any Errors
• Cloud-ready Docker Images for Kubernets/OpenShift
• Sakuli UI for Maintenance, Execution and Reporting (beta)
• "Forwarder" Modules:
− Monitoring (Gearman, Icinga, CheckMK)
− CI (Jenkins, Travis CI, Drone ...)
− Databases (MySQL/MariaDB, ...)

34. OpenShift Build Pipelines: Build -> Test -> Run! Folie 34
Monitoring Integration
• Ready-to-use setup in Kubernetes/OpenShift
• Combinable with „traditional“ monitoring checks
• Performance graphs
• Cron scheduled check execution
• Error screenshots
• Mail / Chat Notification
• Live watching possible

35. OpenShift Build Pipelines: Build -> Test -> Run! Folie 35
Sakuli End-2-End Testing Container
ConSol/docker-headless-vnc-container

36. OpenShift Build Pipelines: Build -> Test -> Run! Folie 36
QA ► Tested Docker Image
Docker Image
Artifcat
Citrus Slave Image SakuliImage
build.pipeline.yml Jenkinsfile.qa

37. DEMO
toschneck/openshift-example-bakery-ci-pipeline

38. OpenShift Build Pipelines: Build -> Test -> Run! Folie 38
Review
• Jenkins is not designed for containers or pods
Plugins, Startup, Java Agents, local Files, Development Roadmap
• Migration to Jenkinsfiles
• A bunch of wrapper scripts needed, to be flexible
• Rethink the CI workflow due to distributed approach
• Access to CI runtime data is not so easy any more
Logs, Screenshots, Artifacts
• Broad mix of technologies
• Jenkins slave image is intransparent -> do it yourself!
Base image: github.com/openshift/jenkins/slave-base
• Distributed documentation
OpenShift, Kubernetes, Jenkins, Jenkins-Plugins, Docker, …

39. OpenShift Build Pipelines: Build -> Test -> Run! Folie 39
Review
• Scaling is solved by the Cluster
• "oc CLI client" instead of OpenShift Jenkins Plugin
► Pipeline can be tested/developed step-by-step
► Independent from the plugin release lifecycle
• Parameterized YAML templates for OpenShift
• Infrastructur as Code
• New stages are easy to create
• Deployment will be tested by default
• Secrets are maintainable by cluster admins
• Builds can be controlled by CLI and UI
• Extendable trough Open Source approach

40. OpenShift Build Pipelines: Build -> Test -> Run! Folie 40
Alternative Approaches
hasura/gitkube
⇨ Gitkube
• “git push” based Workflow on remote Git
• Simple: combination of bash scripts
⇨ Skaffold:
• Simple scripted CI/CD Workflow
• 2 Stages: Local & Remote
⇨ JenkinsX
• Predefined workflow for CI/CD in Kubernetes
• NO new Jenkins -> just a Wrapper
• GitHub only (currently)
⇨ DroneCI / KubeCI
• Container first and easy understandable concept
• Local execution
• Layered build concept for Pods (in Development)
GoogleContainerTools/skaffold
kubeciiodrone/drone
jenkins-x/jx

41. KubeCI
● Drone Kubernetes Runtime
● Plugins:
○ kubectl
○ helm
○ img
(daemon-less build)
○ … more will follow
pipeline:
test:
image: python:2-slim
commands:
- python test/wait-for-it.py
build-docker:
image: plugins/docker
repo: toschneck/wait-for-it
dockerfile: Dockerfile
WIP

42. OpenShift Build Pipelines: Build -> Test -> Run! Folie 42
Links
toschneck/openshift-example-bakery-ci-pipeline
citrusframework/citrus www.citrusframework.org
ConSol/sakuli www.sakuli.org
@toschneck
@citrus_test
@sakuli_e2e

43. Questions?

44. We are hiring!
Hamburg, Berlin, Munich, San Francisco, Gdańsk, …
www.loodse.com
@Loodse

45. Thank you Munich!
Tobias Schneck, Loodse
@toschneck