6. Process of executing automated tests as part
of the software delivery pipeline to obtain
immediate feedback on the business risks
associated with a software release candidate.
Continuous Testing
6
9. Sample project from my experience
On-premises server version of a product
shipped as virtual image (OVA or AMI)
● Server with CLI
● Admin web panel
● API
● Clients: web, mac, win, linux, ios, android
9
12. Our example pipeline
● Merge PR into component repo
● Trigger component build
● Run component unit tests and build .deb file
● Trigger system build
● Build product and run “test_build” tests
● Trigger all CI plans (acceptance, CLI, perf, etc)
● Press “Deploy” button
12
13. Version control
● All tests code under version control (Git, svn, etc)
○ No excuse not to use
● Distributed version control (GitHub, Bitbucket)
● Same PRs flow
○ Don’t hesitate to add devs as reviewers
● Store tests with production code (Unit, Integration)
○ Easy to get older version of the software with related tests
● Separate repo for Acceptance test, Perf, etc
13
14. Testing plans
● Basic regression (selenium, nose, python)
● CLI (bash, selenium)
● AD/LDAP (another flow of users setup)
● Integrations (special pre conditions and selenium)
● Web client (nodejs, karma)
● Perf (locust, new relic)
● Export -> Import
● Upgrade plans (RC, dev -1, with reboot, etc)
● Component integration tests
● ...
14
15. Basic rules
● Test plans owned by QA team
● Fight tests flakiness
● Plans should be green
● Test team checks all “reds”
● Jira tickets linked to every valid test fail
● Dev team checks labeled CI tickets
● Skip test if needed (@skip(“JIRA-ID”))
● Execution time for plan < 1 hour (*)
15
20. Do not use PhantomJS (*)
● not a real browser
● additional setup
● limitations (e.g. incorrect alerts handing)
● “own” bugs (e.g. drag&drop)
● not stable on “big” runs (*IMHO)
● Xvfb + FF or Chrome are faster
* all above valid for version 1.x
Lesson learned: Run Real Browsers
Headless.
20
22. Tip: two browsers on one screen
1. Pass parameter num_of_browser_instances
2. Calculate driver screen resolution:
screen_width() / num_of_browser_instances
1. Launch browser at desired position
driver.set_window_position(width_coordinate, 0)
22
23. ●Long running test environment
vs
●Launch new fresh env on each CI run
Test environment
23
24. AWS
EC2 AMIs For The Win:
● production like environment
● fast deployment
● “super” speed between S3 buckets
● secure isolated environment (VPC)
● pay as you go
24
25. ● AWS CLI for your CI
● Route53 for DNS
● AWS Lambda - serverless computing
● Additional servers for infrastructure stuff like
AD, mail or proxy server
● Different instance types
● CI agents in EC2
● and many more...
and more AWS
25
26. Typical test development env problems
● difficult to set up and maintain
● hard to track dependencies
- python versions
- browser versions
- selenium and other packages versions
- other tools and software
● managing several environments
● OS differences between CI and local env
● inconsistency of executions
26
27. Docker
Docker containers wrap a piece of software in a
complete filesystem that contains everything
needed to run: code, runtime, system tools,
system libraries – anything that can be installed
on a server. This guarantees that the software
will always run the same, regardless of its
environment.
27
30. Docker
Build container:
$ docker build -t firefox:42 .
Push to registry:
$ docker push your_registry/firefox:42
Pull from registry:
$ docker pull your_registry/firefox:42
Run container:
$ docker run -t -i -v “/app:/app” firefox:42
30
31. ● Docker is not a virtual machine solution
● thousands images in docker hub
● own private docker registry
● run the same container on linux, mac, win
● lightweight to run
● lightweight to store
● lightweight to change
Docker
31
33. Practical tip: CI your images
1. Put Dockerfiles under version control
2. Create CI plans to build and push images
3. Trigger plans on repo commit
4. Profit
33
