This document discusses testing infrastructure components that lie beneath applications. It defines infrastructure as the building blocks applications depend on, like hardware, virtualization, containers, and software. The author argues that testers should care about infrastructure testing because infrastructure problems are often found at the wrong level, and the product as a whole is symbiotic between applications and infrastructure. Various testing principles, examples, tactics, and tools are provided for both human/deterministic and machine/deterministic testing of infrastructure, as well as human/random and machine/random approaches.

1. “Look deep into nature, and
then you’ll understand
everything better” ~ Albert
Einstein

2. Ash Winter
Short stuff here:
@northern_tester
Long stuff here:

3. Testing Below the
Application

4. What do I mean by
infrastructure…
• Tons of
definitions
• Building blocks
which applications
depend upon:
–Hardware
–Virtualisation
–Containerisation
–Software

5. What lies beneath?
As an incomplete
list:
• Software
dependencies
• Load Balancers
• Webservers
• Network routes
• Weird and
wonderful

6. Barriers
• Impenetrable
ops teams
• Disinterested
dev teams
• DevOps is
sensible
• Pro-tip – pair
widely

7. Why should testers care?
• We find infra problems all the
time, at the wrong level.
• Symbiotic with application, the
product is a whole.
• Thinking at scale in a
distributed world.

8. Tool assisted world …
think technical awareness
• Test vs Prod
• Redhat -> rpm
• CentOS -> yum
• Separate
playbooks for
both
• Central Chef
server manages
nodes
• Consistent
• But propagates
problems
widely

9. Demo – getting started

10. Testing in an
infrastructure context…
• Insight to
those who
matter
• Complete
testing is
impossible
• Performance

11. Testability

12. Logging
• Tons of
info
• All
useful?
• syslog
• E.L.K.

13. Visualise…
• Add Extras
dimensions:
• Time
• CPU
• Memory
• Scheduled Jobs
• Events
• There are
millions
• Choose wisely

14. Demo – under the cloud hood

15. Control
• To test
effectively you
need control
• To support you
also need control
• Be an advocate
• Try:
• Headers like X-
ID and X-Log-
Level
• A/B Testing of
Infrastructure

16. Balanced Approach

17. HT to Dan North

18. Testing Corners

19. Human/Deterministic
Principles
• Logic and/or rules, according to some
oracles(s)
• Small as possible tests, the longer it
gets the less deterministic it is
Example
• A set of scheduled jobs to move log
files to a centralised log server.
• Subsequent archiving on a temporal
basis.
• Achieved via the Linux Cron.
Tactics
• State Transition
• Vary Critical Factpors
• Follow the Data
Tools
• Crontab Tester
• http://cron.schlitt.info/
• Cronitor
• https://cronitor.io/

20. Machine/Deterministic
Principles
• Large scale, across many instances
• Testing a single source of a
repeatable process and its
consistency
Example
• A team wish to create a set of new
webservers on existing physical
hardware in a repeatable way.
• Dependencies for the target
application are well known.
Tactics
• CRUMBS by Albert Gareev
• http://automation-
beyond.com/2012/07/14/follow-the-
crumbs/
Tools
• ServerSpec
• http://serverspec.org/
• ChefSpec
• https://docs.chef.io/chefspec.html

21. Demo – creating and running
ServerSpec tests

22. Human/Random
Principles
• Consider a blend of thresholds which
have relationships with other system
factors.
• Such as network configuration, rarely
tested, even more rarely automated.
Example
• New hardware added to pools of
servers accessed by a load balancer.
• Subject to thresholds such as:
• Round Robin
• Best resource availability
Tactics
• Boundary value proving
• Background noise
• Replay traffic
Tools
• (Good Old) cURL
• https://curl.haxx.se/
• Postman/Charles/Fiddler
• https://www.getpostman.com/
• https://www.charlesproxy.com,/
• http://www.telerik.com/fiddler

23. Machine/Random
Principles
• Most neglected
• Machine augmented pseudo random
approach to bust biases.
• Target breadth, uncover problems
early.
Example
• Set of NodeJS API instances on n old
version of Node with a security flaw.
Tactics
• Application routes analysis
• Concurrency
• Positive and negative flooding
• Starvation
Tools
• The Chaos Monkey Army
• https://github.com/Netflix/SimianAr
my.git
• Siege
• https://www.joedog.org/siege-home/

24. Demo – adding randomness

25. Since the webinar…
• Contributed to building a
containerised dev environment
• Consulted on a new Kubernetes
platform
• New SAN (Named Project Big Bang)

26. Questions?