Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2tbpYqX. Stefanos Zachariadis talks about some techniques for bringing our testing to production. He shows how to sanity check a live system using end to end testing, limiting interference with real user interactions and outputs. Finally he suggests ways to observe and integrate real production data into a continuous delivery pipeline and assert on the validity of the production system's outputs. Filmed at qconlondon.com. Stefanos Zachariadis is an independent developer at motocode ltd and the coder behind CycleMaps. His career has taken various twists and turns: from academia to writing satellite software for the European Space Agency; test automation for various banks, leading up to writing low latency code that can process 50,000 orders a second on a single CPU core as a team lead for LMAX Exchange.

1. A er Acceptance: Reasoning About System Outputs
Dr. Stefanos Zachariadis
@thenewstef
https://moto.co.de
https://cyclema.ps
http://itv.com

2. InfoQ.com: News & Community Site
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
testing-production
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

4. OUTLINE
Issues not typically caught by a CI environment
What we'd like to test a er the acceptance testing
phase
How we can achieve this systematically

5. WHY ARE WE HERE

9. actual photo of ATM in Uruguay taken Feb 2017

10. OH LOOK, TOUCHSCREEN STILL ACTIVE

12. enum AccountType {
PERSONAL, BUSINESS, PRIVATE
}
public class Account {
private final String name;
private final AccountType type;
}

13. enum AccountType {
PERSONAL, BUSINESS
}

15. public class Account {
@NotNull
private final String name;
private final AccountType type;
public boolean equals(Object obj) {
return name.equals(obj.name) &&
type.equals(obj.type);
}
}

18. public class Account {
private final String name;
private final AccountType type;
private final CurrencyCode currency;
}

22. @Test
public void getsTheCorrectBalanceAfterADeposit() {
//given
Account account = new Account();
//when
account.deposit(200);
//then
assertEquals(200, account.getBalance());
}

23. so ware is complicated
class StarWarsMovies {
private boolean[] seen = new boolean[200];
}

24. 2^200 + 1
seen = null;

25. 1 606 938 044 258
990 275 541 962 092
341 162 602 522 202
993 782 792 835 301
376 + 1

26. testing cannot be exhaustive

27. continuous delivery of stateful systems is hard

28. 2^200

29. Check in
Check in
Check in
Trigger
Trigger
Trigger
Trigger
Trigger
Feedback
Feedback
Feedback
Feedback
Feedback
Feedback
Approval
Approval
Delivery team Version control Build & unit
tests
Automated
acceptance tests
User acceptance
tests
Release

31. @Test
public void getsTheCorrectBalanceAfterADeposit() {
//given
Account account = new Account();
//when
account.deposit(200);
//then
assertEquals(200, account.getBalance());
}

32. Your current account balance is a result of:
Deposits and withdrawals
Charges and deposits by other actors
Data migrations
Exchange rates
Multiple system releases
over lots of time

33. WHY IT MAY BREAK
system state
over multiple releases

34. Check in
Check in
Check in
Trigger
Trigger
Trigger
Trigger
Trigger
Feedback
Feedback
Feedback
Feedback
Feedback
Feedback
Approval
Approval
Delivery team Version control Build & unit
tests
Automated
acceptance tests
User acceptance
tests
Release

36. A NEW HOPE

37. WHAT TO TEST
Data validity

38. All data can be loaded
@Test
public void allAccountsAreReadable() {
final AccountDao accountDao = new AccountDao();
for(User user : users) {
final Account account
= accountDao.loadAccountFor(user);
verify(account);
}
}

39. Business level validation
@Test
public void accountsBalanceAboveOverdraftLimit() {
for(User user : users) {
Account account =
accountDao.loadAccountFor(user);
assertTrue(account.getBalance()
> NEGATIVE_OVERDRAFT_LIMIT);
}
}

40. WHAT TO TEST
Data invariance
1. capture invariants
2. upgrade
3. ???
4. verify invariants (& profit)

41. @Test
public void accountBalanceIsMaintained() {
for(User user : users) {
Account account =
accountDao.loadAccountFor(user);
BigDecimal productionBalance =
prodData.getBalance(account.getId());
assertEqual(productionBalance,
account.getBalance());
}
}

42. WHAT TO TEST
Migration integrity
@Test
public void allAccountsAreInUSD() {
for(User user : users) {
Account account =
accountDao.loadAccountFor(user);
assertEqual(CurrencyCode.USD,
account.getCurrency())
}
}

43. WHAT TO TEST
Data volume
@Test
public void generatesARiskReport() {
final AccountDao accountDao = new AccountDao();
final RiskReportGenerator riskReportGenerator =
new RiskReportGenerator(accountDao);
time(riskReportGenerator::make, 60, TimeUnit.MINUTES)
}

47. DATA SANITISATION
first_name last_name account_id
John Smith 7
↓
first_name last_name account_id
Name 3 Surname 2 7

49. data migration
Production data → latest commit ?

50. Data is owned by the application. This means that the
migration process is owned by the application, and
new migrations should ship with the application and be
performed with the deployment of every new version.
latestData = migration(prodData)

51. 1. import the sanitised data that the cleanser
produced
2. capture invariants (e.g. account balances)
3. migrate it to the latest version of the application
4. run the tests

53. CONSIDERATIONS
fight the power
no downtime
flip it around - send the tests to production
frequency

54. MORE THAN JUST TESTING
staging
passwords

55. OH IT CAN BE SO SLOW
Sampling
Incremental updates
Rolling back

56. CONCLUSIONS
data testing: integration testing of commit with prod
- like data
bring production - like data into your CD pipeline
use a cleanser to make this legal
migration is integral to the application
allows you to catch a whole new category of bugs
facilitates frequent releases

58. Thank you!
pictures by @thenewstef apart from CD (wikipedia),
Data, Jar Jar, bank of chthulhu (unattributed)
https://moto.co.de
https://cyclema.ps
http://itv.com

59. Watch the video with slide synchronization on
InfoQ.com!
https://www.infoq.com/presentations/testing-
production