Workshop process improvement Testing BI/DWH

1. 2014
Workshop process improvement
Testing BI/ DWH
AGENDA
Personal introduction & expectations 15m
Niels Bor and Marcus Drost (workshop leaders)
Participants workshop
Contents
Part 1 Short presentation: Waterfall versus Agile/ Scrum (Marcus Drost) 15m
Part 2 Problem awareness game (insight) test problems BI/ DWH systems (Niels Bor) 20m
Part 3 Short presentation: More about testing (Marcus Drost) 15m (then break 10m)
Part 4 Example from practice: Agile regression test tool DREAM (Marcus Drost) 30m
Part 5 Problem awareness game (from cause to action) (Niels Bor) 20m
Finish 10m
Highlights workshop (Niels Bor and Marcus Drost) as well as feedback participants (15m extra time)
V2.3

2. Introduction and expectations
Goal workshop
“Getting insight in the problems of waterfall and
agile test processes, and identifying solutions.
Especially for data intensive environments like
Business Intelligence, Data Warehousing and
Database Applications.”

3. Part 1
Short presentation:
Waterfall versus Agile/ Scrum
(Marcus Drost)

4. The Agile Manifesto introduced the term in 2001. Since
then, the Agile Movement, with all its values, principles,
methods and cultures, has significantly changed the
landscape of the modern software engineering.
The manifesto's values:
1.Individuals and interactions over Processes and tools
2.Working software over Comprehensive documentation
3.Customer collaboration over Contract negotiation
4.Responding to change over Following a plan
Source: Wikipedia

5. The Agile Manifesto is based on twelve principles:
1.Customer satisfaction by rapid delivery of useful software
2.Welcome changing requirements, even late in development
3.Working software is delivered frequently (weeks rather than months)
4.Working software is the principal measure of progress
5.Sustainable development, able to maintain a constant pace
6.Close, daily cooperation between business people and developers
7.Face-to-face conversation is the best form of communication (co-location)
8.Projects are built around motivated individuals, who should be trusted
9.Continuous attention to technical excellence and good design
10.Simplicity—the art of maximizing the amount of work not done—is essential
11.Self-organizing teams
12.Regular adaptation to changing circumstances
Source: Wikipedia

6. Iterative vs. Agile
First of all, Agile is iterative already but it is way more than just iterative. Here are a number of
differences between Agile and “just” Iterative development:
Mini-waterfall is still waterfall
Iterative is still waterfall, just on a smaller scale. A series of mini-waterfalls is certainly better
and less risky than one big waterfall but mini-waterfall still is fundamentally waterfall and
comes with all its known problems such as difficulty to adapt to change (“Nice idea but sorry,
the requirements have been signed off months ago”), cascading delays (“Oops, we need to
shorten the testing phase”) and low quality (“We don’t have time to fix those bugs. We’ll fix
them in a later phase/iteration”).
Source: Sandy Mamoli

7. Comparing Agile and Traditional approaches
Source: Scott Ambler

8. A look back at waterfall

9. Currently in AGILE/ SCRUM

10. Change frequency
LOW
HIGH

11. Regression test frequency
LOW
HIGH

12. Regression in AGILE/ SCRUME/
SCRUM

13. AGILE/ SCRUM test bottleneck

14. Frequent application changing

15. Part 2
Problem awareness game (insight)
based on a BI/DWH system
(Niels Bor)

16. Problem awareness game (insight)
The participants of the workshop discuss potential problems which they do encounter in
practice during testing of data intensive systems. They identify testing problems and describe
where and when the problems occur regularly. As auxiliary means, the participants could use
the diagram of the data warehouse system and the V-model on the next page.
Describe the problems in detail in order to come to know the root cause.

17. Source: Wikipedia

18. Typical problems (examples)
1. Data from production and test systems is functional not stable.
2. Test data delivered automatically or manually is not available in time.
3. Setting up test environments and preparation of tests is very time consuming.
4. Run time of tests is often very long due to bad performance.
5. Test cases are not clear and without connection to physical test cases; i.e. test
cases are made ad hoc during the test phase; test coverage is random
6. No description of output of test cases; high workload during the test phase
7. Often double work and overlap between UT, FAT and GAT.
8. Test cases are not re-usable; no connection between functionality (use cases)
and functional and physical test cases.
9. No standard process for adaption and extension of test cases; often ad hoc
10. The pressure on testers during test phases is very high.

19. Part 3
Short presentation: More about testing
and agile testing (Marcus Drost)

20. Testing levels (for information)
Tests are frequently grouped by the level of specificity of the test.
Unit Test (UT): A method by which individual units of source code is tested to determine
if they are fit for use.
Integration Test (IT): Integration testing is any type of software testing that seeks to
verify the interfaces between components against a software design.
System Test (ST): System testing, or end-to-end testing, tests a completely integrated
system to verify that it meets its requirements.
Regression Test (RT): Regression testing focuses on finding defects after a major code
change has occurred.
Functional Acceptance Test (FAT): Functional testing refers to activities that verify a
specific action or function of the code.
User Acceptance Test (UAT): Acceptance testing performed by the customer, often in
their lab environment on their own hardware.
Operational Acceptance Test (OAT): Also known as operational readiness testing, this
refers to the checking done to a system to ensure that processes and procedures are in
place to allow the system to be used and maintained.
Source: Wikipedia

21. Methods for regression testing
Random Sample: Controlling just a few results in order to generalize them
Advantage: Fast, and no automation necessary
Disadvantage: Inexact and incomplete
Proof Total: By totalizing some figures, the total result will be forecasted.
Advantage: Can be automated relatively fast
Disadvantage: Incomplete; root cause of differences are difficult to explain; strong
dependency on application logic
Data Model: As the output data is the fingerprint of applications' functionality, a data
model approach can be used to find every differences.
Advantage: complete coverage of 100%; independent of application logic; fast
analysis of root cause
Disadvantage: Test data management is necessary
Source: Marcus Drost

22. Spoken about regression...

23. More details…
U.K. financial regulators are investigating an IT failure at British bank RBS (The Royal Bank
of Scotland) that left millions of users without access to their accounts last summer. The
problem, which stemmed from an error during a software update, has already cost RBS £175
million ($270 million) in interest waivers, reversed charges and compensation payments,
InformationWeek reported. Additional penalties could be forthcoming based on the
determination made by the Financial Conduct Authority (FCA), a new regulatory agency in
charge of policing the U.K. financial services market.
The error last summer occurred when RBS, which owns British chains NatWest and Ulster
Bank, performed an update to the CA-7 software that controls the bank’s batch processing
systems. The program is used to automate large sequences of work involved in handling
daily transactions from ATMs, bank-to-bank payments and elsewhere.
Source: http://www.klocwork.com

24. Agile Development Team Testing Strategies
Agile development teams generally follow a whole team strategy where people with testing
skills are effectively embedded into the development team and the team is responsible for the
majority of the testing.
This strategy works well for the majority of situations but when your environment is more
complex you'll find that you also need an independent test team working in parallel to the
development and potentially performing end-of-lifecycle testing as well.
Regardless of the situation, agile development teams will adopt practices such as continuous
integration (CI) which enables them to do continuous regression testing, either with a testdriven development (TDD) or test-immediately after approach.
Source: Scott Ambler

25. TOP 5 actions for DWH test automation
Automatic regression test: Fast and frequent changing as well as code refactoring
requires continuous regression testing.
Simulation of source systems: Source systems do not deliver accurate test data;
delivery is often not in time; the generation of synthetic test data makes the DWH
software development independent of source systems.
Automatic output control: Automatic output control allows to control the output
directly after the test run; thereby the duration of test cycles will be reduced
Process automation: In practice many processes like scheduling are not
automated in test and development environments. This will become expensive in
the long run as processes in agile development occur more frequently.
Deployment automation: Deployment processes in test and production
environments take much time. As they are very frequent in agile processes, the lack
of automation can lead to a slowdown of processes.
Source: Marcus Drost

28. Part 4
Example from practice
Agile Test Tool DREAM
(Marcus Drost)

29. DEMO TEST TOOL DREAM (www.testtooldream.com)
World’s first tool for continuous database and data warehouse regression testing.
Shouldn’t we be doing better? (Scott W. Ambler)Mission-critical business functionality is
implemented in RDBMSs. In the survey, 63.7% of respondents indicated that their
organizations did this, but of those only 46% had regression tests in place to validate the
logic. Shouldn’t we be doing better? Author: Scott W.Ambler

32. Part 5
Problem awareness game
Agile Test Tool DREAM
(Marcus Drost)

33. Problem awareness game (from problem to action)
The participants of the workshop talk with each other about the problems which they have
found in the first part of the game. Now, the problems will be grouped together. Then, the
participants try to learn more about the problems by evaluating them in a dimensional
problem analysis. Use the net chart on the next page for the analysis. The result is a good
start in order to think about possible solutions and actions.
Try to find for every group of problems suitable actions!

34. Example dimensional problem analysis
Source: Esther Derby (agile retrospectives )

35. Possible solutions
1.Regression testing for all subsystems in a chain.
2.Simulation of test data and generation of syntethic test data.
3.Automation of deployment processes.
4.Improving performance by usage of faster machines in the cloud.
5.Define test cases for every user story and functional specification.
6.Define expected output in advance, and formally for automatic output control.
7.Arrange when to test what; try to shift all tests to the left in the SD process
8.Establish test data management by splitting test data to the level of user stories
9.Make test data management flexible, continuous and re-useable.
10. Automatic regression testing and output controll to prevent manual work.

36. Finish
Recapitulation of the workshop
Feedback participants