EXTENT-2016: Software Testing & Trading Technology Trends 22 June, 2016, 10 Paternoster Square, London Key Challenges and Lessons Learned from Testing a New Trading System Dr. Matthias Burghardt, Head of Business Development, Boerse Stuttgart Would like to know more? Visit our website: extentconf.com Follow us: https://www.linkedin.com/company/exactpro-systems-llc?trk=biz-companies-cym https://twitter.com/exactpro #extent2016 #exactpro

1. Key Challenges and
Lessons Learned from
Testing a New Trading System
The Xitaro Project
Dr Matthias Burghardt
Head of Business Development
Boerse Stuttgart
London, 22 June 2016

2. • About Boerse Stuttgart
• Overview of the Xitaro Project
• Ten Lessons Learned
2
Agenda
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

3. 3
Where exactly is Stuttgart?
About Boerse Stuttgart
We are here!
Stuttgart
is the capital and largest city
of the state of
Baden-Württemberg
in southwest
Germany.
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

4. 4
What is Stuttgart known for?
About Boerse Stuttgart
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

5. 5
Boerse Stuttgart is Europe‘s leading market for securitised derivatives
About Boerse Stuttgart
Source: Respective Stock Exchange. Figures based on volume of securitised derivative orders. Last updated: April 2016.
33,6%
9,7%
16,2%
9,4%
23,7%
1,8% 1,8%
0,4%
Stuttgart incl. Nordic Derivatives Exchange
Börse Frankfurt Zertifikate AG
SIX Swiss Exchange
Euronext
Borsa Italiana
BME
NASDAQ Nordic & Baltic
Others
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

6. 1,500,000
500,000
1,000,000
0
2,382 20,305
1995
1,388,600
2015
1,292,471
20102005
535,794
70,010
2000
6
Currently more than 1.4m instruments are listed at Boerse Stuttgart
This number is driven by the large amount of securitised derivatives
About Boerse Stuttgart
# of instruments of other asset
classes listed at Boerse Stuttgart
Debt instruments: 12,186
Equities: 7,973
Investment
fund units:
3,410
Number of securitised derivatives
listed at Boerse Stuttgart
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

7. 7
Boerse Stuttgart ranks among Europe’s Top 10 Stock Exchanges in terms of
turnover
About Boerse Stuttgart
Source: Federation of European Securities Exchanges, London Stock Exchange, Borsa Italiana Basis: Order book turnover for whole of
2015 in all asset classes, electronic order book orderbook trades
Rank Stock Exchange
Turnover (2015)
in billion €*
1 London Stock Exchange Group 2,871.5
2 Euronext 2,017.0
3 Deutsche Börse 1,628.7
4 BME (Spanish Exchanges) 1,020.1
5 SIX Swiss Exchange 999.1
6 NASDAQ Nordic & Baltic 711.1
7 Borsa Istanbul 430.1
8 Oslo Børs 135.8
9 Boerse Stuttgart Group 106.6
10 Warsaw Stock Exchange 45.6
11 CEESEG - Vienna 29.6
12 Irish Stock Exchange 19.3
13 Athens Stock Exchange 17.1
14 CEESEG – Budapest 7.2
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

8. • About Boerse Stuttgart
• Overview of the Xitaro Project
• Ten Lessons Learned
8
Agenda
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

9. 9
The Xitaro Project – replacing Boerse Stuttgart’s exchange system in a
complex environment to gain flexibility and independence
About the Xitaro Project – Mission
Xitaro
Boerse Stuttgart‘s
new trading
system
Powered by
Nasdaq
Genium INET
XONTRO
• Almost 25 years old
• Shared host based
system
• High degree of
dependency
Flexible and independent
Competitive costs
State-of-the-Art
Faster innovations
Future proof
Project
Xitaro
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

10. 10
Five dedicated work streams took the challenge
About the Xitaro Project – Project Organisation
Joint work stream with Nasdaq BSG work stream
(Joint) Steering CommitteeRisk-Controlling
• Management of all
stakeholder
activities, e.g.
trading participants,
employees, boards,
etc.
• Specification of all
Xitaro functionalities
• interfaces, internal
systems, network
setup
• Solving technical
problems and issues
• Preparation for the
operational phase
• setup of processes,
resources,
infrastructure
• Cutover Preparation
• Member onboarding
• Ensure software and
system quality by
intensive testing
(automated and
manual)
IT Security & Data protection PMO & Quality Assurance
Internal Audit
Project Management
Legal Advisory
Stakeholder
Management
Functional
Design
Techn. Design &
Customizing
Setup Operations
& Cutover
Management
Test and
Acceptance
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

11. 4 %
6 %
33 %
36 %
21 %
11
Accounting for one third of the total effort, Test & Acceptance was Xitaro‘s
largest work stream
About the Xitaro Project – Test Effort
Joint work stream with Nasdaq BSG work stream
Stakeholder
Management
Functional
Design
Techn. Design &
Customizing
Setup Operations
& Cutover
Management
Test and
Acceptance
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

12. 1: System Integration Test
After a 2 years implementation project, Xitaro went live in September 2015
About the Xitaro Project – Time Line
Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2
SIT1
Detailed Design
Implementation
Q3
Member Tests
Operations
Beta Tests
Project Xitaro
2013 2014 2015
Dress
Rehearsals
Hardware / Network Setup for
SIT and UAT
Hardware / Network Setup
For Production
Member Onboarding
Member
Certification
Cutover Concept
Test
Preparation
Regression
Tests
User Acceptance
Tests
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System 12

13. • About Boerse Stuttgart
• Overview of the Xitaro Project
• Ten Lessons Learned
13
Agenda
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

14. 14
Key challenges in testing cover a wide range of areas
Key Challenges
Test automation
Outsourcing
Regression Testing
Test Case Design
Beta Testing
Early Member Testing
Test environments
Defect Management
Non-functional Testing
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
Acceptance Test Preparation

15. Testing / Contingency
15
You cannot separate functional design and test case design
Lesson #1 – Test Case Design
Functional
Descriptions #1
Functional
Descriptions # 2
Functional
Descriptions #3
Functional
Descriptions #4
Test Case Design
Test Case Design
#1
Test Case Design
#2
Test Case Design
#3
Test Case Design
#4
Functional
Descriptions
Testing
Testing
Test case design
requires close alignment
between functional and
test experts
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

16. Day 3 Marc Tom Andi
Env 1 TC 1 TC 3
Env 2 TC 2 TC 5
Env 3 TC 7 TC 6
…
Day 2 Marc Tom Andi
Env 1 TC 1 TC 3
Env 2 TC 2 TC 5
Env 3 TC 7 TC 6
…
Day 2
16
Take one step at a time – from acceptance test specifications to the detailed
acceptance test plan
Lesson #2 – Acceptance Test Preparation
Acceptance Test
Specifications
Acceptance Test
Procedures
Acceptance Test
Plan
Acceptance Test
Plan Detailed
Acceptance Test
Draft Review Update Approve
Day 1 Marc Tom Andi
Env 1 TC 1 TC 3
Env 2 TC 2 TC 5
Env 3 TC 7 TC 6
…
ATS:
•TC1:
•TC2:
•TC3:
•TC4
…
ATP:
•TC1
• Step 1
• Step 2
• Step 3
• …
AT Plan:
•Day 1
• TC 1
• TC 2
• TC 3
• …
ATP:
•TC2
• Step 1
• Step 2
• Step 3
• …
ATP:
•TC3
• Step 1
• Step 2
• Step 3
• …
ATP:
•TC3
• Step 1
• Step 2
• Step 3
• …
Review Cycle helps aligning customer and supplier expectations
1 2 3 4
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

17. 17
Choose wisely which functionality outsiders could test
Outsourcing may prove difficult when testing complex functionality
Lesson #3 - Outsourcing
36% 64%
43% 57%
43% 57%
44% 56%
46% 54%
52% 48%
70% 30%
91% 9%
100%
100%
100%
Functionality 1 100%
93%7%Functionality 15
17% 83%
22% 78%
Functional Experts Test Experts
decreasing
complexity
Test cases executed by Functionalities with higher
complexity e.g. Trading Halts
Functionalities with lower complexity
e.g. GUI User Configuration
Overall percentage of executed test cases
53%47%
Test Experts
Functional Experts
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

18. 18
Early beta deliveries are good but you need to test them properly
Lesson #4 – Beta Testing
1: User Acceptance Test
Q1 2014 Q2 2014 Q3 2014 Q4 2014
Partly included functionalities
Completely included functionalities
21.02. 25.03. 09.05. 16.06. 08.09. 13.10.
Beta
1.1
Beta
1.2
Beta
1.3
Beta
2.1
Beta
2.2
Beta
2.3
UAT1
5
2 12
2
14
8
7
27
1
61
76
Test
Phases
Number of
functionalities
delivered
Six Beta Releases were delivered during the project covering 76 functions of varying complexity
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
What we should have done
Beta test effort
What we actually did

19. • Creating automated test cases is more
time-consuming than manual test cases
• Not easy to combine several interfaces and
reports into one test scenario
• Test scripts can have defects and require
debugging
• Lack of flexibility when functional
requirements change
• Quick and efficient test case execution
• Good for interface tests
• Allows for cost-effective frequent test
execution during regression tests
• More interesting for testers
19
Automating tests frees up valuable experts – if you do it right
Lesson #5 – Test automation
Used to regression
test 8 Releases
before Go-Live
+ -
Test automation during the project – the pros & cons
+
+
+
-
-
-Project Xitaro test cases
Q1 2015 Q2 2015 Q3 2015
AT 1
1,000
4,000
AT 2
1,700
3,900
Regression Tests
1,800
2,500
Automated Test Cases
Manual Test Cases
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

20. 20
Create different regression test sets for different purposes
Lesson #6 – Regression Testing
1 Evaluation
6,500
Test Cases
Functional
Frequency
Frequency of using
the functionality in
operations
Damage Potential
Potential damage /
impact in the case
of defects in the
functionality
Defect Frequency
General frequency
of defect
occurrence in this
functionality / test
case
2,200
Test Cases
4,300
Test Cases
Possibility for
Automation?
2 Selection
3 Categorisation
Small
regression test set
1,800
Medium
regression test set
2,300
Large
regression test set
4,300
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

21. Start early with member testing to avoid surprises
Lesson #7 – Early Member Testing
Q4 2014 Q1 2015 Q2 2015 Q3 2015
Early Member Tests
Dress
Rehearsals
Member Onboarding
Member
Readiness Checks
Simulation Period
• Pilot & Early Member connection
• Participant setup
• Contract management
• Free and assisted testing
• End-to-end Tests
• Non-functional tests
• Connection and interface tests
• Tests of participant configuration
• Debugging activities, both in Xitaro
and customer systems
• 4 Waves of assisted tests
to validate Go-Live
readiness of all connected
members
• 4 weekends of
Go-Live rehearsals
• One participation
mandatory
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System 21
Xitaro Member testing activities started about 12 months before Go-Live – early commitment was rewarded
Defect reports by
members are important
to help your test team
detect all errors
Structured dress
rehearsals avoid
unpleasant surprises
at Go-Live
Early member
connectivity will give you
certainty and remove
risks of delays

22. 22
Separating test environments by purpose allows for flexibility during the
project and afterwards
Lesson #8 – Test Environments
Non-Production Systems
Xitaro
Simulation
(production-like
hardware)
Xitaro
External Test
(virtualised
environment)
Xitaro
Internal Test
(virtualised
environment)
Xitaro
Configuration
(virtualised
environment)
• User Acceptance
Test
• Performance
testing and
operational
acceptance test
• Member testing
activities
• Automated tests
• Regression tests
• Planned to be
made available
to members
• Internal test
environment for
internal software
development and
integration tests
• Temporary test
environment for
user
configuration
before Go-Live
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System

23. 23
Plan your non-functional tests from end to end and include all relevant
business day activities
Lesson #9 – Non-Functional Tests
1 How we planned it
SLA
Non-Functional Area # Test Cases
Security 2
High availability and disaster recovery 79
Performance 34
Audit 3
Operations 20
Integration with external systems 9
2 How we should have planned it
• Analyse business day activities according to daily
cycle jointly with supplier
• Detect all business relevant performance parameters
• Plan joint activities for the System Integration Test
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System
3
What we did
anyway
Xitaro

24. Close defect management and tracking make sure the project survives the
User Acceptance Test
Lesson #10 – Defect Management
527
149
0
200
400
600
SITBeta 2.3
Resolved incidents
+378
AT1
480
208
0
100
200
300
400
500
Created incidents
+272
AT1SITBeta 2.3
0
20
40
60
80
100
120
140
160
180
200
220
240
1.2. 1.3.
Trend reported vs. resolved incidents
~13.5 incidents
resolved per day
~9.7 incidents
created per day
Reported incidentsResolved incidents Trend reported vs. resolved incidents
15.1.
AT2
release
Average net reduction
of ~4 incidents per
day during AT1
~40–50 open
incidents not yet
retested
End AT1
Average reduction
of ~8 incidents
estimated after AT1
Resolved but
not retested
yet
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System 24

25. Questions & Answers
© Börse Stuttgart, 22 June 2016 Key Challenges and Lessons Learned from Testing a New Trading System 25

26. Thank you for your
attention!
Dr. Matthias Burghardt
Head of Business Development
Tel.: +49 711 222 985-593
Fax.: +49 711 222 985-91-593
Mail: matthias.burghardt@boerse-stuttgart.de
Börse Stuttgart
Börsenstraße 4
D-70174 Stuttgart