QA Fest 2017. Влада Бенюх.Организация End-to-End тестирования в cross-integration команде

Киев 2017
Первый в Украине фестиваль тестирования
E2E testing: Challenges and
recommendation in test process
organization for cross-integration teams
Vlada Benyukh

Киев 2017
Первый в Украине фестиваль тестирования
Speaker info
Vlada Benyukh, Ukraine
QA Manager/QA Consultant at Ciklum, TCoE
•7 years in QA
•Manual QA Engineer, QA Manager, mentor
•Certified: SAFe, ICAgile, ISTQB
in/vlada-benyukh

Киев 2017
Agenda
1. E2E testing in cross-integration teams, What is it?
2. Test process and approach
3. Infrastructure management
4. Collaboration
5. Test team structure
6. Automation setup
7. Questions
E2E testing: Challenges and recommendation in test process organization for cross integration teams

Киев 2017
What is E2E testing?
Main goal is to identify system dependencies and to ensure
that the data integrity is maintained between various
applications/components correctly

Киев 2017
What is E2E testing in cross-
integration teams?
Expectation

Киев 2017
What is E2E testing in cross-
integration teams?
Process and approach
Infrastructure management
Communication and
collaboration
Most of the challenges in a E2E team relates
to cross-integration work:

Киев 2017
Test Process challenges

Киев 2017
Test Process challenges – Team
structure

Киев 2017
Why team structure was a
challenge?
 3 QA Automation
Specialists
 1 Automation QA Lead

Киев 2017
1. Understanding of the systems flows
2. Accesses to the systems
3. Requirements for end-to-end flows
4. Test Design for end-to-end flows
5. Test data for each system
6. Test environment readiness
challenge?

Киев 2017
Do not change team structure until challenges resolved…
challenge?

Киев 2017
Test Process challenges – Solution
1. Systems:
 7 separate systems which integrated into 1 CRM
 1 common aggregation level for all systems
 1 common integration level for all systems
2. Many development and test teams (in different
countries and time zones)

Киев 2017
Gather knowledge about the systems!
 Get access to the systems
 Understand systems specific and
functionality
 Get information about architecture
overview and technologies used
 Figure out scope and readiness dates
of the systems

Киев 2017
Gather knowledge about key representatives in
each system!
 System teams
 Integration development system teams

Киев 2017
Ask key representatives about
end-to-end flows
Gather, document and approve
acceptance criteria
Prioritize acceptance criteria
Requirements challenges – What
was the plan?

Киев 2017
No understanding and goal of
end-to-end team
No requirements for end-to-end
flows
No responsible person for end-
to-end team
Requirements challenges – What
was in reality?

Киев 2017
Requirements challenges
How to get requirements …

Киев 2017
Get system requirements
Explain the goal of end-to-end
testing
Review system requirements
Document, prioritize and
approve with business side
Requirements challenges – Solution

Киев 2017
Requirements challenges – Summary
1. General knowledge about system were gathered
2. Scope and priorities for end-to-end were identified
3. Detailed Requirements for end-to-end flows were
documented

Киев 2017
Test Design challenges

Киев 2017
• No technical skills to create test
documentation based on the best
practices
• No input test data for each
system
Test Design challenges - Problem

Киев 2017
Test Design challenges – Solution
• Explain the necessity of manual QA
specialists to senior management
• Involve manual QA Engineers from
system teams in test cases creation
• Motivate automation team to
collaborate with manual QA
Engineers

Киев 2017
How motivate automation team to create test
cases?

Киев 2017
 Explain the temporarily situation and
what can happen without end-to-end
team involvement
 Be involved and continue to work on
automation direction together

Киев 2017
 Ask involved Manual QA Engineers to
be responsible for test cases creation
 Define test case template which will
be used by manual and automation
QAs
 Structure test cases by functionality
and priorities

Киев 2017
Test Design challenges – Summary
1. Responsible team members for test cases creation
were defined
2. Structure and template were identified
3. Flows for end-to-end were covered by test cases

Киев 2017
Testing approach agreement across
teams
Agreements should be aligned before starting test
execution activities

Киев 2017
Testing approach agreement across
teams
Discuss process/approach with
teams
Involve all stakeholders in
process/approaches discussions
Align procedures across teams
Document all agreements

Киев 2017
Testing approach – Testing Order
Smoke
Acceptance critical flows
Regression
UAT

Киев 2017
Testing approach – Testing Order
Type of Test Scope Outcome
Smoke One End-to-end flow
for most critical
functionality
Blocker bugs
Acceptance critical
flows
End-to-end flows P1/P2/P3/P4
bugs
Regression P1/P2 P1/P2 test cases P1/P2 bugs
Regression P3/P4 P3/P4 test cases P3/P4 bugs
UAT User Acceptance
Testing
P1/P2 and
P3 bugs

Киев 2017
Testing approach – Defect
management
RAISE ON E2E
BOARD
DISCUSS ON
BUG TRIAGE
DUPLICATE
AND ASSIGN
TO A PROPER
TEAM
ANALYSE CASE
WHEN DEFECT
HAPPENED

Киев 2017
Test Execution challenges - Solution
• Ask Manual QA representatives focus their
attention on execution activities from manual
point of view
• Involve Automation QA representatives in
technical details analysis to find a workaround
for infrastructure challenges
!!! Infrastructure challenges will be covered on the next section!!!

Киев 2017
Infrastructure challenges

Киев 2017
Test environment requirements for
verification of end-to-end flows:
 All items should be processed from
start system to end system
automatically

Киев 2017
But what we had….

Киев 2017
Based on the system configurations, limitations and application
specific, items are integrated from start system to end
application into two different ways:
 Semi-automatically
 Non-automatically
Infrastructure challenges - Problem

Киев 2017
Automation QA team was involved to resolve next items:
 Data formats and mappings for items processing
 Access to file transfer folders which can help to
make items processed to end system
 Get permissions to monitor messages to diagnose
the problem
Infrastructure challenges - Solution

Киев 2017
Infrastructure challenges – semi-
automatic procedure
Creation
Status
Verification
Verification
in CRM
Automation QAsManual QAs
• Create items in start
system
• Verify that details
about created items
are stored properly
• Track the processing
status and monitor
issues
• Verify that item details are displayed in end system

Киев 2017
Infrastructure challenges – non-
automated procedure
Creation
Status
Verification
Verification
in CRM
• Create items in start
system
• Verify that details about
created items are stored
properly
• Generate data formats
which required for
processing
• Verify that item details are displayed in end system
Manual
Processing
• Process and track the
processing status and
monitor issues
Automation QAsManual QAs

Киев 2017
Infrastructure challenges –
Summary
1. End-to-end flows were executed successfully
2. End-to-end scenarios found bugs and discrepancies
in the systems functionality

Киев 2017
Collaboration challenges

Киев 2017
Collaboration challenges – Cross
teams
What is it if NO:
1. Frequency and scope for updates
2. Release readiness dates
3. Changes in the systems
4. Possible impact and risks for end-to-end
flows
1. Unexpected issues occur while testing
2. Possibly unexpected configuration issues
3. Dependencies are not tracked and mitigated
in time

Киев 2017
How to avoid issues?
Collaboration challenges – Cross
teams

Киев 2017
Regular cross-teams Sync Ups
Participants:
 Project management teams
representatives
 Integration Program Manager
 Team Tech Leads
 E2E QA Lead
Goals:
 Define scope for updates
 Get system updates dates
 Analyze and mitigate risks that might
affect E2E testing before testing starts
 Escalate critical problems
 Discuss other common problems and
questions

Киев 2017
Collaboration challenges – cross-
teams progress understanding
What is it if NO:
1. Scope and activities for end-to-end testing
2. Understanding of the progress end-to-end
testing
3. Risks and blockers for end-to-end testing
4. Release readiness dates

Киев 2017
How to avoid issues?
Collaboration challenges – cross-
teams progress understanding

Киев 2017
Regular E2E team Sync Up
Participants:
 Integration Program Manager
 E2E QA Lead
 E2E key QA specialists
Goals:
 Show the progress
 Escalate critical problems
 Discuss other common problems and
questions

Киев 2017
Example of Daily Progress Visibility

Киев 2017E2E testing: Challenges and recommendation in test process organization for cross integration teams
Example of Daily Progress Visibility

Киев 2017
Collaboration challenges – Summary
1. Regular updates on system updates and readiness were
completely clear
2. Risks were identified and mitigated regularly
3. Critical problems were escalated in time
4. Test progress and project health were synchronized
between teams

Киев 2017E2E testing: Challenges and recommendation in test process organization for cross integration teams
Test Team structure

Киев 2017
QA Lead
- QA Process and
Approaches
implementation
- Cross team ongoing
QA work planning and
monitoring
- Regular QA reporting
- People management
between teams
- Support in building
and aligning QA
automation approach
Manual QAs
-Main knowledge
keepers in the systems
- Responsible for test
cases preparation and
execution semi-
automated procedure
- Escalate issues
- Define dependencies
and quality risks. Inform
about them in time
Automation QAs
- Support in test cases
creation
- Responsible for
incomplete test
procedure processing
testing
- Responsible for test
automation
implementation
- Escalate issues
- Define dependencies
and quality risks.Inform
about them in time
Test Team structure - Solution

Киев 2017
QA Lead
QA Manual QA AutomationQA Manager
QA Manual Lead
QA Manual Lead
Middle QA
Middle QA
Senior QA
Middle QA
Middle QA
Senior QA Senior QA
Automation team
Test Team structure - Solution
Team 1 Team 2

Киев 2017
Automation challenges

Киев 2017
Automation challenges – What was
it?
2. Complex and obsolete architecture
1. End-to-end flow is not fully automated
3. Many external and internal dependencies

Киев 2017
Automation challenges – What was
solution?
1. Get access to systems via API to create items
2. Generate required data formats and make to be processed
it automatically
3. Automate validation of items in common CRM

Киев 2017
Automation challenges – Order to
automate
Smoke Regression
Acceptance
flows

Киев 2017
Recommendations
1. Do not change team structure until all
important challenges were not clarified
2. Be proactive and look for the solutions
3. Do not try to resolve all issues by yourself
4. Collaborate with stakeholders and systems
team members

Киев 2017
Questions

QA Fest 2017. Влада Бенюх.Организация End-to-End тестирования в cross-integration команде

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to QA Fest 2017. Влада Бенюх.Организация End-to-End тестирования в cross-integration команде

Similar to QA Fest 2017. Влада Бенюх.Организация End-to-End тестирования в cross-integration команде (20)

More from QAFest

More from QAFest (20)

Recently uploaded

Recently uploaded (20)

QA Fest 2017. Влада Бенюх.Организация End-to-End тестирования в cross-integration команде