Critical functionality testing

© 2013 Maveric Systems Limited
Critical Functionality
Testing (CFT)

© 2013 Maveric Systems Limited 2
▸ Introduction – CFT
▸ Need for CFT
▸ CFT Framework
▸ Benefits of CFT
▸ Advantages of CFT over Traditional Approach
▸ Case Study
Presentation Outline

▸ A pre-UAT round that quantifies and qualifies the application under
test for User Acceptance Testing (UAT)
▸ Test evolution – from our experiences across multiple engagements
▸ Is a strategy derived from our Risk Based Testing Framework
▸ Testing of Business Critical Functionalities across modules with
emphasis on Go-live functionalities
▸ CFT will not replace UAT, it only ensures a better UAT and helps
meeting overall project timelines
▸ CFT – cannot pick all showstoppers issues, it only indicates what
trouble may lie ahead in UAT
▸ Can cover about 20-30% of cases planned for UAT
Introduction – Critical
Functionality Testing

▸ Assesses the stability of the application
▸ Quantifies the percent of critical functions which have defects
across modules
▸ Irons out differences if any on functionalities with the business
▸ Missing or wrong parameter set-ups can be identified before UAT
thereby negating unnecessary re-running of scripts in UAT rounds,
especially batch dependent cases due to missing set-ups
▸ Interfaces – understanding the compatibility of the application with
3rd party interfaces
Need for CFT

Plan Specify Build Test Implement
Application Lifecycle Cost
International Software Benchmarking
Standard Group (Data Sample 404 projects)
www.isbsg.org
Real time
Industry
Maveric Approach
5% 23% 41% 21% 9%

Specify Define Design Code Test Deploy Production
Early Detection StrategyNoofdefects
(Increasing) time and cost to fix defects
Time & Cost
Prevention
strategy
Traditional detection
strategy
Early detection
strategy (CFT)

Best Practices – CFT
Framework
CFT framework
Critical
functional
scenarios
Parameter
set-ups
Interfaces
connectivity
and
compatibility
Batch
processes
Test bed
readiness and
database sync
up across all
systems

Best Practices – CFT
Framework
CFT framework
Test bed
readiness and
database sync
up across all
systems
Critical
functional
scenarios
Parameter
set-ups
Interfaces
connectivity
and
compatibility
Batch
processes

▸ Transactions with high financial impact
▸ Transactions involving critical functional flow
▸ Batch dependent scenarios
▸ Focused discussions and meetings with business to
identify critical functional points
▸ Identifying any changes to Business Requirements at an
early stage thereby avoiding the need to raise CRs
Critical Functional Scenarios –
Rationale

▸ Compatibility of interfaces with the core system
▸ Communication between different 3rd party interfaces
▸ Helps in optimizing the Run Plan covering only
functionality of interfaces which are test ready for UAT
Interface Connectivity and
Compatibility

▸ Batch critical reports – MIS reports
» Financial reports
» Regulatory and compliance reports
▸ Focuses more on the data generated in the report than cosmetic
ones like alignment etc.
▸ Example: Premium collection report
» Claim process report
Reports

▸ Ensures all parameter set-up are completed before starting UAT
▸ Special focus on escalations on authorizations and approvals
▸ Negates unnecessary need to re-run cases due to missing set-up
Set-up Parameters

▸ End of day and end of month process are run
▸ End of year process is run by moving dates/defining holidays
▸ Ensures there are no hiccups in execution of test cases as per the
Run Plan during UAT
▸ Defects in batch process adversely effect the schedule of testing,
CFT helps in negating such high risks entering into UAT
Batch Processes

▸ Ensures different databases are in sync with the core
▸ Access to different databases for migrated test data
▸ Gives an idea on the quality of migrated data
Test Bed Readiness and
Database Sync-up

Entry criteria for CFT – Done when SIT is complete and signed off
▸ Business critical functional points have been identified after discussion and
sign-off by Business Line Managers
CFT exit criteria
▸ 70% of the Critical Functions should pass
▸ Number of showstopper defects to be less than 20%
▸ All interfaces should be up and running
▸ All databases should be made in Sync
▸ EOD should be successful
CFT Entry and Exit

Insurance
▸ Customer Creation (CIF)
▸ Opening of Customer & Loan Accounts
▸ Cash Deposits and Withdrawals
▸ Fund Transfers
▸ End of Day Processing
▸ Customer Creation
▸ New Business
▸ Underwriting and Rating Factors
▸ Premium Computation
▸ Mid Term Endorsements – Financial
▸ Mid Term Endorsements – Non Financial
▸ Collections and Refunds
▸ Claim Handling
▸ Renewals
▸ Reports and Accounting
▸ Batch Processing
Examples

Observations and Risk
Classification
Key process/activities Risk classification Potential risk
Critical functional scenarios
Under control, however non conformance
by business users can lead to risk
Test environment readiness
Missing set-ups and a sync databases may
lead to re-running cases thereby increasing
timelines and effort
Test readiness of interfaces
Knowing incompatibility or non-availability
of major functionality during UAT may cause
major set back to timelines
Reports
Non-availability of compliance or regulatory
reports during UAT impacts Go-live date
Batch process
Abnormal timelines or showstopper here will
affect the entire testing schedule
Low Risk
Medium Risk
High Risk

▸ 70% of the critical functions should pass
▸ Number of showstopper defects to be less than 20%
▸ All interfaces should be up and running
▸ All databases should be made in sync
▸ EOD should be successful
CFT Exit – UAT Entry

▸ Reduce costs – Identifying defects at an early stage reduces costs and also only
50-60% of the testing team is present onsite for CFT
▸ CFT design is done parallel with UAT design hence separate effort is not required
▸ Change requests on critical functions if any can be identified at an early stage by
the business
▸ Reduces the effort of business users during UAT
▸ Same design pack can be re-used with minor modifications to do a quick sanity
check after releases or enhancements
▸ Helps the business and also the testing team to understand the differences
between legacy and the new system
Benefits of CFT for Business

▸ Identifying most of the business critical defects at an early stage
gives the development team more time to fix
▸ Helps the development team to reassess priorities and areas to
focus on as the development team is not bombarded with cosmetic
and minor defects earlier
Benefits of CFT for the
Developers’

▸ Gives a metrics based report on defects thereby helping to identify
areas to focus on during UAT
▸ Helps project management team identifying areas to focus-on and
reschedule timelines if necessary
▸ Exactly identify when to start and bring in the full team for UAT
Benefits of CFT for the
Testing Team

▸ Covers both critical and non-critical
transactions hence priorities may get
misplaced on non-critical functions
▸ Critical business defects may be found
at the end of the round 1 which will
take time to fix
▸ Defects on critical function may get
overlooked due to overwhelming
number of defects
▸ May need to stop UAT mid-way due to
large showstoppers and critical defects
thereby wasting time and money
Advantage CFT
Traditional Approach CFT
▸ Focus on critical functions from a very
early stage of the testing lifecycle
▸ Helps the business to know the
percent and the status of defects on
Go-Live critical functions right from the
start
▸ Cuts down on time spent on non-
critical functions at the start of the
testing assignment
▸ Focused discussions and meeting with
business to exactly identify the critical
functions

CFT – Where Does it Fit in the
Testing Calendar
CFT runs parallel
to UAT Design
and CFT uses
30% of UAT
design
CFT ensures a high
quality UAT is met
CFT sets entry criteria
for UAT Entry

Case Study
▸ A leading commercial insurance
organization in the Middle East was
migrating from a legacy core insurance
system to a new core insurance
solution with specific business
customization and enhancements
▸ The core-insurance solution was to be
implemented phase-wise (in 3 phases)
across 73 branches through out the
country
▸ The phase I rollout was being planned
with aggressive timelines as the
insurance company had to do catch-up
with its competitors in converting to
new technology solution which was of
paramount importance to sustain its
market share
Client Context Challenges
▸ Documentation was very minimal and
not updated
▸ Frequent changes to requirements
▸ Need to collaborate with the insurance
company on one side, and the
application vendor’s on the other side
with stiff timelines
▸ An application architecture that
required extensive batch runs for EODs
etc.
▸ Domain competence in core insurance
solution to address country specific
requirements and regulations
especially for reports (compliance)
▸ An approach to ensure production
quality at the same time keeping cost
and timelines at a tight leach

▸ Maveric adopted CFT based testing approach to meet timelines
at the same time ensuring product quality
▸ Only 60% of the team was positioned onsite to cut down the
project cost
▸ The team comprised of domain experts who organized daily
meetings with Business Line Managers to get clear cut
understanding of the customizations
▸ Weekly two meetings were conducted involving both the
development team and the Business Line Managers to ensure that
everyone was in the same page with the respect to the functional
specs of the customizations
▸ Minutes of the meeting were documented and circulated
to all the parties involved and were used as BRS documents
for reference
Maveric Solution – CFT

▸ The end of day was taking 12hrs to complete during CFT, this was immediately
worked upon and brought to 5hrs during UAT Round 1. Further, analysis were
done by the development team and fixing was too complex and took time, but
at the end of Round 2, it was brought down to 1 hour 30 minutes
▸ Due to extensive customization the solution had around 40% defects in business
critical functionalities
▸ Project Management company delayed the start of UAT by 10 days to give more
time for developers’ to fix the critical defects. The remaining of the test team
was not brought board during this time thereby reducing the cost
▸ The development team suggested work around solutions for few defects as the
fix was complex and required more time. Those fixes as promised were delivered
for Round 2
Maveric Solution – CFT (Cont’d)

▸ Gap Analysis – Identified that only five interfaces out of
the 7 were test ready
▸ Priorities were reassessed and only 5 interfaces namely
ATM, I-Score, CBE Files Electronic Cheque Clearing and
Covered Visa were taken up for Phase I
▸ Reports – One resource from Maveric was dedicated to
test CBE reports in CFT
Maveric Solution – CFT (Cont’d)

▸ To Introduce Early Defect Detection (EDD) Strategy,
Essential Start -Stop- Continue Strategies are:
▸ CFT is a Win-Win strategy for the client, development teams and
also for testing team to mutually ensure a quality solution goes into
production within the timelines planned
Conclusion
Start Stop Continue
Planning ahead and release
application for EDD early
Introducing last minute
requirement changes
Focus on Good Quality UAT
Engage specialized testing
partners and build high toll gate
for UAT Entry
Defining poor UAT Entry
Criteria's
To reject poorly built application
Plan for CFT to be included in
the overall project
implementation Plan
Don’t engage Users till CFT toll
gates are met
Engage Users in UAT and build
lifecycle cases

Critical functionality testing

Recommandé

Recommandé

Contenu connexe

Similaire à Critical functionality testing

Similaire à Critical functionality testing (20)

Plus de Maveric Systems

Plus de Maveric Systems (12)

Dernier

Dernier (20)

Critical functionality testing