SlideShare une entreprise Scribd logo
1  sur  28
Télécharger pour lire hors ligne
Understand Regression Testing
Testing Tiger Team (T Cube)
Liang Gao
Agenda
• Regression testing and Regression bugs.
• Regression’s When, What and How
• Importance of the automated regression
testing
• Managing regression testing
What is a regression testing
uncover regression bugs.
Regression Bug?
很多功能以前好好的,加
入新功能以后就不
Work 了!
Why ??
• No software design is 100% modulized.
• IP Network is layered (ISO 7 layers), break one
layer, rest of the layers won’t work
• The introduction of new code breaks the
existing code
• For Example:
– IPSec use Path MTU discovery
– ICMP can probe the MTU value
– IPSec use ICMP to change the MTU valule.
– If ICMP new code breaks the MTU function, part of
the IPSec won’t work
Regression bug is bad, very bad
• For a new product release, customer is ok that if the newly added
function has bugs, since the common understanding is new code is buggy.
And not all customer will use the newly added function anyway.
• But customer is not OK that the previously worked function is broken in
the new release, because that make things worse.
Don’t Piss Off the
Customer…..
How to Catch the Regression Bug
Use your analytical skills,
how to catch the
regression bug?
3 Reasons A Regression Test Case Fail
• It is because of the testbed.
– Testbed instabilities.
– Cable connections. Switch issues. Bad hardware
– Improper testbed configurations.
• It is because of the script issues
– Version update of output changes.
– Bad quality scripts.
• It is a software problem
– Then it could be a regression bug.
Understand Clean Run
• “Clean Run” means after running all your
regression test cases against a release, all
failed test cases has been analyzed on the
reason of failure.
• Test cases result is either “PASS” or “FAIL”
with a defects ID or Issue ID (script or test
bed problem).
Understand “Baseline”
• Before running every regression against certain
release, always ask: What is the baseline of
this release?
• The baseline release is a “Clean Run” release
with each test bases be either “PASS” or “FAIL”
with a defects ID. All failed test cases have
been debugged and found the root cause.
• The current result will always be compared
with the baseline result for delta (differences)
• The best baseline is to have high percentage of
the PASS cases.
Compare with Baseline
• The current release regression result has
the same result as baseline
– Baseline PASS, current release PASS
– Baseline FAIL, current release FAIL
• Different than baseline
– Baseline PASS, current release FAIL
– Baseline FAIL, current release PASS
Benefit of the Baseline
• Identify regression bug quickly.
• Shorten your debugging time.
– Most of the case, if baseline also failed, it
should fail on the same reason.
– when you run 10K+ test cases and have 5%
failure rate, baseline result speed up your
debugging time greatly.
• Better test result management
– Have a traceable result history
When to Start the Regression testing
• Before or after the new feature testing?
• Before or after the system testing?
• Before or after the solution testing?
• Before or after the release?
• How many times you think a regression
testing should be run?
Regression is headache
• You have no choice but to do it. (don’t piss off the customer)
• You have to do it whenever to release a new version
• You may have to do it for every build
• As more features added, regression test cases keep growing and growing
and growing….
• What if your company release a new build every week, or every other
day? (Cisco does)  Regression nightmare.
• Your engineers are bored of doing the same old thing again and again.
Bored  low quality of execution can not find bugs  piss off the
customer with regression bugs
Me and the Machine
• I am flexible and can execute beyond the test case
scope (explore and play a little to find more problems)
• I need to do different things, need to explore and to be
challenged.
• I need to sleep, rest, and play 斗地主,麻将 and go
to 青城山后山的 家农 乐 during the weekend.
• Machine will do exactly what I programmed him to do
• Machine can do it over and over without complaining
with exactly the same quality
• Machine can work overnight, over weekend and over
holidays.
Auto Regression is the Solution
• As regression test cases exploding, it can not
possibly be run by human (Cisco IOS regression
has 100,000 plus test cases, and need to be
finished in 2 days)
• It same time with quality (48 hours, for me,
means 6 days, for machine, means 2 days)
• It speeds up the release cycle: the faster the cycle
of a full regression, the more build can be
released (Cisco can release a build every 2 days,
Juniper can release a build every 4 days. Keep in
mind IOS has more than 6 million lines of code)
In Order to Run Auto Regression…..
• I need someone to write scripts, convert manual test cases in to scripts.
• I need an engine to execute thousands of scripts in batch mode, collect
pass and fail info.
• I need to have libraries. So when I write scripts, I can use APIs to speed up
my development.
• I need a database to store each regression cycles test report (running
results)
• I need a way to debug when scripts are failed (is it script problem? Test
bed problem? Or indeed the image problem)
• It would be nice if I can have a frame work that I can do all this.
Managing Regression Testing
• Schedule the run based on product release
schedule.
• Select suites
– If I can not cover them all, which set of features I need to pick
– How long can I finish it
– How much debugging time I need for failures.
• How many engineers I need to run (1 engineer
per 2 test beds?
• Regression engineers time
– Kick off the run
– Debugging (most of the time)
Find the Way to Reduce the Regression Cycle
• Look at the whole cycle of the regression testing, which part use most of the time
– Test bed management?
– Debugging and defect filing?
– Reporting?
• Lots of things can be automated (process automation) during a regression cycle)
• In Cisco, during a perfect run, a regression engineer don’t need to do anything
– Automatic copy new image after build team finish it
– Automatic load the image to router
– Automatic select the features need to be run based on the image name
(different image has different feature set)
– Automatic run
– Automatic reporting
Regression Bug Management
• Be careful whenever filing regression bugs,
need to be 100% Sure.
• How to be sure if it is a regression bug? Use
your analytical skills.
Make Regression As A Train
• Pull the regression cycle out of the release
cycle.
• Make regression cycle fixed (predictable).
• Make it an on-going, 24/7 operation.
• Benefit
– Any release can schedule a time to be on the
regression train, good to arrange regression
schedule.
– More regression, more stable of the product.
Regression Test Bed Management
• Standard topology test bed.
• Running in parallel (multiple identical
regression test beds)
• Fixed run time on each test bed on each
regression cycle
Don’t let regression team buy test bed
• Regression repository will keep expanding.
• More testbeds will be needed.
• Unlimited budget for regression team to buy
testbed? Probably not.
manage run time by minutes.
• Based on your release cycle, carefully control
total run time on a testbed.
• Each test bed has a maximum run time, if
exceed, consider build more testbeds (with in
budget)
Automat Regression - Engine
• If you have 1000 scripts
– How to run it in batch mode
– How to know which one is fail, which one is pass at any given time
– How to debug when a test case failed (pause on failure)
– How to move job across test bed
– Where to report the test results
– Where to keep the historical running logs
– How to run only certain modules
– How to select cases based on different release version
– How to run scripts even device is crashed
Automat Regression – Test Harness
• Need a place to store all test results
• Need to have version/release info
• Need to be able to compare from release to
release
Version control
• 1st draft, Feb, 2008, Liang Gao
Understand regression testing

Contenu connexe

Tendances

What is Regression Testing? | Edureka
What is Regression Testing? | EdurekaWhat is Regression Testing? | Edureka
What is Regression Testing? | EdurekaEdureka!
 
Software testing methods, levels and types
Software testing methods, levels and typesSoftware testing methods, levels and types
Software testing methods, levels and typesConfiz
 
Automated Testing with Agile
Automated Testing with AgileAutomated Testing with Agile
Automated Testing with AgileKen McCorkell
 
Software testing.ppt
Software testing.pptSoftware testing.ppt
Software testing.pptKomal Garg
 
Agile Testing - presentation for Agile User Group
Agile Testing - presentation for Agile User GroupAgile Testing - presentation for Agile User Group
Agile Testing - presentation for Agile User Groupsuwalki24.pl
 
Regression testing
Regression testingRegression testing
Regression testingMohua Amin
 
Basic Guide to Manual Testing
Basic Guide to Manual TestingBasic Guide to Manual Testing
Basic Guide to Manual TestingHiral Gosani
 
Basic software-testing-concepts
Basic software-testing-conceptsBasic software-testing-concepts
Basic software-testing-conceptsmedsherb
 
Performance and load testing
Performance and load testingPerformance and load testing
Performance and load testingsonukalpana
 
Manual testing concepts course 1
Manual testing concepts course 1Manual testing concepts course 1
Manual testing concepts course 1Raghu Kiran
 
An introduction to Software Testing and Test Management
An introduction to Software Testing and Test ManagementAn introduction to Software Testing and Test Management
An introduction to Software Testing and Test ManagementAnuraj S.L
 
Bug life cycle
Bug life cycleBug life cycle
Bug life cycleBugRaptors
 
Regression testing
Regression testingRegression testing
Regression testingHarsh verma
 
Software Testing - Part 1 (Techniques, Types, Levels, Methods, STLC, Bug Life...
Software Testing - Part 1 (Techniques, Types, Levels, Methods, STLC, Bug Life...Software Testing - Part 1 (Techniques, Types, Levels, Methods, STLC, Bug Life...
Software Testing - Part 1 (Techniques, Types, Levels, Methods, STLC, Bug Life...Ankit Prajapati
 

Tendances (20)

Automation Testing
Automation TestingAutomation Testing
Automation Testing
 
What is Regression Testing? | Edureka
What is Regression Testing? | EdurekaWhat is Regression Testing? | Edureka
What is Regression Testing? | Edureka
 
Software testing methods, levels and types
Software testing methods, levels and typesSoftware testing methods, levels and types
Software testing methods, levels and types
 
Automated Testing with Agile
Automated Testing with AgileAutomated Testing with Agile
Automated Testing with Agile
 
Software testing.ppt
Software testing.pptSoftware testing.ppt
Software testing.ppt
 
Introduction to Agile Testing
Introduction to Agile TestingIntroduction to Agile Testing
Introduction to Agile Testing
 
Agile Testing - presentation for Agile User Group
Agile Testing - presentation for Agile User GroupAgile Testing - presentation for Agile User Group
Agile Testing - presentation for Agile User Group
 
Testing methodology
Testing methodologyTesting methodology
Testing methodology
 
Regression testing
Regression testingRegression testing
Regression testing
 
Basic Guide to Manual Testing
Basic Guide to Manual TestingBasic Guide to Manual Testing
Basic Guide to Manual Testing
 
Introduction to Software Test Automation
Introduction to Software Test AutomationIntroduction to Software Test Automation
Introduction to Software Test Automation
 
Basic software-testing-concepts
Basic software-testing-conceptsBasic software-testing-concepts
Basic software-testing-concepts
 
Performance and load testing
Performance and load testingPerformance and load testing
Performance and load testing
 
Manual testing concepts course 1
Manual testing concepts course 1Manual testing concepts course 1
Manual testing concepts course 1
 
An introduction to Software Testing and Test Management
An introduction to Software Testing and Test ManagementAn introduction to Software Testing and Test Management
An introduction to Software Testing and Test Management
 
Test cases
Test casesTest cases
Test cases
 
Bug life cycle
Bug life cycleBug life cycle
Bug life cycle
 
Regression testing
Regression testingRegression testing
Regression testing
 
Software Testing - Part 1 (Techniques, Types, Levels, Methods, STLC, Bug Life...
Software Testing - Part 1 (Techniques, Types, Levels, Methods, STLC, Bug Life...Software Testing - Part 1 (Techniques, Types, Levels, Methods, STLC, Bug Life...
Software Testing - Part 1 (Techniques, Types, Levels, Methods, STLC, Bug Life...
 
Software testing
Software testingSoftware testing
Software testing
 

Similaire à Understand regression testing

A lean automation blueprint for testing in continuous delivery
A lean automation blueprint for testing in continuous deliveryA lean automation blueprint for testing in continuous delivery
A lean automation blueprint for testing in continuous deliverySauce Labs
 
Performance tuning Grails applications
Performance tuning Grails applicationsPerformance tuning Grails applications
Performance tuning Grails applicationsLari Hotari
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comAviran Mordo
 
Performance tuning Grails applications
 Performance tuning Grails applications Performance tuning Grails applications
Performance tuning Grails applicationsGR8Conf
 
Automate Everything! (No stress development/Tallinn)
Automate Everything! (No stress development/Tallinn)Automate Everything! (No stress development/Tallinn)
Automate Everything! (No stress development/Tallinn)Arto Santala
 
Test Automation
Test AutomationTest Automation
Test Automation99pillar
 
An Introduction To Software Development - Final Review
An Introduction To Software Development - Final ReviewAn Introduction To Software Development - Final Review
An Introduction To Software Development - Final ReviewBlue Elephant Consulting
 
Scrum and-xp-from-the-trenches 06 testing
Scrum and-xp-from-the-trenches 06 testingScrum and-xp-from-the-trenches 06 testing
Scrum and-xp-from-the-trenches 06 testingHossam Hassan
 
Customized Scrum
Customized ScrumCustomized Scrum
Customized ScrumAbdul Karim
 
What Do We Automate First
What Do We Automate FirstWhat Do We Automate First
What Do We Automate Firstrrice2000
 
Why Software Test Performance Matters
Why Software Test Performance MattersWhy Software Test Performance Matters
Why Software Test Performance MattersSolano Labs
 
London Atlassian User Group - February 2014
London Atlassian User Group - February 2014London Atlassian User Group - February 2014
London Atlassian User Group - February 2014Steve Smith
 
Team wide testing
Team wide testingTeam wide testing
Team wide testingEthan Huang
 
5 Steps on the Way to Continuous Delivery
5 Steps on the Way to Continuous Delivery5 Steps on the Way to Continuous Delivery
5 Steps on the Way to Continuous DeliveryXebiaLabs
 
Agile Acceptance testing with Fitnesse
Agile Acceptance testing with FitnesseAgile Acceptance testing with Fitnesse
Agile Acceptance testing with FitnesseClareMcLennan
 
Alm with tfs 2013
Alm with tfs 2013Alm with tfs 2013
Alm with tfs 2013MSDEVMTL
 
What a DevOps specialist has to know about static code analysis
What a DevOps specialist has to know about static code analysisWhat a DevOps specialist has to know about static code analysis
What a DevOps specialist has to know about static code analysisAndrey Karpov
 

Similaire à Understand regression testing (20)

A lean automation blueprint for testing in continuous delivery
A lean automation blueprint for testing in continuous deliveryA lean automation blueprint for testing in continuous delivery
A lean automation blueprint for testing in continuous delivery
 
Performance tuning Grails applications
Performance tuning Grails applicationsPerformance tuning Grails applications
Performance tuning Grails applications
 
Road to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.comRoad to Continuous Delivery - Wix.com
Road to Continuous Delivery - Wix.com
 
Performance tuning Grails applications
 Performance tuning Grails applications Performance tuning Grails applications
Performance tuning Grails applications
 
Tdd
TddTdd
Tdd
 
Automate Everything! (No stress development/Tallinn)
Automate Everything! (No stress development/Tallinn)Automate Everything! (No stress development/Tallinn)
Automate Everything! (No stress development/Tallinn)
 
Test Automation
Test AutomationTest Automation
Test Automation
 
An Introduction To Software Development - Final Review
An Introduction To Software Development - Final ReviewAn Introduction To Software Development - Final Review
An Introduction To Software Development - Final Review
 
Scrum and-xp-from-the-trenches 06 testing
Scrum and-xp-from-the-trenches 06 testingScrum and-xp-from-the-trenches 06 testing
Scrum and-xp-from-the-trenches 06 testing
 
Customized Scrum
Customized ScrumCustomized Scrum
Customized Scrum
 
What Do We Automate First
What Do We Automate FirstWhat Do We Automate First
What Do We Automate First
 
Why Software Test Performance Matters
Why Software Test Performance MattersWhy Software Test Performance Matters
Why Software Test Performance Matters
 
London Atlassian User Group - February 2014
London Atlassian User Group - February 2014London Atlassian User Group - February 2014
London Atlassian User Group - February 2014
 
Team wide testing
Team wide testingTeam wide testing
Team wide testing
 
5 Steps on the Way to Continuous Delivery
5 Steps on the Way to Continuous Delivery5 Steps on the Way to Continuous Delivery
5 Steps on the Way to Continuous Delivery
 
Agile Acceptance testing with Fitnesse
Agile Acceptance testing with FitnesseAgile Acceptance testing with Fitnesse
Agile Acceptance testing with Fitnesse
 
Great! another bug
Great! another bugGreat! another bug
Great! another bug
 
Alm with tfs 2013
Alm with tfs 2013Alm with tfs 2013
Alm with tfs 2013
 
AgileTesting_Ver1.0
AgileTesting_Ver1.0AgileTesting_Ver1.0
AgileTesting_Ver1.0
 
What a DevOps specialist has to know about static code analysis
What a DevOps specialist has to know about static code analysisWhat a DevOps specialist has to know about static code analysis
What a DevOps specialist has to know about static code analysis
 

Plus de gaoliang641

Why we didn't catch that
Why we didn't catch thatWhy we didn't catch that
Why we didn't catch thatgaoliang641
 
Why we didn't catch that application bugs
Why we didn't catch that   application bugsWhy we didn't catch that   application bugs
Why we didn't catch that application bugsgaoliang641
 
Understand release engineering
Understand release engineeringUnderstand release engineering
Understand release engineeringgaoliang641
 
The art of system and solution testing
The art of system and solution testingThe art of system and solution testing
The art of system and solution testinggaoliang641
 
Tester performance evaluation
Tester performance evaluationTester performance evaluation
Tester performance evaluationgaoliang641
 
Tester developer interaction
Tester developer interactionTester developer interaction
Tester developer interactiongaoliang641
 
Tester career path
Tester career pathTester career path
Tester career pathgaoliang641
 
Agile testing for large projects
Agile testing for large projectsAgile testing for large projects
Agile testing for large projectsgaoliang641
 
Project management for qa manager
Project management for qa managerProject management for qa manager
Project management for qa managergaoliang641
 
Make good use of explortary testing
Make good use of explortary testingMake good use of explortary testing
Make good use of explortary testinggaoliang641
 
Bug best practice
Bug best practiceBug best practice
Bug best practicegaoliang641
 
Lessons learned on localization testing
Lessons learned on localization testingLessons learned on localization testing
Lessons learned on localization testinggaoliang641
 
Lessons learned on software testing automation
Lessons learned on software testing automationLessons learned on software testing automation
Lessons learned on software testing automationgaoliang641
 
How to become a testing expert
How to become a testing expertHow to become a testing expert
How to become a testing expertgaoliang641
 
Functionality testing techniqu
Functionality testing techniquFunctionality testing techniqu
Functionality testing techniqugaoliang641
 
Protocol Security Testing best practice
Protocol Security Testing best practiceProtocol Security Testing best practice
Protocol Security Testing best practicegaoliang641
 
Backward thinking design qa system for quality goals
Backward thinking   design qa system for quality goalsBackward thinking   design qa system for quality goals
Backward thinking design qa system for quality goalsgaoliang641
 
Automation framework design and implementation
Automation framework design and implementationAutomation framework design and implementation
Automation framework design and implementationgaoliang641
 
Automation from start to finish
Automation   from start to finishAutomation   from start to finish
Automation from start to finishgaoliang641
 
Agile testing for large projects
Agile testing for large projectsAgile testing for large projects
Agile testing for large projectsgaoliang641
 

Plus de gaoliang641 (20)

Why we didn't catch that
Why we didn't catch thatWhy we didn't catch that
Why we didn't catch that
 
Why we didn't catch that application bugs
Why we didn't catch that   application bugsWhy we didn't catch that   application bugs
Why we didn't catch that application bugs
 
Understand release engineering
Understand release engineeringUnderstand release engineering
Understand release engineering
 
The art of system and solution testing
The art of system and solution testingThe art of system and solution testing
The art of system and solution testing
 
Tester performance evaluation
Tester performance evaluationTester performance evaluation
Tester performance evaluation
 
Tester developer interaction
Tester developer interactionTester developer interaction
Tester developer interaction
 
Tester career path
Tester career pathTester career path
Tester career path
 
Agile testing for large projects
Agile testing for large projectsAgile testing for large projects
Agile testing for large projects
 
Project management for qa manager
Project management for qa managerProject management for qa manager
Project management for qa manager
 
Make good use of explortary testing
Make good use of explortary testingMake good use of explortary testing
Make good use of explortary testing
 
Bug best practice
Bug best practiceBug best practice
Bug best practice
 
Lessons learned on localization testing
Lessons learned on localization testingLessons learned on localization testing
Lessons learned on localization testing
 
Lessons learned on software testing automation
Lessons learned on software testing automationLessons learned on software testing automation
Lessons learned on software testing automation
 
How to become a testing expert
How to become a testing expertHow to become a testing expert
How to become a testing expert
 
Functionality testing techniqu
Functionality testing techniquFunctionality testing techniqu
Functionality testing techniqu
 
Protocol Security Testing best practice
Protocol Security Testing best practiceProtocol Security Testing best practice
Protocol Security Testing best practice
 
Backward thinking design qa system for quality goals
Backward thinking   design qa system for quality goalsBackward thinking   design qa system for quality goals
Backward thinking design qa system for quality goals
 
Automation framework design and implementation
Automation framework design and implementationAutomation framework design and implementation
Automation framework design and implementation
 
Automation from start to finish
Automation   from start to finishAutomation   from start to finish
Automation from start to finish
 
Agile testing for large projects
Agile testing for large projectsAgile testing for large projects
Agile testing for large projects
 

Dernier

UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 

Dernier (20)

UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 

Understand regression testing

  • 1. Understand Regression Testing Testing Tiger Team (T Cube) Liang Gao
  • 2. Agenda • Regression testing and Regression bugs. • Regression’s When, What and How • Importance of the automated regression testing • Managing regression testing
  • 3. What is a regression testing uncover regression bugs.
  • 5. Why ?? • No software design is 100% modulized. • IP Network is layered (ISO 7 layers), break one layer, rest of the layers won’t work • The introduction of new code breaks the existing code • For Example: – IPSec use Path MTU discovery – ICMP can probe the MTU value – IPSec use ICMP to change the MTU valule. – If ICMP new code breaks the MTU function, part of the IPSec won’t work
  • 6. Regression bug is bad, very bad • For a new product release, customer is ok that if the newly added function has bugs, since the common understanding is new code is buggy. And not all customer will use the newly added function anyway. • But customer is not OK that the previously worked function is broken in the new release, because that make things worse. Don’t Piss Off the Customer…..
  • 7. How to Catch the Regression Bug Use your analytical skills, how to catch the regression bug?
  • 8. 3 Reasons A Regression Test Case Fail • It is because of the testbed. – Testbed instabilities. – Cable connections. Switch issues. Bad hardware – Improper testbed configurations. • It is because of the script issues – Version update of output changes. – Bad quality scripts. • It is a software problem – Then it could be a regression bug.
  • 9. Understand Clean Run • “Clean Run” means after running all your regression test cases against a release, all failed test cases has been analyzed on the reason of failure. • Test cases result is either “PASS” or “FAIL” with a defects ID or Issue ID (script or test bed problem).
  • 10. Understand “Baseline” • Before running every regression against certain release, always ask: What is the baseline of this release? • The baseline release is a “Clean Run” release with each test bases be either “PASS” or “FAIL” with a defects ID. All failed test cases have been debugged and found the root cause. • The current result will always be compared with the baseline result for delta (differences) • The best baseline is to have high percentage of the PASS cases.
  • 11. Compare with Baseline • The current release regression result has the same result as baseline – Baseline PASS, current release PASS – Baseline FAIL, current release FAIL • Different than baseline – Baseline PASS, current release FAIL – Baseline FAIL, current release PASS
  • 12. Benefit of the Baseline • Identify regression bug quickly. • Shorten your debugging time. – Most of the case, if baseline also failed, it should fail on the same reason. – when you run 10K+ test cases and have 5% failure rate, baseline result speed up your debugging time greatly. • Better test result management – Have a traceable result history
  • 13. When to Start the Regression testing • Before or after the new feature testing? • Before or after the system testing? • Before or after the solution testing? • Before or after the release? • How many times you think a regression testing should be run?
  • 14. Regression is headache • You have no choice but to do it. (don’t piss off the customer) • You have to do it whenever to release a new version • You may have to do it for every build • As more features added, regression test cases keep growing and growing and growing…. • What if your company release a new build every week, or every other day? (Cisco does)  Regression nightmare. • Your engineers are bored of doing the same old thing again and again. Bored  low quality of execution can not find bugs  piss off the customer with regression bugs
  • 15. Me and the Machine • I am flexible and can execute beyond the test case scope (explore and play a little to find more problems) • I need to do different things, need to explore and to be challenged. • I need to sleep, rest, and play 斗地主,麻将 and go to 青城山后山的 家农 乐 during the weekend. • Machine will do exactly what I programmed him to do • Machine can do it over and over without complaining with exactly the same quality • Machine can work overnight, over weekend and over holidays.
  • 16. Auto Regression is the Solution • As regression test cases exploding, it can not possibly be run by human (Cisco IOS regression has 100,000 plus test cases, and need to be finished in 2 days) • It same time with quality (48 hours, for me, means 6 days, for machine, means 2 days) • It speeds up the release cycle: the faster the cycle of a full regression, the more build can be released (Cisco can release a build every 2 days, Juniper can release a build every 4 days. Keep in mind IOS has more than 6 million lines of code)
  • 17. In Order to Run Auto Regression….. • I need someone to write scripts, convert manual test cases in to scripts. • I need an engine to execute thousands of scripts in batch mode, collect pass and fail info. • I need to have libraries. So when I write scripts, I can use APIs to speed up my development. • I need a database to store each regression cycles test report (running results) • I need a way to debug when scripts are failed (is it script problem? Test bed problem? Or indeed the image problem) • It would be nice if I can have a frame work that I can do all this.
  • 18. Managing Regression Testing • Schedule the run based on product release schedule. • Select suites – If I can not cover them all, which set of features I need to pick – How long can I finish it – How much debugging time I need for failures. • How many engineers I need to run (1 engineer per 2 test beds? • Regression engineers time – Kick off the run – Debugging (most of the time)
  • 19. Find the Way to Reduce the Regression Cycle • Look at the whole cycle of the regression testing, which part use most of the time – Test bed management? – Debugging and defect filing? – Reporting? • Lots of things can be automated (process automation) during a regression cycle) • In Cisco, during a perfect run, a regression engineer don’t need to do anything – Automatic copy new image after build team finish it – Automatic load the image to router – Automatic select the features need to be run based on the image name (different image has different feature set) – Automatic run – Automatic reporting
  • 20. Regression Bug Management • Be careful whenever filing regression bugs, need to be 100% Sure. • How to be sure if it is a regression bug? Use your analytical skills.
  • 21. Make Regression As A Train • Pull the regression cycle out of the release cycle. • Make regression cycle fixed (predictable). • Make it an on-going, 24/7 operation. • Benefit – Any release can schedule a time to be on the regression train, good to arrange regression schedule. – More regression, more stable of the product.
  • 22. Regression Test Bed Management • Standard topology test bed. • Running in parallel (multiple identical regression test beds) • Fixed run time on each test bed on each regression cycle
  • 23. Don’t let regression team buy test bed • Regression repository will keep expanding. • More testbeds will be needed. • Unlimited budget for regression team to buy testbed? Probably not.
  • 24. manage run time by minutes. • Based on your release cycle, carefully control total run time on a testbed. • Each test bed has a maximum run time, if exceed, consider build more testbeds (with in budget)
  • 25. Automat Regression - Engine • If you have 1000 scripts – How to run it in batch mode – How to know which one is fail, which one is pass at any given time – How to debug when a test case failed (pause on failure) – How to move job across test bed – Where to report the test results – Where to keep the historical running logs – How to run only certain modules – How to select cases based on different release version – How to run scripts even device is crashed
  • 26. Automat Regression – Test Harness • Need a place to store all test results • Need to have version/release info • Need to be able to compare from release to release
  • 27. Version control • 1st draft, Feb, 2008, Liang Gao