SlideShare une entreprise Scribd logo
1  sur  37
Télécharger pour lire hors ligne
Software Analysis and Testing
Métodos Formais em Engenharia de Software


                                      January 2010    Arent Janszoon Ernststraat 595-H
                                       Joost Visser   NL-1082 LD Amsterdam
                                                      info@sig.nl
                                                      www.sig.nl
Me


CV                                                                                                                      2 I 112

 • Technical University of Delft, Computer Science, MSc 1997
 • University of Leiden, Philosophy, MA 1997
 • CWI (Center for Mathematics and Informatics), PhD 2003
 • Software Improvement Group, developer, consultant, etc, 2002-2003
 • Universidade do Minho, Post-doc, 2004-2007
 • Software Improvement Group, Head of Research, 2007-…
Research
 • Grammars, traversal, transformation, generation
 • Functional programming, rewriting strategies
 • Software quality, metrics, reverse engineering

        Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Software Improvement Group


Company                                                                                                                 3 I 112

 • Spin-off from CWI in 2000, self-owned, independent
 • Management consultancy grounded in source code analysis
 • Innovative, strong academic background, award-winning, profitable
Services
 • Software Risk Assessments (snapshot) and Software Monitoring (continuous)
 • Toolset enables to analyze source code in an automated manner
 • Experienced staff transforms analysis data into recommendations
 • We analyze over 50 systems annually
 • Focus on technical quality, primarily maintainability / evolvability


        Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Services



                                                                                                                      4 I 112
            DocGen
              • Automated generation of technical documentation
              • Reduce learning time, assist impact analysis, support migration, …


            Software Risk Assessment
              • In-depth investigation of software quality and risks
              • Answers specific research questions


            Software Monitoring
              • Continuous measurement, feedback, and decision support
              • Guard quality from start to finish


            Software Product Certification
              • Five levels of technical quality
              • Evaluation by SIG, certification by TÜV Informationstechnik

      Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Who is using our services?



                                                                                                                      5 I 112

     Financial                            Public                 Logistics                   IT                   Other




      Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Structure of the lecture


                                                                                                                        6 I 112

 • Introduction SIG


 • General overview of software analysis and testing
 • Testing
 • Patterns
 • Quality & metrics
 • Reverse engineering




        Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Software Engineering



                                                                                                                       7 I 112




      Create                                         Change                                         Analyze

 requirements analysis                    refactor, fix, patch                             understand, assess
 design, code, compile                   maintain, renovate                                   evaluate, test
    configure, install                 evolve, update, improve                               measure, audit




       Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Software Analysis
(and Testing)


                                                                                                                      8 I 112

                                             Analysis

            Static                                                                        Dynamic
           Analysis                                                                       Analysis
     syntax checking                                                                    testing
      type checking                                                                   debugging
       code metrics                                                                program spectra
      style checking                                                               instrumentation
        verification                                                                   profiling
     reverse engineering                                                            benchmarking
      decompilation                                                                  log analysis


      Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Is testing un-cool?


                                                                                                                        9 I 112

Edsger Wybe Dijkstra (1930 - 2002)




 • “Program testing can be used to show the presence of bugs,
    but never to show their absence!”
   Notes On Structured Programming, 1970

 • “Program testing can be a very effective way to show the presence of bugs,
    but is hopelessly inadequate for showing their absence.”
   The Humble Programmer, ACM Turing Award Lecture, 1972

Does not mean: “Don’t test!!”

        Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Is testing un-cool?


Industry                                                                                                                   10 I 112

 • Testers earn less then developers
 • Testing is “mechanical”, developing is “creative”
 • Testing is done with what remains of the budget in what remains of the time


Academia
 • Testing is not part of the curriculum, or very minor part
 • Verification is superior to testing
 • Verification is more challenging than testing




           Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Software Analysis. How much?



                                                                                                                      11 I 112




                                                          50 - 75%




      Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Software Analysis. Enough?



                                                                                                                      12 I 112




                                                           $60 ×                           109




      Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Software Analysis. More?



                                                                                                                      13 I 112




                                                                   high profile
                                                                 low frequency




      Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Software Analysis
Room for improvement?


                                                                                                                             14 I 112

                      1994                                                                             2004

                                      Succeeded                                        Failed
                                        16%                                             18%


                                                                                                                           Succeeded
                                                                                                                             29%
  Failed
   31%




                                     Challenged
                                       53%                                                Challenged
                                                                                            53%




                      Standish Group, “The CHAOS Report”
           Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
So


                                                                                                                        15 I 112

 • Testing ⊂ Dynamic analysis ⊂ Analysis ⊂ S.E.
 • Analysis is a major and essential part of software engineering
 • Inadequate analysis costs billions


⇒
 • More effective and more efficient methods are needed
 • Interest will keep growing in both industry and research




        Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Structure of the lectures


                                                                                                                      16 I 112

                                                             Analysis

                           Static                                                                         Dynamic
                          Analysis                                                                        Analysis




metrics                 patterns                          models                                       testing



      Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
17 I 112




                                           TESTING




Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Testing



Kinds                                    Ways                                                                            18 I 112

  • Conformance                           • Manual
  • Interoperability                      • Automated
  • Performance                           • Randomized
  • Functional                            • Independent
  • White-box                             • User
  • Black-box                             • Developer
  • Acceptance
  • Integration                          With
  • Unit                                  • Plans
  • Component                             • Harness
  • System                                • Data
  • Smoke                                 • Method
  • Stress                                • Frameworks
         Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Testing
V-model


                                                                                                                       19 I 112




V-model =                                                                                     No testing while
 waterfall-1 • waterfall                                                                      programming!
       Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Testing
Eliminate waste


Waste                                                                                                                   20 I 112

 • Coding and debugging go hand-in-hand
 • Coding effort materializes in the delivered program
 • Debugging effort? Evaporates!


Automated tests
 • Small programs that capture debugging effort.
 • Invested effort is consolidated …
 • … and can be re-used without effort ad-infinitum


Unit testing

        Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
What is unit testing?



A unit test is …                                                                                                         21 I 112

  • fully automated and repeatable
  • easy to write and maintain
  • non-intrusive                                                                          TestCase
  • documenting
  • applies to the simplest piece of software

Tool support
 • JUnit and friends
                                            public void testMyMethod {
                                              X x = …;
                                              Y y = myMethod(x);
                                              Y yy = …;
                                              assertEquals(“WRONG”,yy,y)
                                            }
         Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Testing goals



Unit testing has the following goals:                                                                                  22 I 112

 • Improve quality
    • Test as specification
    • Test as bug repellent
    • Test as defect localization
 • Help to understand
    • Test as documentation
 • Reduce risk
    • Test as a safety net
    • Remove fear of change




       Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Observing unit-testing maturity in the wild
(characterization of the population)


Organization                                                                                                            23 I 112

  • public, financial, logistics
  • under contract, in house, product software
  • with test departments, without test departments
Architecture & Process
  • under architecture, using software factories
  • model driven, handwritten
  • open source frameworks, other frameworks
  • using use-cases/requirements
  • with blackbox tools, t-map
Technology
  • information systems, embedded
  • webbased, desktop apps
  • java, c#, 4GL’s, legacy
  • latest trend: in-code asserts (java.spring)
        Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Stage 1
No unit testing


Observations:                                                             24 I 112

 • Very few organizations use unit testing
 • Also brand new OO systems without any unit tests
 • Small software shops and internal IT departments
 • In legacy environments: programmers describe in words what tests they have
   done.

Symptoms:
 • Code is instable and error-prone
 • Lots of effort in post-development testing phases




        Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Stage 1
No unit testing


Excuses:                                                                                                                 25 I 112

  • “It is just additional code to maintain”
  • “The code is changing too much”
  • “We have a testing department”
  • “Testing can never prove the absence of errors”
  • “Testing is too expensive, the customer does not want to pay for it”
  • “We have black-box testing”

Action
  • Provide standardized framework to lower
    threshold
  • Pay for unit tests as deliverable, not as effort



         Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Stage 2
Unit test but no coverage measurement


Observations                                                                                                             26 I 112

  • Contract requires unit testing, not enforced
  • Revealed during conflicts
  • Unit testing receives low priority
  • Developers relapse into debugging practices without unit testing
  • Good initial intentions, bad execution
  • Large service providers

Symptoms:
  • Some unit tests available
  • Excluded from daily build
  • No indication when unit testing is sufficient
  • Producing unit test is an option, not a requirement



         Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Stage 2
Unit test but no coverage measurement


Excuses:                                                                                                                27 I 112

 • “There is no time, we are under pressure”
 • “We are constantly stopped to fix bugs”

Actions
 • Start measuring coverage
 • Include coverage measurement into nightly build
 • Include coverage result reports into process




        Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Stage 3
Coverage, not approaching 100%


Observations                                                                                                             28 I 112

 • Coverage is measured but gets stuck at 20%-50%
 • Ambitious teams, lacking experience
 • Code is not structured to be easily unit-testable




Symptoms:
 • Complex code in GUI layer
 • Libraries in daily build, custom code not in daily build




         Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Stage 3
Coverage, not approaching 100%


Excuses                                                                                                                  29 I 112

 • “we test our libraries thoroughly, that affects more customers”

Actions:
 • Refactor code to make it more easily testable
 • Teach advance unit testing patterns
 • Invest in set-up and mock-up




         Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Stage 4
Approaching 100%, but no test quality


Observations                                                                                                             30 I 112

  • Formal compliance with contract
  • Gaming the metrics
  • Off-shored, certified, bureaucratic software factories

Symptoms:
  • Empty tests
  • Tests without asserts.
  • Tests on high-level methods, rather than basic units

  • Need unit tests to test unit tests




         Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Stage 4
Approaching 100%, but no test quality


Anecdotes:                                                                                                               31 I 112

 • Tell me how you measure me, and I tell you how I behave
 • We have generated our unit tests (at first this seems a stupid idea)

Action:
 • Measure test quality
 • Number of asserts per unit test
 • Number of statements tested per unit test
 • Ratio of number of execution paths versus number of tests




         Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Stage 5
Measuring test quality


Enlightenment:                                                                                                           32 I 112

 • Only one organization: a Swiss company
 • Measure:
    • Production code incorporated in tests
    • number of assert and fail statements
    • low complexity (not too many ifs)
 • The process
    • part of daily build
    • “stop the line process”, fix bugs first by adding more tests
    • happy path and exceptions
    • code first, test first, either way




         Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Testing
Intermediate conclusion


Enormous potential for improvement:                                                                                    33 I 112

 • Do unit testing
 • Measure coverage
 • Measure test quality

 • May not help Ariane 5
 • Does increase success ratio for “normal” projects




       Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Randomized Testing
(quickcheck)


Randomized testing:                                                                                                     34 I 112

 • QuickCheck: initially developed for Haskell
 • Parameterize tests in the test data
 • Property = parameterized test
 • Generate test data randomly
 • Test each property in 100 different ways each time

Test generation

                                           -- | Range of inverse is domain.
Model-driven testing
                                           prop_RngInvDom r
                                             = rng (inv r) == dom r
Fault-injection                                where
                                                 types = r::Rel Int Integer

        Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Is testing un-cool?


                                                                                                                         35 I 112

Edsger Wybe Dijkstra (1930 - 2002)

  • “Program testing can be used to show the presence of bugs,
     but never to show their absence!”




Martin Fowler
 • “Don’t let the fear that testing can’t catch all bugs stop you
   from writing the tests that will catch most bugs.”




         Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Simple test metrics


Line coverage                                                                                                            36 I 112

  • Nr of test lines / nr of tested lines
Decision coverage
  • Nr of test methods / Sum of McCabe complexity index

Test granularity
 • Nr of test lines / nr of tests

Test efficiency
 • Decision coverage / line coverage




         Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
Testing
Exercises


Write unit tests                                                                                                         37 I 112

 • Using JUnit
 • E.g. for one of your own projects

Measure coverage
 • E.g. using Emma plug-in for Eclipse



Randomize one of your unit tests
 • Turn test into property with extract method refactoring
 • Write generator for test data
 • Instantiate property 100 times with random test data
 • Solution to j.visser@sig.eu


         Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.

Contenu connexe

En vedette

Risk management in Software Industry
Risk management in Software IndustryRisk management in Software Industry
Risk management in Software IndustryRehan Akhtar
 
Risk analysis Chapter
Risk analysis ChapterRisk analysis Chapter
Risk analysis ChapterSINGHZEE
 
16103271 software-testing-ppt
16103271 software-testing-ppt16103271 software-testing-ppt
16103271 software-testing-pptatish90
 
Sensitivity analysis
Sensitivity analysisSensitivity analysis
Sensitivity analysisMohamed Yaser
 
Risk analysis technique (ITLC version)
Risk analysis technique (ITLC version)Risk analysis technique (ITLC version)
Risk analysis technique (ITLC version)Trung. Le Thanh
 
Risk analysis and management
Risk analysis and managementRisk analysis and management
Risk analysis and managementgnitu
 
Software Risk Management
Software Risk ManagementSoftware Risk Management
Software Risk ManagementGunjan Patel
 
Software Risk Analysis
Software Risk AnalysisSoftware Risk Analysis
Software Risk AnalysisBrett Leonard
 
Risk management in software engineering
Risk management in software engineeringRisk management in software engineering
Risk management in software engineeringdeep sharma
 

En vedette (14)

[HCMC STC Jan 2015] Risk-Based Software Testing Approaches
[HCMC STC Jan 2015] Risk-Based Software Testing Approaches[HCMC STC Jan 2015] Risk-Based Software Testing Approaches
[HCMC STC Jan 2015] Risk-Based Software Testing Approaches
 
Risk management in Software Industry
Risk management in Software IndustryRisk management in Software Industry
Risk management in Software Industry
 
Risk analysis Chapter
Risk analysis ChapterRisk analysis Chapter
Risk analysis Chapter
 
16103271 software-testing-ppt
16103271 software-testing-ppt16103271 software-testing-ppt
16103271 software-testing-ppt
 
Sensitivity analysis
Sensitivity analysisSensitivity analysis
Sensitivity analysis
 
Risk Analysis
Risk AnalysisRisk Analysis
Risk Analysis
 
Project risk analysis
Project risk analysisProject risk analysis
Project risk analysis
 
Risk analysis technique (ITLC version)
Risk analysis technique (ITLC version)Risk analysis technique (ITLC version)
Risk analysis technique (ITLC version)
 
Risk analysis and management
Risk analysis and managementRisk analysis and management
Risk analysis and management
 
Software Risk Management
Software Risk ManagementSoftware Risk Management
Software Risk Management
 
Software Risk Analysis
Software Risk AnalysisSoftware Risk Analysis
Software Risk Analysis
 
Risk analysis
Risk analysisRisk analysis
Risk analysis
 
Risk management
Risk managementRisk management
Risk management
 
Risk management in software engineering
Risk management in software engineeringRisk management in software engineering
Risk management in software engineering
 

Similaire à 2010 01 lecture SIG UM MFES 1 - Intro and testing

2010 01 lecture SIG UM MFES 2 - Patterns metrics quality
2010 01 lecture SIG UM MFES 2 - Patterns metrics quality2010 01 lecture SIG UM MFES 2 - Patterns metrics quality
2010 01 lecture SIG UM MFES 2 - Patterns metrics qualityjstvssr
 
2010 01 lecture SIG UM MFES 3 - Reverse engineering
2010 01 lecture SIG UM MFES 3 - Reverse engineering2010 01 lecture SIG UM MFES 3 - Reverse engineering
2010 01 lecture SIG UM MFES 3 - Reverse engineeringjstvssr
 
Software Security
Software SecuritySoftware Security
Software SecuritySue Ganguli
 
Make a career in software testing: MobilePro - Mobile Testing Professional Pr...
Make a career in software testing: MobilePro - Mobile Testing Professional Pr...Make a career in software testing: MobilePro - Mobile Testing Professional Pr...
Make a career in software testing: MobilePro - Mobile Testing Professional Pr...CleanSoft Academy
 
Conventional and Object Oriented Software Engineering
Conventional and Object Oriented Software EngineeringConventional and Object Oriented Software Engineering
Conventional and Object Oriented Software Engineeringssrkai2020
 
Uni of Auckland Lecture 20110823
Uni of Auckland Lecture 20110823Uni of Auckland Lecture 20110823
Uni of Auckland Lecture 20110823Farid Vaswani
 
SDPM - Lecture 8 - Software quality assurance
SDPM - Lecture 8 - Software quality assuranceSDPM - Lecture 8 - Software quality assurance
SDPM - Lecture 8 - Software quality assuranceOpenLearningLab
 
QA in Digitalized World Kari Kakkonen WCSQ
QA in Digitalized World Kari Kakkonen WCSQQA in Digitalized World Kari Kakkonen WCSQ
QA in Digitalized World Kari Kakkonen WCSQKari Kakkonen
 
Software testing
Software testingSoftware testing
Software testingfatboysec
 
Software Testing For Software Development Lifecycle
Software Testing For Software Development LifecycleSoftware Testing For Software Development Lifecycle
Software Testing For Software Development LifecycleMonroe Anderton
 
Continuous Testing Landscape.pptx
Continuous Testing Landscape.pptxContinuous Testing Landscape.pptx
Continuous Testing Landscape.pptxMarc Hornbeek
 
Pentest is yesterday, DevSecOps is tomorrow
Pentest is yesterday, DevSecOps is tomorrowPentest is yesterday, DevSecOps is tomorrow
Pentest is yesterday, DevSecOps is tomorrowAmien Harisen Rosyandino
 
Software Process Improvement
Software Process ImprovementSoftware Process Improvement
Software Process ImprovementBilal Shah
 

Similaire à 2010 01 lecture SIG UM MFES 1 - Intro and testing (20)

2010 01 lecture SIG UM MFES 2 - Patterns metrics quality
2010 01 lecture SIG UM MFES 2 - Patterns metrics quality2010 01 lecture SIG UM MFES 2 - Patterns metrics quality
2010 01 lecture SIG UM MFES 2 - Patterns metrics quality
 
2010 01 lecture SIG UM MFES 3 - Reverse engineering
2010 01 lecture SIG UM MFES 3 - Reverse engineering2010 01 lecture SIG UM MFES 3 - Reverse engineering
2010 01 lecture SIG UM MFES 3 - Reverse engineering
 
Symbioun_Corporate_Profile
Symbioun_Corporate_ProfileSymbioun_Corporate_Profile
Symbioun_Corporate_Profile
 
Software Security
Software SecuritySoftware Security
Software Security
 
Make a career in software testing: MobilePro - Mobile Testing Professional Pr...
Make a career in software testing: MobilePro - Mobile Testing Professional Pr...Make a career in software testing: MobilePro - Mobile Testing Professional Pr...
Make a career in software testing: MobilePro - Mobile Testing Professional Pr...
 
Vinodhkumar
VinodhkumarVinodhkumar
Vinodhkumar
 
5 Quality
5 Quality5 Quality
5 Quality
 
Conventional and Object Oriented Software Engineering
Conventional and Object Oriented Software EngineeringConventional and Object Oriented Software Engineering
Conventional and Object Oriented Software Engineering
 
Uni of Auckland Lecture 20110823
Uni of Auckland Lecture 20110823Uni of Auckland Lecture 20110823
Uni of Auckland Lecture 20110823
 
Rohit_Pandey
Rohit_PandeyRohit_Pandey
Rohit_Pandey
 
SDPM - Lecture 8 - Software quality assurance
SDPM - Lecture 8 - Software quality assuranceSDPM - Lecture 8 - Software quality assurance
SDPM - Lecture 8 - Software quality assurance
 
QA in Digitalized World Kari Kakkonen WCSQ
QA in Digitalized World Kari Kakkonen WCSQQA in Digitalized World Kari Kakkonen WCSQ
QA in Digitalized World Kari Kakkonen WCSQ
 
Introduction to DevOps
Introduction to DevOpsIntroduction to DevOps
Introduction to DevOps
 
Software testing
Software testingSoftware testing
Software testing
 
Software Testing For Software Development Lifecycle
Software Testing For Software Development LifecycleSoftware Testing For Software Development Lifecycle
Software Testing For Software Development Lifecycle
 
Continuous Testing Landscape.pptx
Continuous Testing Landscape.pptxContinuous Testing Landscape.pptx
Continuous Testing Landscape.pptx
 
project_details
project_detailsproject_details
project_details
 
Smef2009
Smef2009Smef2009
Smef2009
 
Pentest is yesterday, DevSecOps is tomorrow
Pentest is yesterday, DevSecOps is tomorrowPentest is yesterday, DevSecOps is tomorrow
Pentest is yesterday, DevSecOps is tomorrow
 
Software Process Improvement
Software Process ImprovementSoftware Process Improvement
Software Process Improvement
 

Dernier

The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxheathfieldcps1
 
Education and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptxEducation and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptxraviapr7
 
How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17Celine George
 
Over the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptxOver the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptxraviapr7
 
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptxClinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptxraviapr7
 
5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...CaraSkikne1
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?TechSoup
 
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptx
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptxSOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptx
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptxSyedNadeemGillANi
 
3.21.24 The Origins of Black Power.pptx
3.21.24  The Origins of Black Power.pptx3.21.24  The Origins of Black Power.pptx
3.21.24 The Origins of Black Power.pptxmary850239
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICESayali Powar
 
Optical Fibre and It's Applications.pptx
Optical Fibre and It's Applications.pptxOptical Fibre and It's Applications.pptx
Optical Fibre and It's Applications.pptxPurva Nikam
 
Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.EnglishCEIPdeSigeiro
 
Prescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxPrescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxraviapr7
 
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdf
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdfP4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdf
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdfYu Kanazawa / Osaka University
 
How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17Celine George
 
The Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsThe Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsEugene Lysak
 
Department of Health Compounder Question ‍Solution 2022.pdf
Department of Health Compounder Question ‍Solution 2022.pdfDepartment of Health Compounder Question ‍Solution 2022.pdf
Department of Health Compounder Question ‍Solution 2022.pdfMohonDas
 
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxAUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxiammrhaywood
 
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRADUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRATanmoy Mishra
 

Dernier (20)

The basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptxThe basics of sentences session 10pptx.pptx
The basics of sentences session 10pptx.pptx
 
Education and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptxEducation and training program in the hospital APR.pptx
Education and training program in the hospital APR.pptx
 
How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17How to Create a Toggle Button in Odoo 17
How to Create a Toggle Button in Odoo 17
 
Over the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptxOver the counter (OTC)- Sale, rational use.pptx
Over the counter (OTC)- Sale, rational use.pptx
 
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptxClinical Pharmacy  Introduction to Clinical Pharmacy, Concept of clinical pptx
Clinical Pharmacy Introduction to Clinical Pharmacy, Concept of clinical pptx
 
5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...5 charts on South Africa as a source country for international student recrui...
5 charts on South Africa as a source country for international student recrui...
 
What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?What is the Future of QuickBooks DeskTop?
What is the Future of QuickBooks DeskTop?
 
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptx
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptxSOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptx
SOLIDE WASTE in Cameroon,,,,,,,,,,,,,,,,,,,,,,,,,,,.pptx
 
3.21.24 The Origins of Black Power.pptx
3.21.24  The Origins of Black Power.pptx3.21.24  The Origins of Black Power.pptx
3.21.24 The Origins of Black Power.pptx
 
Quality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICEQuality Assurance_GOOD LABORATORY PRACTICE
Quality Assurance_GOOD LABORATORY PRACTICE
 
Optical Fibre and It's Applications.pptx
Optical Fibre and It's Applications.pptxOptical Fibre and It's Applications.pptx
Optical Fibre and It's Applications.pptx
 
Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.Easter in the USA presentation by Chloe.
Easter in the USA presentation by Chloe.
 
Prescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptxPrescribed medication order and communication skills.pptx
Prescribed medication order and communication skills.pptx
 
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdf
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdfP4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdf
P4C x ELT = P4ELT: Its Theoretical Background (Kanazawa, 2024 March).pdf
 
How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17How to Add a New Field in Existing Kanban View in Odoo 17
How to Add a New Field in Existing Kanban View in Odoo 17
 
The Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George WellsThe Stolen Bacillus by Herbert George Wells
The Stolen Bacillus by Herbert George Wells
 
Prelims of Kant get Marx 2.0: a general politics quiz
Prelims of Kant get Marx 2.0: a general politics quizPrelims of Kant get Marx 2.0: a general politics quiz
Prelims of Kant get Marx 2.0: a general politics quiz
 
Department of Health Compounder Question ‍Solution 2022.pdf
Department of Health Compounder Question ‍Solution 2022.pdfDepartment of Health Compounder Question ‍Solution 2022.pdf
Department of Health Compounder Question ‍Solution 2022.pdf
 
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptxAUDIENCE THEORY -- FANDOM -- JENKINS.pptx
AUDIENCE THEORY -- FANDOM -- JENKINS.pptx
 
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRADUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
DUST OF SNOW_BY ROBERT FROST_EDITED BY_ TANMOY MISHRA
 

2010 01 lecture SIG UM MFES 1 - Intro and testing

  • 1. Software Analysis and Testing Métodos Formais em Engenharia de Software January 2010 Arent Janszoon Ernststraat 595-H Joost Visser NL-1082 LD Amsterdam info@sig.nl www.sig.nl
  • 2. Me CV 2 I 112 • Technical University of Delft, Computer Science, MSc 1997 • University of Leiden, Philosophy, MA 1997 • CWI (Center for Mathematics and Informatics), PhD 2003 • Software Improvement Group, developer, consultant, etc, 2002-2003 • Universidade do Minho, Post-doc, 2004-2007 • Software Improvement Group, Head of Research, 2007-… Research • Grammars, traversal, transformation, generation • Functional programming, rewriting strategies • Software quality, metrics, reverse engineering Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 3. Software Improvement Group Company 3 I 112 • Spin-off from CWI in 2000, self-owned, independent • Management consultancy grounded in source code analysis • Innovative, strong academic background, award-winning, profitable Services • Software Risk Assessments (snapshot) and Software Monitoring (continuous) • Toolset enables to analyze source code in an automated manner • Experienced staff transforms analysis data into recommendations • We analyze over 50 systems annually • Focus on technical quality, primarily maintainability / evolvability Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 4. Services 4 I 112 DocGen • Automated generation of technical documentation • Reduce learning time, assist impact analysis, support migration, … Software Risk Assessment • In-depth investigation of software quality and risks • Answers specific research questions Software Monitoring • Continuous measurement, feedback, and decision support • Guard quality from start to finish Software Product Certification • Five levels of technical quality • Evaluation by SIG, certification by TÜV Informationstechnik Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 5. Who is using our services? 5 I 112 Financial Public Logistics IT Other Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 6. Structure of the lecture 6 I 112 • Introduction SIG • General overview of software analysis and testing • Testing • Patterns • Quality & metrics • Reverse engineering Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 7. Software Engineering 7 I 112 Create Change Analyze requirements analysis refactor, fix, patch understand, assess design, code, compile maintain, renovate evaluate, test configure, install evolve, update, improve measure, audit Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 8. Software Analysis (and Testing) 8 I 112 Analysis Static Dynamic Analysis Analysis syntax checking testing type checking debugging code metrics program spectra style checking instrumentation verification profiling reverse engineering benchmarking decompilation log analysis Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 9. Is testing un-cool? 9 I 112 Edsger Wybe Dijkstra (1930 - 2002) • “Program testing can be used to show the presence of bugs, but never to show their absence!” Notes On Structured Programming, 1970 • “Program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence.” The Humble Programmer, ACM Turing Award Lecture, 1972 Does not mean: “Don’t test!!” Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 10. Is testing un-cool? Industry 10 I 112 • Testers earn less then developers • Testing is “mechanical”, developing is “creative” • Testing is done with what remains of the budget in what remains of the time Academia • Testing is not part of the curriculum, or very minor part • Verification is superior to testing • Verification is more challenging than testing Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 11. Software Analysis. How much? 11 I 112 50 - 75% Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 12. Software Analysis. Enough? 12 I 112 $60 × 109 Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 13. Software Analysis. More? 13 I 112 high profile low frequency Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 14. Software Analysis Room for improvement? 14 I 112 1994 2004 Succeeded Failed 16% 18% Succeeded 29% Failed 31% Challenged 53% Challenged 53% Standish Group, “The CHAOS Report” Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 15. So 15 I 112 • Testing ⊂ Dynamic analysis ⊂ Analysis ⊂ S.E. • Analysis is a major and essential part of software engineering • Inadequate analysis costs billions ⇒ • More effective and more efficient methods are needed • Interest will keep growing in both industry and research Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 16. Structure of the lectures 16 I 112 Analysis Static Dynamic Analysis Analysis metrics patterns models testing Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 17. 17 I 112 TESTING Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 18. Testing Kinds Ways 18 I 112 • Conformance • Manual • Interoperability • Automated • Performance • Randomized • Functional • Independent • White-box • User • Black-box • Developer • Acceptance • Integration With • Unit • Plans • Component • Harness • System • Data • Smoke • Method • Stress • Frameworks Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 19. Testing V-model 19 I 112 V-model = No testing while waterfall-1 • waterfall programming! Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 20. Testing Eliminate waste Waste 20 I 112 • Coding and debugging go hand-in-hand • Coding effort materializes in the delivered program • Debugging effort? Evaporates! Automated tests • Small programs that capture debugging effort. • Invested effort is consolidated … • … and can be re-used without effort ad-infinitum Unit testing Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 21. What is unit testing? A unit test is … 21 I 112 • fully automated and repeatable • easy to write and maintain • non-intrusive TestCase • documenting • applies to the simplest piece of software Tool support • JUnit and friends public void testMyMethod { X x = …; Y y = myMethod(x); Y yy = …; assertEquals(“WRONG”,yy,y) } Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 22. Testing goals Unit testing has the following goals: 22 I 112 • Improve quality • Test as specification • Test as bug repellent • Test as defect localization • Help to understand • Test as documentation • Reduce risk • Test as a safety net • Remove fear of change Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 23. Observing unit-testing maturity in the wild (characterization of the population) Organization 23 I 112 • public, financial, logistics • under contract, in house, product software • with test departments, without test departments Architecture & Process • under architecture, using software factories • model driven, handwritten • open source frameworks, other frameworks • using use-cases/requirements • with blackbox tools, t-map Technology • information systems, embedded • webbased, desktop apps • java, c#, 4GL’s, legacy • latest trend: in-code asserts (java.spring) Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 24. Stage 1 No unit testing Observations: 24 I 112 • Very few organizations use unit testing • Also brand new OO systems without any unit tests • Small software shops and internal IT departments • In legacy environments: programmers describe in words what tests they have done. Symptoms: • Code is instable and error-prone • Lots of effort in post-development testing phases Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 25. Stage 1 No unit testing Excuses: 25 I 112 • “It is just additional code to maintain” • “The code is changing too much” • “We have a testing department” • “Testing can never prove the absence of errors” • “Testing is too expensive, the customer does not want to pay for it” • “We have black-box testing” Action • Provide standardized framework to lower threshold • Pay for unit tests as deliverable, not as effort Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 26. Stage 2 Unit test but no coverage measurement Observations 26 I 112 • Contract requires unit testing, not enforced • Revealed during conflicts • Unit testing receives low priority • Developers relapse into debugging practices without unit testing • Good initial intentions, bad execution • Large service providers Symptoms: • Some unit tests available • Excluded from daily build • No indication when unit testing is sufficient • Producing unit test is an option, not a requirement Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 27. Stage 2 Unit test but no coverage measurement Excuses: 27 I 112 • “There is no time, we are under pressure” • “We are constantly stopped to fix bugs” Actions • Start measuring coverage • Include coverage measurement into nightly build • Include coverage result reports into process Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 28. Stage 3 Coverage, not approaching 100% Observations 28 I 112 • Coverage is measured but gets stuck at 20%-50% • Ambitious teams, lacking experience • Code is not structured to be easily unit-testable Symptoms: • Complex code in GUI layer • Libraries in daily build, custom code not in daily build Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 29. Stage 3 Coverage, not approaching 100% Excuses 29 I 112 • “we test our libraries thoroughly, that affects more customers” Actions: • Refactor code to make it more easily testable • Teach advance unit testing patterns • Invest in set-up and mock-up Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 30. Stage 4 Approaching 100%, but no test quality Observations 30 I 112 • Formal compliance with contract • Gaming the metrics • Off-shored, certified, bureaucratic software factories Symptoms: • Empty tests • Tests without asserts. • Tests on high-level methods, rather than basic units • Need unit tests to test unit tests Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 31. Stage 4 Approaching 100%, but no test quality Anecdotes: 31 I 112 • Tell me how you measure me, and I tell you how I behave • We have generated our unit tests (at first this seems a stupid idea) Action: • Measure test quality • Number of asserts per unit test • Number of statements tested per unit test • Ratio of number of execution paths versus number of tests Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 32. Stage 5 Measuring test quality Enlightenment: 32 I 112 • Only one organization: a Swiss company • Measure: • Production code incorporated in tests • number of assert and fail statements • low complexity (not too many ifs) • The process • part of daily build • “stop the line process”, fix bugs first by adding more tests • happy path and exceptions • code first, test first, either way Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 33. Testing Intermediate conclusion Enormous potential for improvement: 33 I 112 • Do unit testing • Measure coverage • Measure test quality • May not help Ariane 5 • Does increase success ratio for “normal” projects Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 34. Randomized Testing (quickcheck) Randomized testing: 34 I 112 • QuickCheck: initially developed for Haskell • Parameterize tests in the test data • Property = parameterized test • Generate test data randomly • Test each property in 100 different ways each time Test generation -- | Range of inverse is domain. Model-driven testing prop_RngInvDom r = rng (inv r) == dom r Fault-injection where types = r::Rel Int Integer Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 35. Is testing un-cool? 35 I 112 Edsger Wybe Dijkstra (1930 - 2002) • “Program testing can be used to show the presence of bugs, but never to show their absence!” Martin Fowler • “Don’t let the fear that testing can’t catch all bugs stop you from writing the tests that will catch most bugs.” Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 36. Simple test metrics Line coverage 36 I 112 • Nr of test lines / nr of tested lines Decision coverage • Nr of test methods / Sum of McCabe complexity index Test granularity • Nr of test lines / nr of tests Test efficiency • Decision coverage / line coverage Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.
  • 37. Testing Exercises Write unit tests 37 I 112 • Using JUnit • E.g. for one of your own projects Measure coverage • E.g. using Emma plug-in for Eclipse Randomize one of your unit tests • Turn test into property with extract method refactoring • Write generator for test data • Instantiate property 100 times with random test data • Solution to j.visser@sig.eu Software Analysis and Testing, MFES Universidade do Minho by Joost Visser, Software Improvement Group © 2010.