SlideShare une entreprise Scribd logo
1  sur  34
Télécharger pour lire hors ligne
Video automation testing at Skype


         Pierre Gronlier - pierre.gronlier@skype.net

Video Software Development Engineer in Test - Microsoft Skype division



                        April 2012 - Kiev
1   The Video Library
2   Continuous Integration
      Building
      Testing
      Feedback
3   Unit, Component, System testing
      Some wrappers for testing.
      Test Driven Development
4   Cross-platform testing
      CI team
      Plugin mechanisms
5   NFR
      Denition
      KPIs
      Increase visibility
6   Conclusion
The Video Library
What is Skype made of ?


                                     UI

                                  Network

                    Video          Audio      Messaging



                            Figure: Inside Skype
What is Skype made of ?


                                                   UI

                                                Network

                                 Video           Audio         Messaging



                                         Figure: Inside Skype


                              Video Codec      Streaming         ToolBox
                                               Platforms
                              Apple, Android, Windows, Linux, Embedded, ...


                                 Figure: Inside the Video Library

  Platforms contains specic code like capturing, rendering methods.
Continuous Integration
Continuous integration means :
    building continuously.

    testing continuously.

    having an immediate feedback.
Quickbuild

  http://www.pmease.com/features/




                                Figure: Quickbuild
Quickbuild
     There is, only for the Video Library, around 20 dierent build congurations for dierent
     platforms and compilation modes.
         release/debug
         internal/external
         stable/experimental
         ...
     We have a farm of building computers.
     To enable compilation and maintenance across platforms, Makefile is used for compiling
     and farm agents are in Java




                                       Figure: HeatMap
Cross branches builds




   Example

  Network :             Video :              Codec :
      trunk/                trunk/               trunk/
      branches/             branches/            branches/
          network-69              video-42             codec-23
          network-68              video-41             codec-22
          ...                     ...                  ...
Cross branches builds


   Example

  Network :                        Video :                            Codec :
      trunk/                            trunk/                            trunk/
      branches/                         branches/                         branches/
          network-69 *                       video-42                           codec-23 *
          network-68                         video-41 *                         codec-22
          ...                                ...                                ...
       To enable two dierent dependent teams to develop new features without becoming
       incompatible, we compile our code with the latest stable release of the dependencies.

       In addition to trunk source code, we build our latest Long Term Support branch (*) every
       time there is a backport of a x.
Cross branches builds

   Example

  Network :                      Video :                        Codec :
      trunk/                         trunk/                         trunk/
      branches/                      branches/                      branches/
          network-69 *                     video-42                        codec-23 *
          network-68                       video-41 *                      codec-22
          ...                              ...                             ...

                   Mode                    Network      Video    Codec
                   Video stable              ∅       video-41   codec-23
                   Video release             ∅         trunk    codec-23
                   Video experimental        ∅         trunk      trunk
                   Network release         trunk     video-41   codec-23
                   Network experimental    trunk       trunk      trunk
CI as a daily tool




   Continuous integration means that :
     1 every 10 mins, a script checks for new commits on video trunk/ or the branches/.

     2 once a build for a platform is done successfully, it triggers a list of short tests. Every test

       lasts around 30 seconds.
     3 at night, a list of longer tests is executed.

     4 for every test execution, a report is generated in a database and the results are aggregated

       on a web page for Devs and QEs
CI as a daily tool




                     Figure: Test results
The importance of visual feedback




                   Figure: TVs with build/test feedback

                           Make it visible ! !
Unit, Component, System testing
Who writes and maintains the tests ?
       Writing tests is writing code.
       When you automate testing, QE are software developers in test.
  The closer and deeper you get into the production source code, more probably it will be a
  developer test.
                                              UI
                                                                         Python
                                           Network

                            Video            Audio         Messaging



                                          Figure: Inside Skype

                                                                         Lua, C#
                         Video Codec      Streaming         ToolBox
                                          Platforms                     Lua
                         Apple, Android, Windows, Linux, Embedded, ...



                                    Figure: Inside the Video Library
QE and Devs together




  1   Don't wait for developers to write your
      tests.

  2   Dene the tests when you dene the
      Acceptance Criteria of your PBI.

  3   Evaluate the value of your tests (e.g. code
      coverage).

  4   KISS : Keep it Stupid Short and Simple.



                                                    Figure: Test plan
Cross-platform testing
Requirements
  We want to have those features :
     run our tests on dierent platforms
     run our tests with dierent builds
     retrieve the results of our tests and analyze it
     save the result of the analysis
     output a report, trigger alarms

  The cross-platform CI team can provide :
      a pool of devices, platform and capture devices.
      access to various builds.
      provide uniform alarming systems (chat, email, sms)
      a database.
      a storage space.
              It is only a matter of contract denition between you and the CI team
How to conceive a modular testing framework ?
           Targets:
           - tablets, mobile, notebook w/ and w/o hardware encoding camera, desktop
           - Windows (desktop + mobile), Linux, Mac (desktop + mobile), Android

                                              +/-




                Full logs

                                     Insert/Update entry

                                                                                      Frontend
                                                               DataBase                Server




            Parsing
            Server                                                                    Web Rendering
                                 Reduced logs


                                                            Storage Server




                                           Figure: Framework
How to conceive a modular testing framework ?
           Targets:
           - tablets, mobile, notebook w/ and w/o hardware encoding camera, desktop
           - Windows (desktop + mobile), Linux, Mac (desktop + mobile), Android

                                              +/-




                Full logs

                                     Insert/Update entry

                                                                                      Frontend
                                                               DataBase                Server




            Parsing
            Server                                                                    Web Rendering
                                 Reduced logs


                                                            Storage Server




                                           Figure: Framework
NFR
What is non-functional ?




  Functional vs Non-Functional
                           the video works = we see something
                                            vs
                  the video has a good quality = we enjoy our video call
Key performance indicators

  list of kpis

      resolution and frame rate
      bitrate
      dropped frames and freeze durations
      frame-quality
      ...

  list of usecases
      for every codec
      for every media protocol version
      1-to-1 call and Group Video Calling
      software encoding vs hardware encoding
      for dierent network conditions
Pass/Fail vs Score



                                    Video Call A        Network Emulation        Video Call B
                               with controlled inputs                       with analyzed outputs




                 x                    x
    1. VGA = 640 480, QVGA = 320 240, QQVGA = 160 120               x
    2. Image Quality measurement algorithms
Pass/Fail vs Score



                                     Video Call A        Network Emulation        Video Call B
                                with controlled inputs                       with analyzed outputs



  Example   :
                     KPI                   Functional                                   Non-functional
                                             pass/fail                                   0% → 100%
                  resolution                  = 0x0                                     max = VGA 1
                  framerate                    =0                                        max = 15fps
                    bitrate       in the range of [20..5000]kb                         350kbps ± 10 %
                frame-quality              frame exist                               PSNR or SSIM 2 score
                       .
                       .                         .
                                                 .                                            .
                                                                                              .
                       .                         .                                            .
    Everything is automated using stats and feedback values from the Video Library.

                 x                     x
    1. VGA = 640 480, QVGA = 320 240, QQVGA = 160 120                x
    2. Image Quality measurement algorithms
How to evaluate the best available quality for a call ?

  The best quality of a call is given by :
                              optimal settings   = gcd(sender , receiver )

  with
             sender     = gcd (max (Encoding     power   ) , max (Network) , max (Camera))
             receiver   = gcd (max (Decoding     power   ) , max (Network) , max (Screen))




   (gcd = greatest common divisor)
How to evaluate the best available quality for a call ?

  The best quality of a call is given by :
                              optimal settings   = gcd(sender , receiver )

  with
             sender     = gcd (max (Encoding     power   ) , max (Network) , max (Camera))
             receiver   = gcd (max (Decoding     power   ) , max (Network) , max (Screen))
   where, with some simplications,
              Encoding power      = f1 (CPU power, Power supply mode, Codec        perf.)
              Network             = f2 (Bandwidth, RTT, Relay/P2P)
              Camera              = f3 (Resolution, Framerate)
              Decoding power      = f4 (CPU power, Power supply mode, Codec        perf.)
              Screen              = f5 (Resolution)

   (gcd = greatest common divisor)
Compare across revisions / branches
Compare across revisions / branches
Conclusion
Summary



   1   Quick feedback between development and testing.

   2   Devs and QE in the same team.

   3   Collocation helps a lot !

   4   Don't over-complicate your tests/frameworks.

   5   Measure the eciency/value of your tests.
Questions

 1   The Video Library
 2   Continuous Integration
       Building
       Testing
       Feedback
 3   Unit, Component, System testing
       Some wrappers for testing.
       Test Driven Development         Äÿêóþ çà óâàãó !
 4   Cross-platform testing
       CI team
       Plugin mechanisms                Çàïèòàííÿ ?
 5   NFR
       Denition
       KPIs
       Increase visibility
 6   Conclusion

Contenu connexe

Tendances

안드로이드 MediaPlayer & VideoView
안드로이드 MediaPlayer & VideoView안드로이드 MediaPlayer & VideoView
안드로이드 MediaPlayer & VideoViewEunjoo Im
 
Inside Android's Dalvik VM - NEJUG Nov 2011
Inside Android's Dalvik VM - NEJUG Nov 2011Inside Android's Dalvik VM - NEJUG Nov 2011
Inside Android's Dalvik VM - NEJUG Nov 2011Doug Hawkins
 
Embedded Linux Multimedia
Embedded Linux MultimediaEmbedded Linux Multimedia
Embedded Linux MultimediaCaglar Dursun
 
Linux on System z Update: Current & Future Linux on System z Technology
Linux on System z Update: Current & Future Linux on System z TechnologyLinux on System z Update: Current & Future Linux on System z Technology
Linux on System z Update: Current & Future Linux on System z TechnologyIBM India Smarter Computing
 
Learning, Analyzing and Protecting Android with TOMOYO Linux (JLS2009)
Learning, Analyzing and Protecting Android with TOMOYO Linux (JLS2009)Learning, Analyzing and Protecting Android with TOMOYO Linux (JLS2009)
Learning, Analyzing and Protecting Android with TOMOYO Linux (JLS2009)Toshiharu Harada, Ph.D
 
Comp tia a+_session_11
Comp tia a+_session_11Comp tia a+_session_11
Comp tia a+_session_11Niit Care
 
Comp tia a+_session_03
Comp tia a+_session_03Comp tia a+_session_03
Comp tia a+_session_03Niit Care
 
Raju_Biswas_Resume
Raju_Biswas_ResumeRaju_Biswas_Resume
Raju_Biswas_ResumeRaju Biswas
 
EclipseCon 2011: Deciphering the CDT debugger alphabet soup
EclipseCon 2011: Deciphering the CDT debugger alphabet soupEclipseCon 2011: Deciphering the CDT debugger alphabet soup
EclipseCon 2011: Deciphering the CDT debugger alphabet soupBruce Griffith
 
Comp tia a+_session_13
Comp tia a+_session_13Comp tia a+_session_13
Comp tia a+_session_13Niit Care
 
Comp tia a+_session_10
Comp tia a+_session_10Comp tia a+_session_10
Comp tia a+_session_10Niit Care
 
Open Source Licenses and Tools
Open Source Licenses and ToolsOpen Source Licenses and Tools
Open Source Licenses and Toolsg2ix
 

Tendances (20)

안드로이드 MediaPlayer & VideoView
안드로이드 MediaPlayer & VideoView안드로이드 MediaPlayer & VideoView
안드로이드 MediaPlayer & VideoView
 
Improve Android System Component Performance
Improve Android System Component PerformanceImprove Android System Component Performance
Improve Android System Component Performance
 
Inside Android's Dalvik VM - NEJUG Nov 2011
Inside Android's Dalvik VM - NEJUG Nov 2011Inside Android's Dalvik VM - NEJUG Nov 2011
Inside Android's Dalvik VM - NEJUG Nov 2011
 
Embedded Linux Multimedia
Embedded Linux MultimediaEmbedded Linux Multimedia
Embedded Linux Multimedia
 
Linux on System z Update: Current & Future Linux on System z Technology
Linux on System z Update: Current & Future Linux on System z TechnologyLinux on System z Update: Current & Future Linux on System z Technology
Linux on System z Update: Current & Future Linux on System z Technology
 
TOMOYO Linux on Android
TOMOYO Linux on AndroidTOMOYO Linux on Android
TOMOYO Linux on Android
 
Learning, Analyzing and Protecting Android with TOMOYO Linux (JLS2009)
Learning, Analyzing and Protecting Android with TOMOYO Linux (JLS2009)Learning, Analyzing and Protecting Android with TOMOYO Linux (JLS2009)
Learning, Analyzing and Protecting Android with TOMOYO Linux (JLS2009)
 
ARM
ARMARM
ARM
 
Divya_Resume
Divya_ResumeDivya_Resume
Divya_Resume
 
Performance Analysis of Various Video Compression Techniques
Performance Analysis of Various Video Compression TechniquesPerformance Analysis of Various Video Compression Techniques
Performance Analysis of Various Video Compression Techniques
 
Comp tia a+_session_11
Comp tia a+_session_11Comp tia a+_session_11
Comp tia a+_session_11
 
Comp tia a+_session_03
Comp tia a+_session_03Comp tia a+_session_03
Comp tia a+_session_03
 
Harsh gaurav
Harsh gauravHarsh gaurav
Harsh gaurav
 
Raju_Biswas_Resume
Raju_Biswas_ResumeRaju_Biswas_Resume
Raju_Biswas_Resume
 
EclipseCon 2011: Deciphering the CDT debugger alphabet soup
EclipseCon 2011: Deciphering the CDT debugger alphabet soupEclipseCon 2011: Deciphering the CDT debugger alphabet soup
EclipseCon 2011: Deciphering the CDT debugger alphabet soup
 
Comp tia a+_session_13
Comp tia a+_session_13Comp tia a+_session_13
Comp tia a+_session_13
 
Comp tia a+_session_10
Comp tia a+_session_10Comp tia a+_session_10
Comp tia a+_session_10
 
Open Source Licenses and Tools
Open Source Licenses and ToolsOpen Source Licenses and Tools
Open Source Licenses and Tools
 
Android cameraoverview
Android cameraoverviewAndroid cameraoverview
Android cameraoverview
 
4
44
4
 

Similaire à Skype testing overview

Srikanth_PILLI_CV_latest
Srikanth_PILLI_CV_latestSrikanth_PILLI_CV_latest
Srikanth_PILLI_CV_latestSrikanth Pilli
 
20110917 saitama iphone_dev
20110917 saitama iphone_dev20110917 saitama iphone_dev
20110917 saitama iphone_devKaoru NAKAMURA
 
Code Factory avec GitLab CI et Rancher
Code Factory avec GitLab CI et RancherCode Factory avec GitLab CI et Rancher
Code Factory avec GitLab CI et RancherSUSE
 
Code Factory avec GitLab CI et Rancher
Code Factory avec GitLab CI et RancherCode Factory avec GitLab CI et Rancher
Code Factory avec GitLab CI et RancherSUSE
 
Richard Bronson Full Resume 2015
Richard Bronson Full Resume 2015Richard Bronson Full Resume 2015
Richard Bronson Full Resume 2015Richard Bronson
 
LlinuxKit security, Security Scanning and Notary
LlinuxKit security, Security Scanning and NotaryLlinuxKit security, Security Scanning and Notary
LlinuxKit security, Security Scanning and NotaryDocker, Inc.
 
IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...
IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...
IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...Christopher Diamantopoulos
 
Making Networking Apps Scream on Windows with DPDK
Making Networking Apps Scream on Windows with DPDKMaking Networking Apps Scream on Windows with DPDK
Making Networking Apps Scream on Windows with DPDKMichelle Holley
 
DCC Labs Company Presentation
DCC Labs Company PresentationDCC Labs Company Presentation
DCC Labs Company PresentationDCC Labs
 
Embedded system-Introduction to development cycle and development tool
Embedded system-Introduction to development cycle and development  toolEmbedded system-Introduction to development cycle and development  tool
Embedded system-Introduction to development cycle and development toolPantech ProLabs India Pvt Ltd
 
Build, Publish, Deploy and Test Docker images and containers with Jenkins Wor...
Build, Publish, Deploy and Test Docker images and containers with Jenkins Wor...Build, Publish, Deploy and Test Docker images and containers with Jenkins Wor...
Build, Publish, Deploy and Test Docker images and containers with Jenkins Wor...Docker, Inc.
 
Docker adventures in Continuous Delivery - Alex Vranceanu
Docker adventures in Continuous Delivery - Alex VranceanuDocker adventures in Continuous Delivery - Alex Vranceanu
Docker adventures in Continuous Delivery - Alex VranceanuITCamp
 
Kahuna Systems : Product Engineering Services
Kahuna Systems : Product Engineering ServicesKahuna Systems : Product Engineering Services
Kahuna Systems : Product Engineering Serviceskahunasystems
 
Announcing AWS CodeBuild - January 2017 Online Teck Talks
Announcing AWS CodeBuild - January 2017 Online Teck TalksAnnouncing AWS CodeBuild - January 2017 Online Teck Talks
Announcing AWS CodeBuild - January 2017 Online Teck TalksAmazon Web Services
 
Agile Bodensee - Testautomation & Continuous Delivery Workshop
Agile Bodensee - Testautomation & Continuous Delivery WorkshopAgile Bodensee - Testautomation & Continuous Delivery Workshop
Agile Bodensee - Testautomation & Continuous Delivery WorkshopMichael Palotas
 
Automated Build using teamcity
Automated Build using teamcityAutomated Build using teamcity
Automated Build using teamcityMd Jawed
 
Eclipse Che - A Revolutionary IDE for Distributed & Mainframe Development
Eclipse Che - A Revolutionary IDE for Distributed & Mainframe DevelopmentEclipse Che - A Revolutionary IDE for Distributed & Mainframe Development
Eclipse Che - A Revolutionary IDE for Distributed & Mainframe DevelopmentDevOps.com
 

Similaire à Skype testing overview (20)

Arm
ArmArm
Arm
 
Srikanth_PILLI_CV_latest
Srikanth_PILLI_CV_latestSrikanth_PILLI_CV_latest
Srikanth_PILLI_CV_latest
 
20110917 saitama iphone_dev
20110917 saitama iphone_dev20110917 saitama iphone_dev
20110917 saitama iphone_dev
 
Code Factory avec GitLab CI et Rancher
Code Factory avec GitLab CI et RancherCode Factory avec GitLab CI et Rancher
Code Factory avec GitLab CI et Rancher
 
Code Factory avec GitLab CI et Rancher
Code Factory avec GitLab CI et RancherCode Factory avec GitLab CI et Rancher
Code Factory avec GitLab CI et Rancher
 
Video Drivers
Video DriversVideo Drivers
Video Drivers
 
Richard Bronson Full Resume 2015
Richard Bronson Full Resume 2015Richard Bronson Full Resume 2015
Richard Bronson Full Resume 2015
 
LlinuxKit security, Security Scanning and Notary
LlinuxKit security, Security Scanning and NotaryLlinuxKit security, Security Scanning and Notary
LlinuxKit security, Security Scanning and Notary
 
IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...
IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...
IMAGE CAPTURE, PROCESSING AND TRANSFER VIA ETHERNET UNDER CONTROL OF MATLAB G...
 
Making Networking Apps Scream on Windows with DPDK
Making Networking Apps Scream on Windows with DPDKMaking Networking Apps Scream on Windows with DPDK
Making Networking Apps Scream on Windows with DPDK
 
DCC Labs Company Presentation
DCC Labs Company PresentationDCC Labs Company Presentation
DCC Labs Company Presentation
 
Embedded system-Introduction to development cycle and development tool
Embedded system-Introduction to development cycle and development  toolEmbedded system-Introduction to development cycle and development  tool
Embedded system-Introduction to development cycle and development tool
 
Build, Publish, Deploy and Test Docker images and containers with Jenkins Wor...
Build, Publish, Deploy and Test Docker images and containers with Jenkins Wor...Build, Publish, Deploy and Test Docker images and containers with Jenkins Wor...
Build, Publish, Deploy and Test Docker images and containers with Jenkins Wor...
 
Deep Dive into WinRT
Deep Dive into WinRTDeep Dive into WinRT
Deep Dive into WinRT
 
Docker adventures in Continuous Delivery - Alex Vranceanu
Docker adventures in Continuous Delivery - Alex VranceanuDocker adventures in Continuous Delivery - Alex Vranceanu
Docker adventures in Continuous Delivery - Alex Vranceanu
 
Kahuna Systems : Product Engineering Services
Kahuna Systems : Product Engineering ServicesKahuna Systems : Product Engineering Services
Kahuna Systems : Product Engineering Services
 
Announcing AWS CodeBuild - January 2017 Online Teck Talks
Announcing AWS CodeBuild - January 2017 Online Teck TalksAnnouncing AWS CodeBuild - January 2017 Online Teck Talks
Announcing AWS CodeBuild - January 2017 Online Teck Talks
 
Agile Bodensee - Testautomation & Continuous Delivery Workshop
Agile Bodensee - Testautomation & Continuous Delivery WorkshopAgile Bodensee - Testautomation & Continuous Delivery Workshop
Agile Bodensee - Testautomation & Continuous Delivery Workshop
 
Automated Build using teamcity
Automated Build using teamcityAutomated Build using teamcity
Automated Build using teamcity
 
Eclipse Che - A Revolutionary IDE for Distributed & Mainframe Development
Eclipse Che - A Revolutionary IDE for Distributed & Mainframe DevelopmentEclipse Che - A Revolutionary IDE for Distributed & Mainframe Development
Eclipse Che - A Revolutionary IDE for Distributed & Mainframe Development
 

Plus de QA Club Kiev

QA Club Kiev #20. Mobile Testing. Tips & Tricks
QA Club Kiev #20. Mobile Testing. Tips & TricksQA Club Kiev #20. Mobile Testing. Tips & Tricks
QA Club Kiev #20. Mobile Testing. Tips & TricksQA Club Kiev
 
QA Club Kiev #20. Making life easier with Docker
QA Club Kiev #20. Making life easier with DockerQA Club Kiev #20. Making life easier with Docker
QA Club Kiev #20. Making life easier with DockerQA Club Kiev
 
QA Club Kiev #19 - ISTQB to be or not to be
QA Club Kiev #19 - ISTQB to be or not to beQA Club Kiev #19 - ISTQB to be or not to be
QA Club Kiev #19 - ISTQB to be or not to beQA Club Kiev
 
QA Club Kiev #18 - Test Management in Google Sheets
QA Club Kiev #18 - Test Management in Google SheetsQA Club Kiev #18 - Test Management in Google Sheets
QA Club Kiev #18 - Test Management in Google SheetsQA Club Kiev
 
QA Club Kiev 18 - Test Management and Approaches
QA Club Kiev 18 - Test Management and ApproachesQA Club Kiev 18 - Test Management and Approaches
QA Club Kiev 18 - Test Management and ApproachesQA Club Kiev
 
Performance Testing
Performance TestingPerformance Testing
Performance TestingQA Club Kiev
 
Whay QA- engineers should know how to code
Whay QA- engineers should know how to codeWhay QA- engineers should know how to code
Whay QA- engineers should know how to codeQA Club Kiev
 
Mobile Testing. What to do?
Mobile Testing. What to do?Mobile Testing. What to do?
Mobile Testing. What to do?QA Club Kiev
 
Мифы Автоматизации
Мифы АвтоматизацииМифы Автоматизации
Мифы АвтоматизацииQA Club Kiev
 
QA Club Kiev #17 Measuring quality by Volodymyr Prymakov
QA Club Kiev #17 Measuring quality by Volodymyr PrymakovQA Club Kiev #17 Measuring quality by Volodymyr Prymakov
QA Club Kiev #17 Measuring quality by Volodymyr PrymakovQA Club Kiev
 
QA Club Kiev #17 QA Challenge by Oleksandr Maidaniuk
QA Club Kiev #17 QA Challenge by Oleksandr MaidaniukQA Club Kiev #17 QA Challenge by Oleksandr Maidaniuk
QA Club Kiev #17 QA Challenge by Oleksandr MaidaniukQA Club Kiev
 
QA Club Kiev #16: BA in IT
QA Club Kiev #16: BA in ITQA Club Kiev #16: BA in IT
QA Club Kiev #16: BA in ITQA Club Kiev
 
Agile performance testing
Agile performance testingAgile performance testing
Agile performance testingQA Club Kiev
 
QAClubKiev Performance-Structure
QAClubKiev Performance-StructureQAClubKiev Performance-Structure
QAClubKiev Performance-StructureQA Club Kiev
 
QA Club Kiev #13 Performance Testing - introduction
QA Club Kiev #13  Performance Testing - introductionQA Club Kiev #13  Performance Testing - introduction
QA Club Kiev #13 Performance Testing - introductionQA Club Kiev
 
Qa club kiev #12 istqb сertification
Qa club kiev #12  istqb сertification Qa club kiev #12  istqb сertification
Qa club kiev #12 istqb сertification QA Club Kiev
 
ISTQB Certification
ISTQB CertificationISTQB Certification
ISTQB CertificationQA Club Kiev
 
Test management in scrum
Test management in scrumTest management in scrum
Test management in scrumQA Club Kiev
 
Qa club kiev #11 test documentation - introduction
Qa club kiev #11  test documentation - introductionQa club kiev #11  test documentation - introduction
Qa club kiev #11 test documentation - introductionQA Club Kiev
 
Agile testing - introduction
Agile testing - introductionAgile testing - introduction
Agile testing - introductionQA Club Kiev
 

Plus de QA Club Kiev (20)

QA Club Kiev #20. Mobile Testing. Tips & Tricks
QA Club Kiev #20. Mobile Testing. Tips & TricksQA Club Kiev #20. Mobile Testing. Tips & Tricks
QA Club Kiev #20. Mobile Testing. Tips & Tricks
 
QA Club Kiev #20. Making life easier with Docker
QA Club Kiev #20. Making life easier with DockerQA Club Kiev #20. Making life easier with Docker
QA Club Kiev #20. Making life easier with Docker
 
QA Club Kiev #19 - ISTQB to be or not to be
QA Club Kiev #19 - ISTQB to be or not to beQA Club Kiev #19 - ISTQB to be or not to be
QA Club Kiev #19 - ISTQB to be or not to be
 
QA Club Kiev #18 - Test Management in Google Sheets
QA Club Kiev #18 - Test Management in Google SheetsQA Club Kiev #18 - Test Management in Google Sheets
QA Club Kiev #18 - Test Management in Google Sheets
 
QA Club Kiev 18 - Test Management and Approaches
QA Club Kiev 18 - Test Management and ApproachesQA Club Kiev 18 - Test Management and Approaches
QA Club Kiev 18 - Test Management and Approaches
 
Performance Testing
Performance TestingPerformance Testing
Performance Testing
 
Whay QA- engineers should know how to code
Whay QA- engineers should know how to codeWhay QA- engineers should know how to code
Whay QA- engineers should know how to code
 
Mobile Testing. What to do?
Mobile Testing. What to do?Mobile Testing. What to do?
Mobile Testing. What to do?
 
Мифы Автоматизации
Мифы АвтоматизацииМифы Автоматизации
Мифы Автоматизации
 
QA Club Kiev #17 Measuring quality by Volodymyr Prymakov
QA Club Kiev #17 Measuring quality by Volodymyr PrymakovQA Club Kiev #17 Measuring quality by Volodymyr Prymakov
QA Club Kiev #17 Measuring quality by Volodymyr Prymakov
 
QA Club Kiev #17 QA Challenge by Oleksandr Maidaniuk
QA Club Kiev #17 QA Challenge by Oleksandr MaidaniukQA Club Kiev #17 QA Challenge by Oleksandr Maidaniuk
QA Club Kiev #17 QA Challenge by Oleksandr Maidaniuk
 
QA Club Kiev #16: BA in IT
QA Club Kiev #16: BA in ITQA Club Kiev #16: BA in IT
QA Club Kiev #16: BA in IT
 
Agile performance testing
Agile performance testingAgile performance testing
Agile performance testing
 
QAClubKiev Performance-Structure
QAClubKiev Performance-StructureQAClubKiev Performance-Structure
QAClubKiev Performance-Structure
 
QA Club Kiev #13 Performance Testing - introduction
QA Club Kiev #13  Performance Testing - introductionQA Club Kiev #13  Performance Testing - introduction
QA Club Kiev #13 Performance Testing - introduction
 
Qa club kiev #12 istqb сertification
Qa club kiev #12  istqb сertification Qa club kiev #12  istqb сertification
Qa club kiev #12 istqb сertification
 
ISTQB Certification
ISTQB CertificationISTQB Certification
ISTQB Certification
 
Test management in scrum
Test management in scrumTest management in scrum
Test management in scrum
 
Qa club kiev #11 test documentation - introduction
Qa club kiev #11  test documentation - introductionQa club kiev #11  test documentation - introduction
Qa club kiev #11 test documentation - introduction
 
Agile testing - introduction
Agile testing - introductionAgile testing - introduction
Agile testing - introduction
 

Dernier

Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfDaniel Santiago Silva Capera
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 

Dernier (20)

Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdfIaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
IaC & GitOps in a Nutshell - a FridayInANuthshell Episode.pdf
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 

Skype testing overview

  • 1. Video automation testing at Skype Pierre Gronlier - pierre.gronlier@skype.net Video Software Development Engineer in Test - Microsoft Skype division April 2012 - Kiev
  • 2. 1 The Video Library 2 Continuous Integration Building Testing Feedback 3 Unit, Component, System testing Some wrappers for testing. Test Driven Development 4 Cross-platform testing CI team Plugin mechanisms 5 NFR Denition KPIs Increase visibility 6 Conclusion
  • 4. What is Skype made of ? UI Network Video Audio Messaging Figure: Inside Skype
  • 5. What is Skype made of ? UI Network Video Audio Messaging Figure: Inside Skype Video Codec Streaming ToolBox Platforms Apple, Android, Windows, Linux, Embedded, ... Figure: Inside the Video Library Platforms contains specic code like capturing, rendering methods.
  • 7. Continuous integration means : building continuously. testing continuously. having an immediate feedback.
  • 9. Quickbuild There is, only for the Video Library, around 20 dierent build congurations for dierent platforms and compilation modes. release/debug internal/external stable/experimental ... We have a farm of building computers. To enable compilation and maintenance across platforms, Makefile is used for compiling and farm agents are in Java Figure: HeatMap
  • 10. Cross branches builds Example Network : Video : Codec : trunk/ trunk/ trunk/ branches/ branches/ branches/ network-69 video-42 codec-23 network-68 video-41 codec-22 ... ... ...
  • 11. Cross branches builds Example Network : Video : Codec : trunk/ trunk/ trunk/ branches/ branches/ branches/ network-69 * video-42 codec-23 * network-68 video-41 * codec-22 ... ... ... To enable two dierent dependent teams to develop new features without becoming incompatible, we compile our code with the latest stable release of the dependencies. In addition to trunk source code, we build our latest Long Term Support branch (*) every time there is a backport of a x.
  • 12. Cross branches builds Example Network : Video : Codec : trunk/ trunk/ trunk/ branches/ branches/ branches/ network-69 * video-42 codec-23 * network-68 video-41 * codec-22 ... ... ... Mode Network Video Codec Video stable ∅ video-41 codec-23 Video release ∅ trunk codec-23 Video experimental ∅ trunk trunk Network release trunk video-41 codec-23 Network experimental trunk trunk trunk
  • 13. CI as a daily tool Continuous integration means that : 1 every 10 mins, a script checks for new commits on video trunk/ or the branches/. 2 once a build for a platform is done successfully, it triggers a list of short tests. Every test lasts around 30 seconds. 3 at night, a list of longer tests is executed. 4 for every test execution, a report is generated in a database and the results are aggregated on a web page for Devs and QEs
  • 14. CI as a daily tool Figure: Test results
  • 15. The importance of visual feedback Figure: TVs with build/test feedback Make it visible ! !
  • 17. Who writes and maintains the tests ? Writing tests is writing code. When you automate testing, QE are software developers in test. The closer and deeper you get into the production source code, more probably it will be a developer test. UI Python Network Video Audio Messaging Figure: Inside Skype Lua, C# Video Codec Streaming ToolBox Platforms Lua Apple, Android, Windows, Linux, Embedded, ... Figure: Inside the Video Library
  • 18. QE and Devs together 1 Don't wait for developers to write your tests. 2 Dene the tests when you dene the Acceptance Criteria of your PBI. 3 Evaluate the value of your tests (e.g. code coverage). 4 KISS : Keep it Stupid Short and Simple. Figure: Test plan
  • 20. Requirements We want to have those features : run our tests on dierent platforms run our tests with dierent builds retrieve the results of our tests and analyze it save the result of the analysis output a report, trigger alarms The cross-platform CI team can provide : a pool of devices, platform and capture devices. access to various builds. provide uniform alarming systems (chat, email, sms) a database. a storage space. It is only a matter of contract denition between you and the CI team
  • 21. How to conceive a modular testing framework ? Targets: - tablets, mobile, notebook w/ and w/o hardware encoding camera, desktop - Windows (desktop + mobile), Linux, Mac (desktop + mobile), Android +/- Full logs Insert/Update entry Frontend DataBase Server Parsing Server Web Rendering Reduced logs Storage Server Figure: Framework
  • 22. How to conceive a modular testing framework ? Targets: - tablets, mobile, notebook w/ and w/o hardware encoding camera, desktop - Windows (desktop + mobile), Linux, Mac (desktop + mobile), Android +/- Full logs Insert/Update entry Frontend DataBase Server Parsing Server Web Rendering Reduced logs Storage Server Figure: Framework
  • 23. NFR
  • 24. What is non-functional ? Functional vs Non-Functional the video works = we see something vs the video has a good quality = we enjoy our video call
  • 25. Key performance indicators list of kpis resolution and frame rate bitrate dropped frames and freeze durations frame-quality ... list of usecases for every codec for every media protocol version 1-to-1 call and Group Video Calling software encoding vs hardware encoding for dierent network conditions
  • 26. Pass/Fail vs Score Video Call A Network Emulation Video Call B with controlled inputs with analyzed outputs x x 1. VGA = 640 480, QVGA = 320 240, QQVGA = 160 120 x 2. Image Quality measurement algorithms
  • 27. Pass/Fail vs Score Video Call A Network Emulation Video Call B with controlled inputs with analyzed outputs Example : KPI Functional Non-functional pass/fail 0% → 100% resolution = 0x0 max = VGA 1 framerate =0 max = 15fps bitrate in the range of [20..5000]kb 350kbps ± 10 % frame-quality frame exist PSNR or SSIM 2 score . . . . . . . . . Everything is automated using stats and feedback values from the Video Library. x x 1. VGA = 640 480, QVGA = 320 240, QQVGA = 160 120 x 2. Image Quality measurement algorithms
  • 28. How to evaluate the best available quality for a call ? The best quality of a call is given by : optimal settings = gcd(sender , receiver ) with sender = gcd (max (Encoding power ) , max (Network) , max (Camera)) receiver = gcd (max (Decoding power ) , max (Network) , max (Screen)) (gcd = greatest common divisor)
  • 29. How to evaluate the best available quality for a call ? The best quality of a call is given by : optimal settings = gcd(sender , receiver ) with sender = gcd (max (Encoding power ) , max (Network) , max (Camera)) receiver = gcd (max (Decoding power ) , max (Network) , max (Screen)) where, with some simplications, Encoding power = f1 (CPU power, Power supply mode, Codec perf.) Network = f2 (Bandwidth, RTT, Relay/P2P) Camera = f3 (Resolution, Framerate) Decoding power = f4 (CPU power, Power supply mode, Codec perf.) Screen = f5 (Resolution) (gcd = greatest common divisor)
  • 33. Summary 1 Quick feedback between development and testing. 2 Devs and QE in the same team. 3 Collocation helps a lot ! 4 Don't over-complicate your tests/frameworks. 5 Measure the eciency/value of your tests.
  • 34. Questions 1 The Video Library 2 Continuous Integration Building Testing Feedback 3 Unit, Component, System testing Some wrappers for testing. Test Driven Development Äÿêóþ çà óâàãó ! 4 Cross-platform testing CI team Plugin mechanisms Çàïèòàííÿ ? 5 NFR Denition KPIs Increase visibility 6 Conclusion