SlideShare une entreprise Scribd logo
1  sur  27
Lessons Learned in Software DevelopmentQA Infrastructure – Maintaining Robustness in Commercial Software,[object Object],Marcus Lagergren,[object Object],Consulting Member of Technical Staff,[object Object],Oracle Corporation,[object Object]
About the Speaker,[object Object],Marcus Lagergren holds a master’s degree from KTH, major in Theoretical Computer Science,[object Object],Marcus was one of the founders of Appeal Virtual Machines that was acquired by BEA in 2002, which in turn was acquired by Oracle in 2008.,[object Object],Marcus has worked on almost all aspects of the JRockit Virtual Machine and is now working with Virtualization technology,[object Object],Marcus likes power tools and scuba diving.,[object Object]
Agenda,[object Object],Robustness in commercial apps with tight release schedules.,[object Object],Utopian vision: perpetual stable bits so we can spin off a release at any time,[object Object],Build Systems,[object Object],Source control and Development,[object Object],Tests,[object Object],Functionality,[object Object],Performance,[object Object],Regression testing,[object Object]
Agenda,[object Object],Result databases,[object Object],Automatic Testing,[object Object],Complex and not-so-standard testing,[object Object],Development aspects,[object Object]
Why listen to me?,[object Object],We’vespent the last 10 yearsdeveloping a JVM and the last 3 yearsdeveloping a Guest Operating system for twocommercial hypervisors.,[object Object],Hundreds of thousands on man hoursspent on robustnessalone,[object Object],Harder-to-debug software hardlyexists. ,[object Object],We’vehad to invent stuff from dayone.,[object Object],Ok, from day 365 or so, lessonslearned,[object Object],We’vemademanymistakesalong the way.,[object Object],No one gets to Utopia, but at leastwehave a reasonablygoodidea of in whichdirection to go,[object Object]
BEA Confidential.  |   6,[object Object],QA infrastructure,[object Object],QA infrastructure is harder and probably even more important than development infrastructure.,[object Object],The most valuable lesson we have learned is that it must be developed parallel to the application and significant effort must be spent on it. ,[object Object],It is at least as important as the application itself.,[object Object],Sometimes the boundaries between app and test infrastructure aren’t even clear.,[object Object]
QA infrastructure,[object Object],QA and Dev, if separate deparments or roles, shouldalways work together. ,[object Object],Preferably as physicallyclose to eachother as possible,[object Object],Theyshould be able to fill in for eachother and be encouraged to doeachothers’ work.,[object Object],Verydangerous to have a separate QA department on anotherfloor.,[object Object],Verydangerous for QA to just do blackbox testing withoutunderstandingwhat’s in the box.,[object Object],QA staffshould be treated as anyotherdeveloper,[object Object]
Build System,[object Object],Build system, test system and sourcecontrol are parts of the same distributed system. ,[object Object],Mobility - Buildanythinganywhere, locally or globally (distributed) - ”Adistributed cross compiler”,[object Object],Build system should be selfcontained & part of sourcecontrol. ,[object Object],Do a sync on a fresh laptop, have all the details.,[object Object],We chose to putbinariesthere as well to producedeterministic bits and provide selfsufficience,[object Object],Not always a goodidea, butmostly a goodidea,[object Object]
Source Control and Development,[object Object],Needgood support for distributeddevelopment,[object Object],Should be able to handledirectories as separate sourcecontrolentitites.,[object Object],Gatekeepers of mainbranches, distributed team baseddevelopment.,[object Object],Sourcecontrol, builds and developmentshouldonlyrequire vi + prompt,[object Object],Morecomplexenvironments on top for ease of use.,[object Object],Easier to extend with different UIs.,[object Object]
Test System,[object Object],Also under sourcecontrol,[object Object],Distributed system – veryimportant.,[object Object],Virtualizeifpossible. Maximizeresourceusage.,[object Object],Local and remote test runspossible.,[object Object],Submitjobs ”crunchthroughthese tests”,[object Object],”Check in ifpasses tests”. ,[object Object],Test Machines,[object Object],Performance test machines (dedicated),[object Object],Functionality test machines (not necessarilydedicated),[object Object],Anymachinecanvolounteer CPU cycles for functional testing.,[object Object]
Building Blocks – Tests,[object Object],Many tests, especially regression tests, for an appneedn’t be morethan a mainclass with a returnvalue.,[object Object],Keep it simple!,[object Object],”I spent a fewhoursdistilling this huge program down to a reproducer for BUG123456”,[object Object],Claim:ifit’s simple enough to write and submit a test, > 50% of the bugscan get regression tests as part of the original bugfix. ,[object Object],I willaddress the other 50% later.,[object Object]
Building Blocks – Tests,[object Object],Easy-to-write tests make it possible for the test suite to grownaturally. ,[object Object],If 10 minutes of spare time canlead to a new test beingwritten, checked in and enabled as part of the global test suite, you havesucceeded. ,[object Object],Encouragedevelopers to check in unit tests for new functionalitytogether with the functionality.,[object Object],Need the infrastructure for it in the app,[object Object],Mightwant to enforce this strictly, but it might hinder developmenttoo.,[object Object]
Building Blocks – Result Database,[object Object],Store results in cheapdatabase with sensible layout somewhere.,[object Object],Any freeware is fine – get it up and running. ,[object Object],Easy to maintain and backup,[object Object],Query from localmachinesabouthistorical test results.,[object Object],”Whenexactlydid this performance regression appear?” ,[object Object],”List all benchmarkscores on this machine for this benchmarksinceJanuary 1”,[object Object],”Has this functional test failedbefore? Whatwere the bugfixes?”,[object Object]
Building Blocks - Tests,[object Object],Use ”terror harnesses” that attack the cross sectionsbetweenmodules. ,[object Object],AllocAndRun,[object Object],RedefineClasses,[object Object],ExceptionInClinit,[object Object]
Building Blocks - Performance,[object Object],Anythingcaneffects performance.,[object Object],EVERYTHING affects performance.,[object Object],Weneedautomatic regression warnings. Anyone who submits a performance regression will get an e-mail from the test system.,[object Object],Continuously make it easy to addmorebenchmarks.,[object Object],Automation: Deviations, baselines, invariants. ,[object Object]
Testing – The need for continuous automatic testing,[object Object],Needcontinuousautomatic testing.,[object Object],Example from real life: JRockit Solaris has beenmadeavailableoff and on over the years. Bit rot sets in immediatelywhenremoved from automated testing. ,[object Object],Release version may break debug version and vice versa.,[object Object],Linux version may break Windows version and vice versa.,[object Object],Useextremelystrict and pickycompilerflags.,[object Object]
Testing – So What About the Other 50%?,[object Object],Simple Java programs with main functions may not be enough for all the bugs.,[object Object],How do we test for a specific optimization bug in the code generator? ,[object Object],How do we test for a strange boundary case that crashes the GC, that happens after two weeks in production?,[object Object],Key observation: We need to export a state.,[object Object]
Testing – So What About the Other 50%?,[object Object],Examples:,[object Object],Create a very special heap with a fewobjects in nastyplaces.  Load it and trigger a garbagecollection. Save it and compare to reference.,[object Object],Serialize an IR from just before an offendingoptimization. Load it and trigger the optimization. Save the resulting IR and compare it to reference.,[object Object],Comparewould be more of an ”equals” than a ”memcmp”,[object Object],Weneed a level of modularizationthat’sgoodenough for this. ,[object Object],The collection of tests shouldgrownaturally, but the VM design shouldallow the ways of testing the VM to grownaturally as well.,[object Object]
Testing – So What About the Other 50%?,[object Object],But of courseit’s not as simple as that.,[object Object],Whataboutmultithreadedapps? Race conditions? ,[object Object],Plenty of threadsoperate on the same memory – e.g. Multithreaded GC. Howcanwe make test cases?,[object Object],Synchronization points. ,[object Object],Randomized input, randomizedsleeps. Try to cover the malicioussideeffects of parallelism.  ,[object Object],ThingsliketheRaceTrackalgorithmcanfindsome (not all) races in staticcode, but the world is dynamic. Testing needs to be.,[object Object]
Testing – So What About the Other 50%?,[object Object],Disclaimer: Sometimeswe just need to crunch a lot of code for a long, long time. Nothingelsesuffices to reproduce a problem or the framework that would make it possibledoesn’texist.,[object Object],So make sure the distributed system burnsthosefree CPU cycles,[object Object],And make the dumps full and comprehensible. ,[object Object],Don’tlosethem, dammit! No wipingthem after 24h. Disk is cheap. ,[object Object],”Phonehome”,[object Object],Suprisinglyeffectiveif you haveenough beta testers. ,[object Object]
Testing – Retrofitting a framework	,[object Object],You willprobablyhave to do this, sincepeopledon’tunderstand the importance of fundamental QA from day 1.,[object Object],Situation: Weneed the QA infrastructurebutdon’thave it. Our app has come a longway.,[object Object],Learn from history,[object Object],For example, go over 500 bug parade entries for HotSpot.,[object Object],Howmanycan be tested by small deterministicreproducers? ,[object Object],Whatabout the rest - brainstormwhatfunctionality the VM wouldneedifwehad to write a simple reproducer for each problem.,[object Object]
Development – The platform matrix,[object Object],Try to keep the amount of common code as large as possible.,[object Object],It is always a choice between platform specific features and test matrix growth.,[object Object],Initially, our performance critical code was native. As our JIT got better, we would write more and more in Java. Native is much worse.  ”premethods”,[object Object],Augmented Java – intrinsics, ”pd_addr”, preprocessed Java files. ,[object Object]
Development – The platform matrix,[object Object],Otherseemlinglyplatformdependentthingscan be madeplatform independent.,[object Object],Example: Native stubs. The bulk of the work is parameter marshalling, the register allocatorcando that already. ,[object Object],Beware of ”falseabstraction”. ,[object Object],That extra parameter that is NULL on all platformsexcept IA64.,[object Object],Implementationlanguage: Debugging is an issue,[object Object],Powerful C/C++ debuggersexist. Meta-debugging is usuallyharder. ,[object Object]
Development,[object Object],Don’tlosefocus. Modularity first.,[object Object],Example: ”the fastest server side JVM”, ”startup time is an issue”, ”clientapplications are an issue” ”weneedzero overhead runtime instrumentation”. Runfool! Run! ,[object Object],It is importantwhenoptimizing for performance not just too look at e.g.SPECjbb™and SPECjvm98™ Real world applicationsdo a lot of otherthings. ,[object Object],”There is no genericcommutative plus operator”. At leastnobodycares.,[object Object]
Development - Policy,[object Object],Don’t be toomuch of a quality fascist whencode is written.,[object Object],If you spend all your time preventinglargercheckins or demand 100% testing on everythingnothingwillever get checked in.,[object Object],If you demand a strictlydocumented process with specifications for everything, all anyonewilleverdo is to writespecifications and holdmeetings.,[object Object],Both of the above are good in smalleramounts.,[object Object],It’smore of an awarenessthing.,[object Object],And the infrastructursshouldquickly and mercilesslyraise the alarm as soon as something breaks to preventfurtherdamage.,[object Object]
Lessons Learned ,[object Object],Summary – The important stuff to bring with you,[object Object],Build the test infrastructure in parallel with the application,[object Object],Start at the same time! Don’tput it off. It is part of the appdevelopment process and should be in the time budget. ,[object Object],IdeallyDevand QA teams should be fused and be able to doeachother’sjobs. No separate compartments.,[object Object],Don’t be afraid to couple it tightly in placesif that is what is required to maintainstability.,[object Object],Use all available CPU cycles for testing,[object Object]
Lessons Learned in Software Development: QA Infrastructure – Maintaining Robustness in Commercial Software

Contenu connexe

Tendances

Test Driven Development by Denis Lutz
Test Driven Development by Denis LutzTest Driven Development by Denis Lutz
Test Driven Development by Denis Lutzjazzman1980
 
Test driven development
Test driven developmentTest driven development
Test driven developmentNascenia IT
 
Test driven development
Test driven developmentTest driven development
Test driven developmentShalabh Saxena
 
Постоянное тестирование интеграции
Постоянное тестирование интеграцииПостоянное тестирование интеграции
Постоянное тестирование интеграцииSQALab
 
Why Test Driven Development?
Why Test Driven Development?Why Test Driven Development?
Why Test Driven Development?Naresh Jain
 
Agile Engineering Sparker GLASScon 2015
Agile Engineering Sparker GLASScon 2015Agile Engineering Sparker GLASScon 2015
Agile Engineering Sparker GLASScon 2015Stephen Ritchie
 
Continuous Integration using Cruise Control
Continuous Integration using Cruise ControlContinuous Integration using Cruise Control
Continuous Integration using Cruise Controlelliando dias
 
Enabling Agile Testing Through Continuous Integration Agile2009
Enabling Agile Testing Through Continuous Integration Agile2009Enabling Agile Testing Through Continuous Integration Agile2009
Enabling Agile Testing Through Continuous Integration Agile2009sstolberg
 
Continuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsContinuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsMike Brittain
 
Tccc10 tooling testingci-vs2010teamcity
Tccc10 tooling testingci-vs2010teamcityTccc10 tooling testingci-vs2010teamcity
Tccc10 tooling testingci-vs2010teamcityBaskin Tapkan
 
Scrum and Test-driven development
Scrum and Test-driven developmentScrum and Test-driven development
Scrum and Test-driven developmenttoteb5
 
Software Design for Testability
Software Design for TestabilitySoftware Design for Testability
Software Design for Testabilityamr0mt
 
Everything you ever wanted to know about deployment but were afraid to ask
Everything you ever wanted to know about deployment but were afraid to askEverything you ever wanted to know about deployment but were afraid to ask
Everything you ever wanted to know about deployment but were afraid to asklauraxthomson
 
AD208 - End to End Quality Processes for Top Notch XPages Apps
AD208 - End to End Quality Processes for Top Notch XPages AppsAD208 - End to End Quality Processes for Top Notch XPages Apps
AD208 - End to End Quality Processes for Top Notch XPages Appsbeglee
 
Automate across Platform, OS, Technologies with TaaS
Automate across Platform, OS, Technologies with TaaSAutomate across Platform, OS, Technologies with TaaS
Automate across Platform, OS, Technologies with TaaSAnand Bagmar
 
Deploying ML models to production (frequently and safely) - PYCON 2018
Deploying ML models to production (frequently and safely) - PYCON 2018Deploying ML models to production (frequently and safely) - PYCON 2018
Deploying ML models to production (frequently and safely) - PYCON 2018David Tan
 
Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)Babul Mirdha
 

Tendances (20)

Test Driven Development by Denis Lutz
Test Driven Development by Denis LutzTest Driven Development by Denis Lutz
Test Driven Development by Denis Lutz
 
Test driven development
Test driven developmentTest driven development
Test driven development
 
Test driven development
Test driven developmentTest driven development
Test driven development
 
Постоянное тестирование интеграции
Постоянное тестирование интеграцииПостоянное тестирование интеграции
Постоянное тестирование интеграции
 
Why Test Driven Development?
Why Test Driven Development?Why Test Driven Development?
Why Test Driven Development?
 
Agile Engineering Sparker GLASScon 2015
Agile Engineering Sparker GLASScon 2015Agile Engineering Sparker GLASScon 2015
Agile Engineering Sparker GLASScon 2015
 
Continuous Integration using Cruise Control
Continuous Integration using Cruise ControlContinuous Integration using Cruise Control
Continuous Integration using Cruise Control
 
Enabling Agile Testing Through Continuous Integration Agile2009
Enabling Agile Testing Through Continuous Integration Agile2009Enabling Agile Testing Through Continuous Integration Agile2009
Enabling Agile Testing Through Continuous Integration Agile2009
 
Unit Testing
Unit TestingUnit Testing
Unit Testing
 
Continuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsContinuous Delivery: The Dirty Details
Continuous Delivery: The Dirty Details
 
Tccc10 tooling testingci-vs2010teamcity
Tccc10 tooling testingci-vs2010teamcityTccc10 tooling testingci-vs2010teamcity
Tccc10 tooling testingci-vs2010teamcity
 
Scrum and Test-driven development
Scrum and Test-driven developmentScrum and Test-driven development
Scrum and Test-driven development
 
Software Design for Testability
Software Design for TestabilitySoftware Design for Testability
Software Design for Testability
 
Test driven development
Test driven developmentTest driven development
Test driven development
 
Everything you ever wanted to know about deployment but were afraid to ask
Everything you ever wanted to know about deployment but were afraid to askEverything you ever wanted to know about deployment but were afraid to ask
Everything you ever wanted to know about deployment but were afraid to ask
 
AD208 - End to End Quality Processes for Top Notch XPages Apps
AD208 - End to End Quality Processes for Top Notch XPages AppsAD208 - End to End Quality Processes for Top Notch XPages Apps
AD208 - End to End Quality Processes for Top Notch XPages Apps
 
Automate across Platform, OS, Technologies with TaaS
Automate across Platform, OS, Technologies with TaaSAutomate across Platform, OS, Technologies with TaaS
Automate across Platform, OS, Technologies with TaaS
 
Deploying ML models to production (frequently and safely) - PYCON 2018
Deploying ML models to production (frequently and safely) - PYCON 2018Deploying ML models to production (frequently and safely) - PYCON 2018
Deploying ML models to production (frequently and safely) - PYCON 2018
 
Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)Test Driven iOS Development (TDD)
Test Driven iOS Development (TDD)
 
Test Driven Development
Test Driven DevelopmentTest Driven Development
Test Driven Development
 

En vedette

Programmatic — агентство — клиент
Programmatic — агентство — клиентProgrammatic — агентство — клиент
Programmatic — агентство — клиентMoscow Digital
 
Matt Stauble Media 2013
Matt Stauble Media 2013Matt Stauble Media 2013
Matt Stauble Media 2013Matt Stauble
 
A Guide to Mbeya Tanzania
A Guide to Mbeya TanzaniaA Guide to Mbeya Tanzania
A Guide to Mbeya TanzaniaMwema Hudson
 
Early Experiences with the OpenMP Accelerator Model
Early Experiences with the OpenMP Accelerator ModelEarly Experiences with the OpenMP Accelerator Model
Early Experiences with the OpenMP Accelerator ModelChunhua Liao
 
Video Priming – How to Give Your Acquisition Campaigns an Unfair Advantage
Video Priming – How to Give Your Acquisition Campaigns an Unfair AdvantageVideo Priming – How to Give Your Acquisition Campaigns an Unfair Advantage
Video Priming – How to Give Your Acquisition Campaigns an Unfair AdvantageGrow.co
 
Cv cover letter laura blanco gonzález
Cv cover letter laura blanco gonzálezCv cover letter laura blanco gonzález
Cv cover letter laura blanco gonzálezLaura Blanco González
 
Tjänstgöringsbetyg för Christin
Tjänstgöringsbetyg för ChristinTjänstgöringsbetyg för Christin
Tjänstgöringsbetyg för ChristinChristin Ljungqvist
 
IoTで畑を監視してみる
IoTで畑を監視してみるIoTで畑を監視してみる
IoTで畑を監視してみるJun Ichikawa
 
Personligt Brev LaszloB 2016
Personligt Brev LaszloB 2016Personligt Brev LaszloB 2016
Personligt Brev LaszloB 2016Laszlo Bodnar
 
Android Test Automation – one year later
Android Test Automation – one year laterAndroid Test Automation – one year later
Android Test Automation – one year laterDominik Dary
 
Grimpeurs et managers : en prise directe sur le risque
Grimpeurs et managers : en prise directe sur le risqueGrimpeurs et managers : en prise directe sur le risque
Grimpeurs et managers : en prise directe sur le risqueChristophe Lachnitt
 
Clinical Trials and the Disney Effect
Clinical Trials and the Disney EffectClinical Trials and the Disney Effect
Clinical Trials and the Disney EffectJohn Reites
 
SXSW GAW Miners - Session Visual Aides
SXSW GAW Miners - Session Visual AidesSXSW GAW Miners - Session Visual Aides
SXSW GAW Miners - Session Visual AidesGaw_Amber
 
Basics of Computer Science
Basics of Computer ScienceBasics of Computer Science
Basics of Computer ScienceMaxim Zaks
 
Newsletters : outil d'hier ou d'aujourd'hui
Newsletters : outil d'hier ou d'aujourd'huiNewsletters : outil d'hier ou d'aujourd'hui
Newsletters : outil d'hier ou d'aujourd'huiBaudy et Compagnie
 

En vedette (20)

BRAND WAR 2
BRAND WAR 2BRAND WAR 2
BRAND WAR 2
 
Programmatic — агентство — клиент
Programmatic — агентство — клиентProgrammatic — агентство — клиент
Programmatic — агентство — клиент
 
Matt Stauble Media 2013
Matt Stauble Media 2013Matt Stauble Media 2013
Matt Stauble Media 2013
 
A Guide to Mbeya Tanzania
A Guide to Mbeya TanzaniaA Guide to Mbeya Tanzania
A Guide to Mbeya Tanzania
 
MYportfolio2016
MYportfolio2016MYportfolio2016
MYportfolio2016
 
Early Experiences with the OpenMP Accelerator Model
Early Experiences with the OpenMP Accelerator ModelEarly Experiences with the OpenMP Accelerator Model
Early Experiences with the OpenMP Accelerator Model
 
Video Priming – How to Give Your Acquisition Campaigns an Unfair Advantage
Video Priming – How to Give Your Acquisition Campaigns an Unfair AdvantageVideo Priming – How to Give Your Acquisition Campaigns an Unfair Advantage
Video Priming – How to Give Your Acquisition Campaigns an Unfair Advantage
 
CV- M Nawaz
CV- M NawazCV- M Nawaz
CV- M Nawaz
 
Cv cover letter laura blanco gonzález
Cv cover letter laura blanco gonzálezCv cover letter laura blanco gonzález
Cv cover letter laura blanco gonzález
 
Tjänstgöringsbetyg för Christin
Tjänstgöringsbetyg för ChristinTjänstgöringsbetyg för Christin
Tjänstgöringsbetyg för Christin
 
IoTで畑を監視してみる
IoTで畑を監視してみるIoTで畑を監視してみる
IoTで畑を監視してみる
 
Personligt Brev LaszloB 2016
Personligt Brev LaszloB 2016Personligt Brev LaszloB 2016
Personligt Brev LaszloB 2016
 
Android Test Automation – one year later
Android Test Automation – one year laterAndroid Test Automation – one year later
Android Test Automation – one year later
 
Grimpeurs et managers : en prise directe sur le risque
Grimpeurs et managers : en prise directe sur le risqueGrimpeurs et managers : en prise directe sur le risque
Grimpeurs et managers : en prise directe sur le risque
 
Clinical Trials and the Disney Effect
Clinical Trials and the Disney EffectClinical Trials and the Disney Effect
Clinical Trials and the Disney Effect
 
Agilept
AgileptAgilept
Agilept
 
SXSW GAW Miners - Session Visual Aides
SXSW GAW Miners - Session Visual AidesSXSW GAW Miners - Session Visual Aides
SXSW GAW Miners - Session Visual Aides
 
Basics of Computer Science
Basics of Computer ScienceBasics of Computer Science
Basics of Computer Science
 
Newsletters : outil d'hier ou d'aujourd'hui
Newsletters : outil d'hier ou d'aujourd'huiNewsletters : outil d'hier ou d'aujourd'hui
Newsletters : outil d'hier ou d'aujourd'hui
 
El bullying
El bullyingEl bullying
El bullying
 

Similaire à Lessons Learned in Software Development: QA Infrastructure – Maintaining Robustness in Commercial Software

[RHFSeoul2017]6 Steps to Transform Enterprise Applications
[RHFSeoul2017]6 Steps to Transform Enterprise Applications[RHFSeoul2017]6 Steps to Transform Enterprise Applications
[RHFSeoul2017]6 Steps to Transform Enterprise ApplicationsDaniel Oh
 
Kelly potvin nosurprises_odtug_oow12
Kelly potvin nosurprises_odtug_oow12Kelly potvin nosurprises_odtug_oow12
Kelly potvin nosurprises_odtug_oow12Enkitec
 
Cloud-powered Continuous Integration and Deployment architectures - Jinesh Varia
Cloud-powered Continuous Integration and Deployment architectures - Jinesh VariaCloud-powered Continuous Integration and Deployment architectures - Jinesh Varia
Cloud-powered Continuous Integration and Deployment architectures - Jinesh VariaAmazon Web Services
 
Virtualising Tier 1 Apps
Virtualising Tier 1 AppsVirtualising Tier 1 Apps
Virtualising Tier 1 AppsIwan Rahabok
 
Automating development-operations-v1
Automating development-operations-v1Automating development-operations-v1
Automating development-operations-v1Sumanth Vepa
 
Testing Hourglass at Jira Frontend - by Alexey Shpakov, Sr. Developer @ Atlas...
Testing Hourglass at Jira Frontend - by Alexey Shpakov, Sr. Developer @ Atlas...Testing Hourglass at Jira Frontend - by Alexey Shpakov, Sr. Developer @ Atlas...
Testing Hourglass at Jira Frontend - by Alexey Shpakov, Sr. Developer @ Atlas...Applitools
 
Machine programming
Machine programmingMachine programming
Machine programmingDESMOND YUEN
 
How We Test Event-Driven Microservices
How We Test Event-Driven MicroservicesHow We Test Event-Driven Microservices
How We Test Event-Driven MicroservicesAntoine Craske
 
Stress Test & Chaos Engineering
Stress Test & Chaos EngineeringStress Test & Chaos Engineering
Stress Test & Chaos EngineeringDiego Pacheco
 
Amol_Koshti_04May16
Amol_Koshti_04May16Amol_Koshti_04May16
Amol_Koshti_04May16Amol Koshti
 
How CapitalOne Transformed DevTest or Continuous Delivery - AppSphere16
How CapitalOne Transformed DevTest or Continuous Delivery - AppSphere16How CapitalOne Transformed DevTest or Continuous Delivery - AppSphere16
How CapitalOne Transformed DevTest or Continuous Delivery - AppSphere16AppDynamics
 
How EVERFI Moved from No Automation to Continuous Test Generation in 9 Months
How EVERFI Moved from No Automation to Continuous Test Generation in 9 MonthsHow EVERFI Moved from No Automation to Continuous Test Generation in 9 Months
How EVERFI Moved from No Automation to Continuous Test Generation in 9 MonthsApplitools
 
Testing in a glance
Testing in a glanceTesting in a glance
Testing in a glanceRajesh Kumar
 
Strata CA 2019: From Jupyter to Production Manu Mukerji
Strata CA 2019: From Jupyter to Production Manu MukerjiStrata CA 2019: From Jupyter to Production Manu Mukerji
Strata CA 2019: From Jupyter to Production Manu MukerjiManu Mukerji
 
AWS Lambda from the Trenches
AWS Lambda from the TrenchesAWS Lambda from the Trenches
AWS Lambda from the TrenchesYan Cui
 
Principles and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyPrinciples and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyMike Brittain
 
Adrian marinica continuous integration in the visual studio world
Adrian marinica   continuous integration in the visual studio worldAdrian marinica   continuous integration in the visual studio world
Adrian marinica continuous integration in the visual studio worldCodecamp Romania
 
Java Tuning White Paper
Java Tuning White PaperJava Tuning White Paper
Java Tuning White Paperwhite paper
 

Similaire à Lessons Learned in Software Development: QA Infrastructure – Maintaining Robustness in Commercial Software (20)

[RHFSeoul2017]6 Steps to Transform Enterprise Applications
[RHFSeoul2017]6 Steps to Transform Enterprise Applications[RHFSeoul2017]6 Steps to Transform Enterprise Applications
[RHFSeoul2017]6 Steps to Transform Enterprise Applications
 
Kelly potvin nosurprises_odtug_oow12
Kelly potvin nosurprises_odtug_oow12Kelly potvin nosurprises_odtug_oow12
Kelly potvin nosurprises_odtug_oow12
 
Cloud-powered Continuous Integration and Deployment architectures - Jinesh Varia
Cloud-powered Continuous Integration and Deployment architectures - Jinesh VariaCloud-powered Continuous Integration and Deployment architectures - Jinesh Varia
Cloud-powered Continuous Integration and Deployment architectures - Jinesh Varia
 
Virtualising Tier 1 Apps
Virtualising Tier 1 AppsVirtualising Tier 1 Apps
Virtualising Tier 1 Apps
 
Automating development-operations-v1
Automating development-operations-v1Automating development-operations-v1
Automating development-operations-v1
 
Testing 101
Testing 101Testing 101
Testing 101
 
Testing Hourglass at Jira Frontend - by Alexey Shpakov, Sr. Developer @ Atlas...
Testing Hourglass at Jira Frontend - by Alexey Shpakov, Sr. Developer @ Atlas...Testing Hourglass at Jira Frontend - by Alexey Shpakov, Sr. Developer @ Atlas...
Testing Hourglass at Jira Frontend - by Alexey Shpakov, Sr. Developer @ Atlas...
 
Machine programming
Machine programmingMachine programming
Machine programming
 
How We Test Event-Driven Microservices
How We Test Event-Driven MicroservicesHow We Test Event-Driven Microservices
How We Test Event-Driven Microservices
 
Stress Test & Chaos Engineering
Stress Test & Chaos EngineeringStress Test & Chaos Engineering
Stress Test & Chaos Engineering
 
Amol_Koshti_04May16
Amol_Koshti_04May16Amol_Koshti_04May16
Amol_Koshti_04May16
 
How CapitalOne Transformed DevTest or Continuous Delivery - AppSphere16
How CapitalOne Transformed DevTest or Continuous Delivery - AppSphere16How CapitalOne Transformed DevTest or Continuous Delivery - AppSphere16
How CapitalOne Transformed DevTest or Continuous Delivery - AppSphere16
 
How EVERFI Moved from No Automation to Continuous Test Generation in 9 Months
How EVERFI Moved from No Automation to Continuous Test Generation in 9 MonthsHow EVERFI Moved from No Automation to Continuous Test Generation in 9 Months
How EVERFI Moved from No Automation to Continuous Test Generation in 9 Months
 
Testing in a glance
Testing in a glanceTesting in a glance
Testing in a glance
 
Strata CA 2019: From Jupyter to Production Manu Mukerji
Strata CA 2019: From Jupyter to Production Manu MukerjiStrata CA 2019: From Jupyter to Production Manu Mukerji
Strata CA 2019: From Jupyter to Production Manu Mukerji
 
Gallio Crafting A Toolchain
Gallio Crafting A ToolchainGallio Crafting A Toolchain
Gallio Crafting A Toolchain
 
AWS Lambda from the Trenches
AWS Lambda from the TrenchesAWS Lambda from the Trenches
AWS Lambda from the Trenches
 
Principles and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyPrinciples and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at Etsy
 
Adrian marinica continuous integration in the visual studio world
Adrian marinica   continuous integration in the visual studio worldAdrian marinica   continuous integration in the visual studio world
Adrian marinica continuous integration in the visual studio world
 
Java Tuning White Paper
Java Tuning White PaperJava Tuning White Paper
Java Tuning White Paper
 

Plus de Cωνσtantίnoς Giannoulis

Model-driven Strategic Awareness: From a Unified Business Strategy Meta-model...
Model-driven Strategic Awareness: From a Unified Business Strategy Meta-model...Model-driven Strategic Awareness: From a Unified Business Strategy Meta-model...
Model-driven Strategic Awareness: From a Unified Business Strategy Meta-model...Cωνσtantίnoς Giannoulis
 
Linking Strategic Innovation to Requirements: a look into Blue Ocean Strategy
Linking Strategic Innovation to Requirements: a look into Blue Ocean StrategyLinking Strategic Innovation to Requirements: a look into Blue Ocean Strategy
Linking Strategic Innovation to Requirements: a look into Blue Ocean StrategyCωνσtantίnoς Giannoulis
 
Modeling Business Strategy for Business-IT Alignment
Modeling Business Strategy for Business-IT AlignmentModeling Business Strategy for Business-IT Alignment
Modeling Business Strategy for Business-IT AlignmentCωνσtantίnoς Giannoulis
 
Modeling Strategy Maps & Balanced Scorecards using i*
Modeling Strategy Maps & Balanced Scorecards using i*Modeling Strategy Maps & Balanced Scorecards using i*
Modeling Strategy Maps & Balanced Scorecards using i*Cωνσtantίnoς Giannoulis
 
Modeling Competition-driven Business Strategy for Business IT Alignment
Modeling Competition-driven Business Strategy for Business IT AlignmentModeling Competition-driven Business Strategy for Business IT Alignment
Modeling Competition-driven Business Strategy for Business IT AlignmentCωνσtantίnoς Giannoulis
 
Modeling Business Strategy: A meta-model of Strategy Maps and Balance Scorecards
Modeling Business Strategy: A meta-model of Strategy Maps and Balance ScorecardsModeling Business Strategy: A meta-model of Strategy Maps and Balance Scorecards
Modeling Business Strategy: A meta-model of Strategy Maps and Balance ScorecardsCωνσtantίnoς Giannoulis
 
Towards a Unified Business Strategy Language: a Meta-model of Strategy Maps
Towards a Unified Business Strategy Language: a Meta-model of Strategy MapsTowards a Unified Business Strategy Language: a Meta-model of Strategy Maps
Towards a Unified Business Strategy Language: a Meta-model of Strategy MapsCωνσtantίnoς Giannoulis
 

Plus de Cωνσtantίnoς Giannoulis (8)

Model-driven Strategic Awareness: From a Unified Business Strategy Meta-model...
Model-driven Strategic Awareness: From a Unified Business Strategy Meta-model...Model-driven Strategic Awareness: From a Unified Business Strategy Meta-model...
Model-driven Strategic Awareness: From a Unified Business Strategy Meta-model...
 
Linking Strategic Innovation to Requirements: a look into Blue Ocean Strategy
Linking Strategic Innovation to Requirements: a look into Blue Ocean StrategyLinking Strategic Innovation to Requirements: a look into Blue Ocean Strategy
Linking Strategic Innovation to Requirements: a look into Blue Ocean Strategy
 
Modeling Business Strategy for Business-IT Alignment
Modeling Business Strategy for Business-IT AlignmentModeling Business Strategy for Business-IT Alignment
Modeling Business Strategy for Business-IT Alignment
 
Modeling Strategy Maps & Balanced Scorecards using i*
Modeling Strategy Maps & Balanced Scorecards using i*Modeling Strategy Maps & Balanced Scorecards using i*
Modeling Strategy Maps & Balanced Scorecards using i*
 
Modeling Competition-driven Business Strategy for Business IT Alignment
Modeling Competition-driven Business Strategy for Business IT AlignmentModeling Competition-driven Business Strategy for Business IT Alignment
Modeling Competition-driven Business Strategy for Business IT Alignment
 
Modeling Business Strategy: A meta-model of Strategy Maps and Balance Scorecards
Modeling Business Strategy: A meta-model of Strategy Maps and Balance ScorecardsModeling Business Strategy: A meta-model of Strategy Maps and Balance Scorecards
Modeling Business Strategy: A meta-model of Strategy Maps and Balance Scorecards
 
Towards a Unified Business Strategy Language: a Meta-model of Strategy Maps
Towards a Unified Business Strategy Language: a Meta-model of Strategy MapsTowards a Unified Business Strategy Language: a Meta-model of Strategy Maps
Towards a Unified Business Strategy Language: a Meta-model of Strategy Maps
 
The GM-VV
The GM-VVThe GM-VV
The GM-VV
 

Dernier

UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 

Dernier (20)

UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 

Lessons Learned in Software Development: QA Infrastructure – Maintaining Robustness in Commercial Software

  • 1.
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.