Recommandé
Contenu connexe
Similaire à Progression testing - on how regression test automation needs to be progressive to keep up with a changing product
Similaire à Progression testing - on how regression test automation needs to be progressive to keep up with a changing product (20)
Progression testing - on how regression test automation needs to be progressive to keep up with a changing product
- 1. PPrrooggrreessssiioonn TTeessttiinngg AAddaamm KKnniigghhtt TTwwiitttteerr:: @@aaddaammppkknniigghhtt BBlloogg:: wwwwww..aa--ssiissyypphheeaann--ttaasskk..ccoomm •hhttttppss::////wwwwww..fflliicckkrr..ccoomm//pphhoottooss//ooppeenneeyyee//55006666001188777799
- 4. TThhiiss iiss RRaaiinnSSttoorr Service Manager n NParchive Archive store (data) Information area (metadata) User Tools Service Manager 3 Service Manager 2 Shared area Staging area 1 Production Database 1 Production Database 2 Service Manager 1 Staging area 2 Working area (queue) Query Tool: npa_query Query via ODBC: NParchive ODBC Driver ODBC-compliant 3rd Party Tools Admin tool: npa_admin Import Tool: npa_import Export Tool: npa_export Import Engine Admin Engine Query Engine Query via ADO: NParchive ODBC Driver ADO-compliant 3rd Party Tools
- 6. LLaacckk ooff PPrrooggrreessssiioonn iiss RReeggrreessssiioonn • BBuuggss ccaann aarriissee wwhheerree ffuunnccttiioonnaalliittyy hhaass nnoott cchhaannggeedd • CCuussttoommeerr eexxppeeccttaattiioonn hhaass cchhaannggeedd • TThhee mmaarrkkeett hhaass mmoovveedd oonn • WWee ccaann mmoovvee bbaacckkwwaarrddss bbyy ssttaannddiinngg ssttiillll •image : ray wewerka http://www.flickr.com/photos/picfix/4409257668
- 7. AAuuttoommaattiioonn iiss ssooffttwwaarree ddeevveellooppmmeenntt • FFooccuuss oonn nneeeeddss rraatthheerr tthhaann ttoooollss • IItteerraattiivveellyy iimmpprroovvee ttoo iinnttrroodduuccee tteesstt ccaappaabbiilliittiieess • IInnccoorrppoorraattee nneewwllyy aaddddeedd pprroodduucctt ffeeaattuurreess • AAddaapptt ttoo nneeww eennvviirroonnmmeennttss aanndd tteecchhnnoollooggiieess •hhttttppss::////wwwwww..fflliicckkrr..ccoomm//pphhoottooss//ffrraannccooiissmmaaiilllloott//88333333442222996644//
- 8. TThhiiss iiss aa RRaaiinnSSttoorr TTeesstt PPaatttteerrnn
- 9. TThhiiss iiss aa RRaaiinnSSttoorr TTeesstt PPaatttteerrnn
- 10. Test HHaarrnneessss RReeqquuiirreemmeennttss CChhaannggee From Through To Data Import and Administration Query Commands Full Cluster Administration Single Server Single Process Multi-server single pack Multi-server parallel Packs Sequential Packs Iterative Execution Parallel Iterative Execution No ODBC/JDBC ODBC/JDBC single thread ODBC/JDBC multi-thread Text Based Reporting Simple HTML Report Interactive HTML with Summary and Differences Linux Only Linux, Solaris, AIX, HPUX and (gasp) Windows Linux Only
- 11. TThhaannkkss • EEmmaaiill:: aaddaammppkknniigghhtt@@aa--ssiissyypphheeaann--ttaasskk..ccoomm • TTwwiitttteerr:: @@aaddaammppkknniigghhtt • BBlloogg:: hhttttpp::////wwwwww..aa--ssiissyypphheeaann--ttaasskk..ccoomm • EEmmaaiill:: aaddaamm..kknniigghhtt@@rraaiinnssttoorr..ccoomm • WWeebbSSiittee:: hhttttpp::////wwwwww..rraaiinnssttoorr..ccoomm
- 12. RReeffeerreenncceess • FFuurrtthheerr RReeaaddiinngg • FFrroomm ssmmaarrtt ppeeooppllee • hhttttpp::////lliizzkkeeoogghh..ccoomm//22001111//0033//0044//sstteepp--aawwaayy--ffrroomm--tthhee--ttoooollss// • hhttttpp::////tteessttoobbsseesssseedd..ccoomm//22001111//0022//tthhee--aattdddd--aarrcchh// • FFrroomm mmee • hhttttpp::////wwwwww..aa--ssiissyypphheeaann--ttaasskk..ccoomm//22001111//0033//wwiitthhoouutt--ssttaabbiilliisseerrss-- wwhhyy--wwrriittiinngg--yyoouurr..hhttmmll • hhttttpp::////wwwwww..aa--ssiissyypphheeaann--ttaasskk..ccoomm//22001122//1100//mmoovviinngg--bbaacckkwwaarrddss-- bbyy--ssttaannddiinngg--ssttiillll--hhooww..hhttmmll
Notes de l'éditeur
- This is Adam He is 26 – he’s currently enjoying travelling the world with his girlfriend but all too soon he’ll be returning to the UK and will need to find somewhere to live. He needs a bedrooom, possibly another for friends to stay He needs a big sitting room for entertaining He needs proximity to pubs and restaurants to enjoy socialising with his friends He doesn’t need the hassle of looking after a garden The idea of living in the suburbs fills him with nausea
- This is also Adam He is now 38 - His needs have changed He needs a house He needs lots of bedrooms to house his growing army of children He needs a garden for his children to play in, or at least stand in when he is sick of them in the house. He needs a drive to park his car and a shed to store all of his families stuff He is the same person, but his needs have changed, and an appropriate environment for him to operate in has changed.
- This is RainStor – or at least a depiction of it. I’m sorry for using my own product as an example but after 8 years in the same company, alternative examples are hard to come up with It is a new product, with no customers as yet. Whilst the marketing team claim the capability of running on multiple servers, realistically it only ever runs on one It has simple import and query functionality driven solely through the command line, and a basic ODBC capability
- This is RainStor in 2014 It is an established big data product It does now have the capability of running on multiple servers managed through cluster management capabilities It has mature ODBC/JDBC and API query functionality, plus a remote client command line tool, It has Hadoop integration with querying through Pig/Hive It has Teradata integration It has many asynchronous processes managing metadata between nodes of the cluster
- So what has this got to do with regression testing? Well the fact is that we need our products to change, to advance If the functionality does not change then this in itself can actually be a regression. My personal definition of a bug is that it is the potential for a negative relationship between a stakeholder and the product. That potential can develop if that stakeholder, for example the customer, perception changes as to what is acceptable I have had bugs raised by existing customers where the functionality has not changed, but the expectation has changed based on what else the product could do, or the situations that they were encountering Therefore the software must change, and will inevitably expose new features, new interfaces and possibly even new applications that were not considered when we started out in our test automation If we treat the automation as a procurement, something which must be purchased up front to meet our requirements, then we risk those tools becoming more and more redundant over time as the products that we test develop and we are stuck attempting to regression test the current product with a tool which was suitable for the product as of 3 years ago.
- I believe that as soon as you include test automation in your development, then you essentially have two development activities, one for the product and one for the automation to keep up with it. Treating automation as a software development rather than a tool procurement does a number of things It takes away the focus on tools and moves it towards your requirements. It allows you to start with the highest priority capabilities and build in more advanced features over time It allows you to cope with changes in the product feature set It allows you to integrate into new operating environments and interface with new third party technologies that your software might need to work with, that would have been missing in the initial requierments gathering for an automation tool Building regression test automation as an ongoing development project rather than a tool procurement provides the flexibility to keep moving forwards in whatever direction your product might take you.
- So let us return to our product under test – well in the early days of automation I created some simple test patterns to share with the team and identify optimal test pack structures and ‘call-out’ suboptimal ones, such as those containing dependencies between tests in different test packs. This is a basic spider pattern which was the fundamental pattern of the initial harness – a BTA (Build Test Archive) process was run at the start and then any number of independent test packs could be run subsquently atgainst the resulting archive … now if we skip forwards a few years
- This is a test pattern for some of the later packs. We now have multiple machines involved with parallel testpacks run from a central test controller server executing tests against remote servers either through the software or via SSH onto the remote servers. As you can see the requirements, and resulting capabilities, of the test harness to support such a pattern are very different from the simple linear pattern that we could support when we started out. Our capabilities within the regression automation suite have evolved as our product has evolved.
- Here is a brief summary of some of the changing requirements that we have seen in the test pack over time. Talk through them I claim that there is no way, if we had started out purchasing a tool that was capable of automating the requirements on the left, that it would still be capable of delivering the current requirements on the right. This is why we don’t just need regression test automation, we need progression test automation.
- 2