Continuous Test Automation via CI (CodeMash 2012) - Automating the Agile way
•
5 j'aime•1,402 vues
There are many ways to consider on how to design and execute effective automated tests and continuously keep the pulse on quality of product delivery. However when it comes to leveraging existing CI pipeline for functional and performance testing many may not realize that main ingredients are already built-in. This presentation will share the recipes on how to propel automated testing with immediate feedback to the entire team. This presentation is based on: • Hudson/Jenkins CI engine • Ruby and Rake to setup, execute and tear-down test environments • Hpricot (Ruby gem) and Hudson plug-ins to report and trend graphical results dynamically • .NET test tools (Visual Studio MS Team System and Telerik WebAii)
Recommandé
Contenu connexe
Tendances
Tendances (18)
Similaire à Continuous Test Automation via CI (CodeMash 2012) - Automating the Agile way
Similaire à Continuous Test Automation via CI (CodeMash 2012) - Automating the Agile way (20)
Dernier
Dernier (20)
Continuous Test Automation via CI (CodeMash 2012) - Automating the Agile way
- 1. Leonard Fingerman (lfingerman@gmail.com/ twitter: @FingermanL) Agile Automation Tester
- 3. “The key to fixing problems quickly is finding them quickly” “Continuous Integration doesn't get rid of bugs, but it does make them dramatically easier to find and remove” (Martin Fowler)
- 4. Fast Feedback Easier to deal with small increments than plow through a large pile
- 5. 9 developers 3 product owners 2 testers 1 designer
- 6. CruiseControl-> Team City -> Hudson -> Jenkins The key – Modular build pipeline Well-supported & distributed version control system(Git) Consistent build pipeline pattern mapping between Rake build task and Jenkins task plus Ruby Gems to do further uplifting
- 7. Jenkins kicks Expanded Packaging Deployment off a build with Unit/Component basic unit tests Testing • Git commit • Build + Unit • Create • Executable triggers CI Testing Executable Deployed To Job remotely • Against Stage via Sinatra proprietory environment Ruby Gem back-end manual • TDD, BDD exploratory • Persistence testing tests with layer req. DB mocks
- 11. “If you don’t have a comprehensive suite of automated tests, a passing build only means that the application could be compiled and assembled.” “Continuous Delivery” - Jez Humble & David Furley
- 12. Typically CI is in developer domain Integration is about communication The whole Team should exploit CI environment for earlier defect detection
- 13. Scheduling tasks/tests is built-in CI platform supports parallel test execution Reporting & trending is available via plug-ins Notification is built-in Visible to the entire team !
- 14. Jenkins Rake Rake task Jenkins post- Jenkins scheduler setup launches Tests build Plug-ins tasks via command- line • Pickup the • Install latest build App & • Save artifacts • Run Test via • Trends Setup mstest • Report test run-time results • Charts environm • Failing test will ent fail Jenkins job
- 15. All automated test projects are under version control (Git) Functional UI and API Telerik WebAii Non-functional performance & data integrity Microsoft VSTS More test types & coverage always desired
- 16. Data-driven performance tests against REST API (via HTTP) Rake to setup/tear down test instance Jenkins plug-ins to report charts/trends
- 17. Rake - Ruby gem or build tool Rake can specify tasks with dependencies Nokogiri / HPricot transform XML into Jenkins acceptable plug-ins format beautiful looking charts/trends/reports
- 23. Automated Acceptance Test stage (Cucumber-based ) Client-side performance tests Performance Diagnostics Dashboard integrated with dynaTrace
- 25. Slide Deck will be available on slideshare.net Look for Leonard Fingerman
- 26. “Continuous Delivery” - Jez Humble & David Furley Martin Fowler blog
Notes de l'éditeur
- Waterfall typical pictureAgile creates an ever greater need to avoid this situation.Frequency of change that translates to # of builds is always highTalk about my personal experience never ending repetitive testingHave you tested against build xxx …how about xyz ?
- How many are practicing Agile?Fast-paced , many changes …challenge to keep a pulse on qualityLean Movement motto – W. Edwards DemingSelf-testing code helpsBetter to have imperfect test running than perfect test still in progress
- Premise behind CI – Bring the Pain Forward or Fail soon, fail fast, fail often …and cheapA lot easier to deal with smaller increments of bugs than large pilesPhysically and psychologically
- Agile , embrace change of any kind…toys, tools, gamesLots of change and fast pace environment requires to scale testing rapidlyWhat do you do ?
- JoEllenSwitching CI is not a big deal thanks to modular approach
- git commit locallygit push to git hubgit hub triggers web hook back to our serverour server starts build pipelineTo by-pass the firewall Sinatra listens receives a web hook post from GitHub & then in turnKicks off a build & Jenkins kicksFigures out the branch & repo from the pushPost-receive hook b/o firewallInsert “Cont DeliverY book terms & concepts adoptedSome history – “pipeline” called not due to plumbing but serial machine instructions that is divided into parallelStreamsIdeally, the compile and test process that you run prior to check-in and on your CI server should take no more than a few minutes.JoEllenBuild pipeline consists of multiple, coupled builds…first build is triggered by checkinEach build after the first one is triggered by the successful completion of the previous build.Failing build means the end of the pipe for that check in..and we get to hear our wonderful build-break audio
- Tests must be optimized to execute very quickly hereWe have a set of nUnut tests plus Jasmine java-script testsThis stage is designed to catch majority of the problemsAnd run really fastIt’s important early milestone/gate to in the journey of release candidateOnce passes it frees developers to move on to the next task
- Point of Ownership – your problem your fixMake it easier for you – psychological factor to fixImportant part is “fix early” – fire alarm is useless is everyone ignores it !We have a sound alarm with a strobing flash light – I have a break !Delivery teams must be disciplined about fixing defects as soon as they are found
- Build confidence progressively with each stage
- LeonardFrequent communication allows team to know quickly about changes.Elaborate on types of tests outside of CITesting is not a phase and not one to begin after dev phase – typical water fall approachIt’s also not just “testers” domain the whole TEAM is responsible for quality all the timeMention previous experience as QA - Leonard’s personal experience from the past – did it the hard way built test harness infrastructureFor test scheduling, execution and reporting- Silo’d
- Run-time environments can be setup and torn down via build tasks (e.g. Ruby Rake)Break it up JoEllenVisible w/o going somewhere else;Visible to remote team members no worries about this tool or that tool, access priileges etcOne platform , One point of access, one point of visibilityUniform/generic presentation of results & readily accessible to everyone (no access or licensing constraints etc)
- Automated test project are version controlled - GitHUbUse rake for auto build taskUse Jenkins at minimalMapping between rake task and Jenkins job – that’s a consistent patternLeonardRake runs the install in the test environmentRake shells out to mstest to execute the testJenkins saves artifacts and produces reportJenkins
- Regression UI and APIAPI-based data integrity and performanceCan always cover more types like acceptance, security, capacity etc.
- No XML files to edit. No quirky Makefile syntax to worry about
- Rake tasks are mapped to Jenkins tasks
- JoellenAdd more screenshots – screenshot test; formatted report with pass/fail info, trend plugin for p/f over time
- Currently this is de-coupled from build pipeline but you can decide what rules to applyIf you had the SLA tie this up with your build pipelineUsing Jenkins Performance Trending Plug-inRuby gem to xForm the XML display data for trending & charting
- LeonardTake another with all green
- Scenarios will have WIP tag will be used as identifierAutomation via API
- To add to CI automated testing practice“Eat dog food early” + Early customer testing