Test Automation in a Continuous Deployment Environment
•Télécharger en tant que PPTX, PDF•
1 j'aime•330 vues
JeST-8 talk by Yehuda Miller from MyHeritage regarding automated and manual testing in a fast paced, continuous delivery agile environment.
Recommandé
Contenu connexe
En vedette
En vedette (11)
Dernier
Dernier (20)
Test Automation in a Continuous Deployment Environment
- 1. Yehuda Miller, QA Engineer Test Automation in a Continuous Deployment Environment
- 2. • Got into QA by accident…just like everyone else • Worked on many different projects • I’m an avid reader (28 books this year) • Moved to Israel July 1, 2014 • I’m colorblind (a challenge for a tester) • My favorite color is teal Hello, who are you?
- 3. “How long does it take you to get one line of code into production?” - Wisdom of the Internet ;)
- 4. Where we were • All manual QA • Repetitive tasks • Time consuming • Two service packs a week • Holding back development • SPC size limits error handling
- 5. • Number of tests • Desktop: 587 (469) (493 IE) • Mobile-web: 69 phone, 23 tablet (0) • Runtime of full suite: 37 min (51 min) • RND size • Number of testers: 12 • Developers: 50 • Grids: 4 (1) • Tests run on prod and semi staging env. • Continuous Deployment Where we are today:
- 6. • We moved to CD in the last 8 months • On average we distribute code to production 30 times per day. • Comes with challenges • Automated test on every code release • Only minimal tests on each release • Dev is responsible to run appropriate tests PRE-RELEASE in staging The CD mindset
- 7. • Risk reduction to production • Small incremental changes are easier to monitor and revert if necessary • Has lower impact on the system • Increasing R&D velocity • Avoiding wasted time on merges and complex coordination before dist. • Allow R&D and product to experiment and innovate more frequently. RND Goals
- 8. • Provide a safety net • Avoid the bottleneck of manual sanity testing • Reliability • No flaky tests • No false failures • QA E2E test should not be the new bottleneck QA Goals
- 9. • QA are responsible for E2E tests only. • Dev write and maintain Unit tests • E2E tests are written during development process • All QA members monitor CD suite results. • CD E2E tests are NOT acceptance tests Responsibilities
- 10. • CD flow from commit to prod is ~25 minutes • Current QA CD suite takes 1.5 minute in best case scenario. • Runs in parallel to some of the jobs in the CD flow • Limited by time frame of parallel jobs • Still have room for further growth Timing
- 11. • Minimal suite – No full regression on commit • Occasional false failures due to env/site performance/human error Risks
- 12. “A challenge only becomes an obstacle when you bow to it.” - Ray A. Davis
- 13. • Reliability • Velocity of tests • Pinpointing the failure • CD halt on failure. • Large scale tests refactoring due to changes. Challenges of CD
- 14. • Maintaining production data integrity • Reliance on pre-existing data • Unable to run on local staging env • Fast adaptation to feature flags • Identifying failure reasons • Need a true BDD/TDD mindset Challenges QA faces
- 15. • Testing on more browsers • Edge • Firefox • Safari? • Suites for our native apps (in progress) • iOS • Android • Upgrading the infra • Gems, grids, Ruby… • Adapt the tests for feature flags • Turn tests on/off based on prod flags • Test “flagged-off” features The road ahead…
- 16. Questions?
- 17. Come join us for our next meetup! Topic: Continuous Deployment Applied @MyHeritage When: Tuesday, December 1, 2015 at 17:00 Where: MyHeritage office, 3 Ariel Sharon 4th floor, Or Yehuda Shameless Plug!
- 18. AA Milne Thanks! We are hiring
Notes de l'éditeur
- Talking today about how we handle end to end automations at myheritage, and how we do it in a CD environment Items: Where we were and where we are CD
- Job 1: Maintaining automated test scripts in IBM Rational Robot for enterprise software (I thought Sprint was a cell phone provider) Job 2: Manual testing of Windows software for small companies (Waterfall). Just beginning automation. Job 3: E-commerce website. Company making a rocky transition to agile. Zero automation
- Next we’ll talk about Continuous Deployment and how we approach it
- A short description of the process we did over the last 2 and a half years.
- Number in parenthesis is beginning of year Desktop tests are on Chrome and IE. Mobile-web tests are running in chrome with emulation mode turned on. Full suite runs at least 2x daily Semi staging is production database with offline code.
- What’s CD – As soon as code is production-ready it is committed and distributed to production. Code deployed to canary server first, unit and UI tests run there. Only distribute if all minimal suites pass. Challenges such as QA holding up a developer from committing code and making further changes in the same file. We started CD only with unit tests. Added QA tests within two weeks because the site went down several times which only our tests could have caught
- Feature flags are a key ingredient. Committing code dark means that it can be tested after distribution if necessary.
- E2E tests are written during development process . preferably before the feature is ready During development process, it’s the dev and QA’s responsibility to run automations against the new code.
- In release cycles this quick it’s impossible to test everything manually or run a large automation suite Canary first, then in prod after web servers get the code Pre-dist is only one which fails the flow, post dist still requires immediate investigation. (All QA monitor this)
- Note: prefer reliability and speed of tests over coverage and safety to enable us to distribute more frequently. Reduce dev time on the actual commit/dist. Adds responsibility to devs to make for damn sure their code works
- We still face some challenges, which are…
- Pinpointing the failure (need strong tests; cucumber helps with this) If one of our tests fail it stops the build Can’t commit test code that is note synched with prod.
- Trying to refactor old tests which rely on existing data to now create and destroy the needed data in the run.
- Ran Levy and Elad Shmitanka why and how MyHeritage made the transition to continuous deployment We will also cover theactual technical implementation of MyHeritage's CD flow.