1. In a Time of Crisis
Speed is Everything

2. Your Strengths
and your Weaknesses
are Amplified

3. Strategy vs. Tactical Testing:
Actions for Today, Plans for
Tomorrow
Chuck Schneider
Vice President of Platform Engineering
and
Testing Strategy - Cerner Corp.

4. Agenda
•Structure of a highly automated testing ecosystem
•Strategic Planning
•Tactical Planning

5. Cerner Corporation

6. Cerner’s Scale
Over 8 billion orders placed annually
500 applications (C#, C++, Java RCP, Browser, Android, iOS)
475,000 concurrent users worldwide
1415 Eggplant users trained across 9 countries
Q1 2018 357 automated scripts; 2064 reusable handlers
Q3 2019 6,482 automated scripts; 16,499 reusable handlers
Over 250,000 automated tests ran headless in past 12 months
Over 2,000,000 launches of Eggplant in past 12 months
Average Concurrent Executions: 31154 mobile devices in our Device Lab
15,000 Jenkins sessions executed per day

7. Test Design and Shift Left
Test design
and shift left
1. Eliminate the tests you don’t need!
2. Shift Left
• Unit tests, API tests, Functional tests
• Isolate the architecture
3. Test Design
• “Living Tests”
• Organized repository
• Reusable Handlers
The Value - Control Costs, Maintainability

8. Test Data Management
Test design
and shift left

9. Test Data Management
Test design
and shift left
1. Determine approach
• Database resets (snapshots)
• Database record generation
• Use testing automation scripts
2. Identify types of data
• Users
• Activity
• Reference
• Configuration
3. Optimize each test’s data needs
• Will need to generate this data prior to each test
The Value - Repeatability

10. Test Automation
Test design
and shift left

11. Test Automation
Test design
and shift left
1. Understand the applications you are testing
2. Start with the tests you run the most
3. Minimize the use of hardcoded values
4. Version your tests (tests are code)
The Value - Efficiency and Quality

12. Controlled Environments
Test design
and shift left

13. Controlled Environments
Test design
and shift left
1. Create & replicate from a master environment
2. Control the testing environment
• Control access
• Control the data builds and configuration
3. Determine needs for on demand & shared
environments
The Value - Repeatability

14. Continuous Integration
Test design
and shift left

15. Continuous Integration
Test design
and shift left
1. Create headless processes
• Credentials
• Data Creation triggers
• Identifying and/or spinning up SUTs
2. Associate tests with code
3. Design code pipeline management
The Value - Early feedback loop of errors

16. CI PIPELINE
Engineer starts
1
2
Engineer
coding/testing on
branch
Engineer checks-in
code
3 4
Automated Tests
5
Package/Release
76
Automated Tests
Deploy

17. Dashboards
Test design
and shift left

18. Dashboards
Test design
and shift left
1. Define each pillar's success metrics
• Static code analysis
• Which data is auto generated
• Test status per run/release
• Environment refresh/creations
• Adoption
2. Automate the measurements
3. Share dashboards broadly
The Value - Visibility of successes and failures

19. World Class Testing Strategy
To create and
run efficient
and
maintainable
tests
With controlled
test data
That does not
require human
intervention
In on demand
virtual
environments
In a
continuous
manner
Which
captures,
collects and
communicates
coverage &
results
Test design
and shift left
TDM
RQM,
Eggplant,
Selenium EOD CD Dashboard

20. Where to Start?
Strategy vs Tactics

21. 6 Steps to Strategic Testing

22. State the Value for the Company
IMPROVE QUALITY REDUCE COSTS RELEASE MORE
OFTEN
HIGHER CUSTOMER
SATISFACTION

23. Get Clear on your
Goals• Increase frequency of releases by X%
• Reduce production defects by X%
• Reduce testing release cycle by X%
• Percent of automated tests run per release
• Pipeline Health = how long a test stays in a failed
state

24. Assess Current State
• Evaluate where your biggest inefficiencies are today
• Where are you in the Maturity Model today?
• Phase 1 - Manual tests
• Phase 2 - Automating Manual Tests
• Phase 3 - Automating the Process
• Phase 4 - CI Pipeline Adoption
• Phase 5 - Well Established CI Pipeline

25. Define Future State
• Determine your desired Maturity Model Phase for each of your
products
• What tooling and infrastructure will you need to purchase
• Two risks when purchasing any tool
• Will the tool meet your expectations
• Will the tool be adopted
• Culture changes
• Organizational change management
• Role responsibilities may change
• What skills are needed

26. Create Incremental Value
• Determine where to start
• What is causing the most friction?
• Spinning up an environment
• Automate tests
• Improve test designs
• Create milestones
• Associate value and costs to each milestone
• Define how each milestone will be measured for success
• Regression tests run nightly in a headless manner
• Increase Unit testing coverage to X%
• Reduce hours spent creating test data by X%
• X% of tests are automated and run for each release

27. All Aboard
• Everyone needs to be onboard
• Sell this to your
• Testers
• Learn automation skills
• Reduce repetitive mundane tasks
• More time to find bugs
• Engineers
• Faster feedback on quality of code
• Design an automated process
• Executives
• Efficiencies in the development process
• Faster to market
• Higher Quality

28. Disaster Planning
• Traditionally disaster recovery and planning was focused on
infrastructure
• The absence of a workforce needs to be accounted for
• How can your testing system work if your workforce is
• Unexpectedly working remote
• At half capacity
• At 10% capacity
• Mitigations
• The more that’s automated the more prepared you will be
• A fully automated CI Pipeline
• The system is running the system
• Mobile device lab

29. 4 Steps to Tactical Testing

30. Assess the Situation
• Understand the impact to your business
• Engage with your customers
• What do they desperately need?
• Look at Production
• What capabilities are being used more or less?
• Which of your products are getting crushed with abrupt change?
• New Use Cases
• What has emerged as something completely new?
• Are there gaps in capabilities
• Engage with other leaders
• Do they have a greater need?

31. Assess the Situation
• What resources do you have on hand
• What tools do you have?
• Don’t have any? Sign up for a 30 day free trial
• What data and reporting can you leverage?
• What infrastructure can you borrow?
• How many people are available?
• With-in your team
• Across departments
• Who is sitting idle amongst all this change?

32. Create a Tactical Plan
• Determine what is the ONE thing you are going to focus on
• Measure in units of Days how you can add value
• It’s tactical it’s not supposed to be perfect
• Ex. Create 1 automated script
• That someone must manually kick off
• And you must manually build test data
• And you hardcode variables for one environment

33. Create a Tactical Plan
• Don’t
• Wait for more resources
• Buy something new
• Create a 3 to 5 year roadmap
• Do
• Be Creative
• Evaluate what work in progress can be put on hold and what can not
• Articulate the impact of these decisions
• Ask for help across your company and your vendors
• Create focus for your team!

34. Execute the Plan
• Communicate the plan
• Take time to build the context around why this plan is happening
• The importance of the project
• Other projects that are being put on hold
• Who is going to be an active participant
• Expected timeline
• Successful outcome
• Communication structure to the team and leadership
• Let them know there will be changes as this progresses
• Thank them for their work and focus

35. Execute the Plan
• Action is everything
• Don’t spend weeks planning, put your plan together in no more
than 5 days

36. Adjust as needed
• Your plan is underway!
• Ask your project team what barriers have been identified?
• How can you remove those barriers so the team can focus
• Communicate frequently to leadership
• Be transparent
• Communicate victories and setbacks

37. Celebrate the Victory
• Once your plan is achieved declare victory!
• End the project
• People will rally around a cause but, will become weary if you
continually expand the project into your annual roadmap
• Praise the team
• Share the victory with your team that made it happen

38. Structure, Strategy & Tactics
• Structure
• Strategy
• State the Value
• Get Clear on your Goals
• Assess Current State
• Define Future State
• Create Incremental Value
• All Aboard
• Tactics
• Assess the Situation
• Create a Tactical Plan
• Execute the Plan
• Adjust as needed and Celebrate
the Victory

39. Questions or
for a copy of these slides contact me
Chuck Schneider at
www.linkedin.com/in/chuck-schneider