Guest presenter Forrester VP and Principal Analyst Diego Lo Giudice joined Worksoft Agile expert Chris Kraus for an exploration of the state of adoption of Agile, DevOps and test automation in the enterprise packaged application space. Learn why it is important to include testing of packaged apps and mainframe as part of an Agile-plus-DevOps strategy and how the adoption of Agile and DevOps varies for packaged vs. custom-built applications. View the recorded event at: https://www.worksoft.com/downloads/worksoft-forrester-webinar-agile-plus-devops-testing-for-packaged-applications.

1. Welcome!
Agile-plus-DevOps
Testing for Packaged
Applications
September 2018
©2018 Worksoft, Inc. All trademarks and product names mentioned herein are the
property of their respective owners in the United States, Canada and/or other countries.
The information contained herein is subject to change without notice. All rights
reserved.

2. Agenda
3
Part 1 – “Why”
• Agile-plus-DevOps in the
enterprise
• Customer trends
• Future outlook
Part 2 – “How”
• Custom apps vs. packaged
apps
• Successfully adopting Agile-
plus-DevOps for packaged
apps

3. © 2018 FORRESTER. REPRODUCTION PROHIBITED.
September, 2018
Packaged App Testing in the Age Of Agile
and DevOps
Diego Lo Giudice, VP & Principal Analyst, serving AD&D

4. 5© 2018 FORRESTER. REPRODUCTION PROHIBITED.
Leading organizations make SW development their
core competency! Why ?
Digital Matters... Software
Makes
Digital
Successful...
Excellence in Delivering
Great Business and SW
Applications: A Must for
everyone !!!..

5. 6© 2018 FORRESTER. REPRODUCTION PROHIBITED.
End to end agility makes it happen!
Agile
DevOps
Speed, Quality and CX
Common Language
AGILE
Infrastructure
Automation
Infrastructure: Grip, Security
Scaled Infrastructure
AGILE
Development

6. 7© 2018 Forrester Research, Inc. Reproduction Prohibited Source: Forrester Agile Panel Survey 2017 Base: 232
Agile + DevOps is better than just Agile….
Faster Business value
41%
Faster Business value
69%
Improved functional
quality 47%
Improved Functional
quality 72%
Greater frequency of
delivery 75%
Improved technical
quality 29%
Greater frequency of
delivery 83%
Improved technical
quality 72%

7. Testing is right in the middle of the
turbulence and becomes continuous too..
Test ReleaseBuild
MonitorIdeate/Plan
Developers UsersFeedback loop
Delivery Pipeline
Insights
Data/analytics
Agile
DevOps+
Continuous SW Delivery

8. 9© 2018 FORRESTER. REPRODUCTION PROHIBITED.
Continuous delivery => Continuous testing

9. 10© 2018 FORRESTER. REPRODUCTION PROHIBITED.
Agile Is Slowly But Steadily Scaling In Firms
This is a Horizontal and Vertical scale
Source: The State of Agile 2017: Agile at Scale.
Agile + DevOps Slowly But Steadily Reaches Enterprise Scale
2013
2017 2013
2017
25% 75%

10. 11© 2018 FORRESTER. REPRODUCTION PROHIBITED.
Departmental IT or app dev team for mobile applications development
(custom or packaged applications for smartphones or tablets)
Departmental IT for internal applications (packaged application
software like ERP, CRM, PLM, SCM, SRM, finance or business
intelligence)
13%
14%
Central IT for internal applications (packaged application software like
ERP, CRM, PLM, SCM, SRM, finance or business intelligence)
Central IT app dev team for mobile applications development (custom
or packaged applications for smartphones or tablets)
36%
19%
> 50%
What kind of software development is your
organization primarily responsible for?
Source: The State of Agile 2017: Agile at Scale.
Agile + DevOps Slowly But Steadily Reaches Enterprise Scale

11. 12© 2018 FORRESTER. REPRODUCTION PROHIBITED.
Seven scaling
areas of lessons
learned:
Lights and
Shadows
But Agile Expert
Firms do it all
better
Source: The State of Agile 2017: Agile at Scale
Agile + DevOps Slowly But Steadily Reaches Enterrse Scale

12. 13© 2018 FORRESTER. REPRODUCTION PROHIBITED.
Key Metrics that
increase agility,
and Agile Expert
firms have
higher scores…
Source: The State of Agile 2017: Agile at Scale
Agile + DevOps Slowly But Steadily Reaches Enterprise Scale

13. 14© 2018 FORRESTER. REPRODUCTION PROHIBITED.
Continuous
Testing, lessons
from Expert
Firms
Source: The State of Agile 2017: Agile at Scale.
Agile + DevOps Slowly But Steadily Reaches Enterprise Scale
Experts
Neophytes

14. 15© 2018 FORRESTER. REPRODUCTION PROHIBITED.
“In three to five years, the TCoE as we know it will change:
shifting to become practice centers” (January 2013)
Source: January 15, 2013, “Consistent Performance In Agile Teams Must Include Testing” Forrester report, Diego Lo Giudice
From:
Centralized
To:
Federated
Packaged Apps:
- Regression Testing
- End-to-end Testing

15. 16© 2018 FORRESTER. REPRODUCTION PROHIBITED.
The new state of
Testing in
Agile+DevOps
packaged app
delivery
Ruthless Automation
Shift Left – QA Built in
New Organization & Roles
Performance testing shifts left
SVT automates integration testing
Security Testing gets automated
Test Data Management
Artificial Intelligence/ML

16. 17© 2018 FORRESTER. REPRODUCTION PROHIBITED.
For automation,
technology does
matter !.........
And for
Packaged apps
technology
requirements
can be different
SAP, Oracle, Salesforce…
CI/CD technology
Testing persona
Areas of Differentiation to look for
API Testing (protocols, etc)
More data driven testing
SaaS
Partner strategy
AI / ML

17. 18© 2018 FORRESTER. REPRODUCTION PROHIBITED.

18. 19© 2018 FORRESTER. REPRODUCTION PROHIBITED.

19. 20© 2018 FORRESTER. REPRODUCTION PROHIBITED.

20. 21© 2018 FORRESTER. REPRODUCTION PROHIBITED.
Can AI improve
the way we test?
Augment and
automate ?

21. 22© 2018 FORRESTER. REPRODUCTION PROHIBITED.
Most GSIs, new startups and even existing test tool
vendors are experimenting new approaches by
leveraging ML, DL and NLP.
To test smarter and
automate less
Augment testers with
better insights
Optimize and
generate test cases
Increase Automation
generation
Prevent bugs besides
minimize MTTR
Optimize test data
Automate CX testing
………..and more

22. 23© 2018 FORRESTER. REPRODUCTION PROHIBITED.
Image source: https://blaesbjerg.com/den-mentale-liste/
If you think that being
in the packaged or
business apps world
you can’t be agile, test
better and faster, think
more…

23. FORRESTER.COM
Thank you
© 2017 FORRESTER. REPRODUCTION PROHIBITED.
Diego Lo Giudice
+39 340 7710775
dlogiudice@forrester.com
@dlogiudice

24. How to Adopt Agile-plus-
DevOps for Enterprise
Applications
Chris Kraus, Sr. Director of Product Management

25. 11/9/2018© 2017 Worksoft, Inc.
Today’s Modern Enterprise Applications.
Houses or Cities?

26. Release Train for Enterprise Operations
Multiple ALM platforms puts the pressure on Continuous Testing
Mobile
Change
ERP
Change
Web
Change
CD Continuous Delivery - Release Train
DEVELOP
GROOM STORIES BACKLOG
CODE CHECK INSPRINT STARTS
RUN
TEST EXECUTION TEST AUTOMATION
CHANGE IMPACT
ANALYSIS
GO LIVE
DEVELOP
GROOM STORIES BACKLOG
CODE CHECK INSPRINT STARTS
SolMan
Panaya
JIRA Project
NewWeb
JIRA Project
NewDevice
Multiple systems for code check in
Subversion
GitHub
SAP Transport
ERP rollouts in different ALM products
More PPM is observed
Sales Order to Cash Business process

27. No Requirements.
No Consolidated Change Lists.
• Understanding of a process lies with the
business
• A single process can include 100s of steps
• Multiple sprints working in parallel but not
on the same board
• No easy way to map from requirements >
source control > change lists > test
management system
28
“We struggle with knowing exactly what to test and exactly are business processes
are. With the adoption of the Fiori interface and SuccessFactors, we see things only
becoming more complicated.”
- SAP Functional Lead
Key Considerations

28. Consider Agile Testing in the Traditional DevOps World
29
Application Owners / DevTest Pre-Prod Prod
Sprint Team
“The only thing more
dangerous than a developer
is a developer conspiring
with Security.”
― Gene Kim, The Phoenix Project: A Novel About IT, DevOps,
and Helping Your Business Win

29. Key Considerations
End-to-End Business Process Testing
• Ownership of test creation and maintenance
• Test reuse/scalability – ability to feed CI/CT cycles
• UI version changes
• Compliance and audit requirements
• Impact on upstream and downstream system
30
“Our biggest concern is down time. We currently schedule at least 40 hours maintenance a
month and we are a 24/7 business. We have too many integrations with SAP today. One of
our biggest challenges is testing. Today, we have to get a bunch of people together to
build test scripts, then get those people back together in a room to execute tests, this is a big
waste of time.”
- Business Application Manager

30. Modern Apps. Modern COE.
Rise of the New Federated Testing Center of Excellence
31
Plan Prepare/Explore Realize: Build Test Deploy Run
Process
Owner
Sprint Team
Federated Testing
COE
Operations
• Business Owner
• Portfolio Management
(LPM)
• Epic Owners
• Enterprise Architect
• Dev Team
• Scrum Master
• Product Owner
• Test Automation
Specialists
• Sprint teams work only on
small stories and do not see
the larger epic
• Sprint teams work in dev
environments with limited
data
• End-to-end automation
specialists kept as centralized
resources
• Testing works with pre-production
systems testing across full
systems with supporting data
Enterprise
Problems

31. The Second Loop: Continuous Testing
32
Realize: Build Test Deploy
N+1 Automation
Functional experts run capture to document the stories.
Once a feature is complete, test automation professionals can finish the
automation and schedule into continuous testing cycles.
Sprint 1 - 3
Sprint 4 - 6
Unit Tests
Unit Tests
Functional
Test
Functional
Test
Functional
Integration Test
Sprint 7 - 10
Sprint 11 - 14
Unit Tests
Unit Tests
Functional
Test
Functional
Test
Functional
Integration Test

32. Scale to Continuous Testing
33
Plan Prepare/Explore Realize: Build Test Deploy Run
Automation Library

33. Demonstration

34. DevOps Tool Chain
• What technologies will you
need
• Who will be using
• What needs to work together
UI Driven
• Logging in and out of system
• Differences in UIs and version
updates
• Orchestration and execution of
tests on demand at scale
35
“We need to extend ERP testing thru CI/CD so we can release changes at the pace we need
to.. Sr. management wants us to be more agile and ensure changes don’t break things
upstream or downstream from where that change took place.”
- Sr. Mgr. Technical Services
Key Considerations

35. Tool Chain Complexity – Many Tools Working Together that
Need to be Integrated
36
Requirements
Management
Defect Reporting
Test Management
Test Data Management
Mobile Device Lab
Test Orchestration
Continuous Integration Continuous Testing
Application Lifecycle Management (ALM)

36. Demonstration

37. Created a robust COE
as a foundational capability
to establish process and
framework.
Adopted agile methodology
and made “automation
everyone’s job”.
Consolidate high
maintenance SAP tests
from HP UFT and Turnkey
cFactory to Certify.
Analyze Capture
leveraged to accelerate
automation creation for
Manhattan.
Automation extended to
RPA and used for SAP
mass configuration.
Builds a Foundation for
Continuous Testing and RPA
GOAL
Continuous Testing &
Integration and RPA
40% reduction in automation maintenance.
60% reduction in regression test execution.
90% automated in sprint test coverage.
Sources: SAP SAPHHIRE NOW 2018 Presentation
and SAP Insider Article Fall 2018
“A Testing Center of Excellence (COE) should be geared toward mission-
critical applications- focused on your most vital projects. The COE
enables execution & shares best practices.” – Nikhil Shah
Cardinal Health is a Fortune 100
health care company serving more
than 100,000 locations world wide.
85% reduction in
mass configuration
efforts.

38. Related Resources
• Info Page for Capture 2.0:
https://www.worksoft.com/products/capture
• DEMO: Daily Continuous Testing with Worksoft and Jenkins
• DEMO: End-to-End Agile Testing for SAP with Worksoft Certify
• ALM Integration Support
• HP ALM: https://www.worksoft.com/micro-focus-hp-alm-integrations-
support
• SAP Solution Manager: https://www.worksoft.com/testing-framework-
sap-solution-manager
• Worksoft University (Free Online Self-Paced Courses)
39

39. Thank YOU!
Questions?
40