Research on drivers of product development innovation success in a software development environment. Presented at the 2015 Frontiers in Service .

1. What Drives Successful Product Development and Innovation in
the Software Development Process?
The Product Development Success Index (PDSI)
PRESENTED BY:
CHARLES COLBY, ROCKBRIDGE
SUNIL MITHAS, ROBERT H. SMITH SCHOOL
TONY ORLANDO, 3 PILLAR
RIICCARDO DE MARCHI TREVISAN, 3 PILLAR
ERIN NORMAN, ROCKBRIDGE
2015 Frontiers in Services Conference
Rockbridge Associates, Inc.

2. Introduction to 3Pillar Global
2
• 3Pillar Global is a product lifecycle management and
development company that helps clients implement products
that drive revenue.
• Services include strategy and consulting, customer experience
design, platform development, software engineering based on
agile practices, and devops.
• Our clients include mid-sized to large companies in the
media/entertainment, edtech, financial services, information
services, and health and wellness sectors.

3. Background and Environment for this Study
3
• Service sector companies increasingly seek to develop new
software applications to…
• Drive their core business
• Add customer value
• Create new revenue streams
• Create entirely new products that monetize information
captured in their processes
• Efforts are usually driven by a functional area within the company
headed by a Chief Innovation Officer or Chief Technology Officer
• Companies have varying degrees of success in this product
development process

4. Objectives of this Study
4
• Determine what drives success in new software product
development – the focus is on companies where software
development is not their core mission (e.g., a bank, TV network,
healthcare provider)
• Create an instrument that can be used to…
• Evaluate success
• Benchmark success against other companies
• Diagnose issues and identify priorities for improvement
This was a collaborative effort between 3Pillar Global and
the Center for Excellence in Service at the Robert H. Smith
School of Business. Rockbridge Associates was a research
partner that spearheaded much of the effort.

5. Methodology
5
• Exploratory
• In-depth meetings with 3Pillar management/experts
• 6 in-depth interviews conducted by phone with heads of
corporate functions
• Survey of relevant literature
• Creation of a Product Development Success framework
• Quantitative
• 20-minute online survey of 204 professionals responsible for
new software product development (145 from online panel, 59
from 3Pillar lists)
• Respondents worked in range of service businesses
• Had decision input for development function
• Mid-sized to large companies (Minimum $1M)

6. Product Development & Innovation Success Framework
6
Business Outcomes
Macro Factors
• New entrants/products
• Globalization
• Regulation/legal constraints
• Speed of technological
development
• Economy
Long Range Industry Shifts
• Move towards product standardization
• Big Data opportunities
• Search for new revenue streams
• Increasing investment uncertainty
Product Development
Success Index
Perceived Development &
Innovation Success
Organizational
Characteristics
• Revenue
• Size
• Industry
• Geographic reach
• Strategic vision
New Software Product
Development Function
Characteristics
• R&D budget
• Number of employees
• Third party services used
Culture
Feedback
Communication
Staffing
Collaboration
Time/Budget Focus

7. The Product Development
Success Index (PDSI)

8. Survey Topics
8
Topic Measures
Product Development
Success
53 items, scaled on a 1 to 7 scale covering culture,
process and resources
Product Development
Process
Degree of structure, development time,
planning/review cycle, testing, capture of high-value
data, product customization, revenue streams
Macro Factors
Competitive environment, industry growth,
globalization, geographic coverage, regulation
Product Development
Outcomes
Confidence in process, success in achieving dev goals,
innovation leader vs. follower, impact on brand,
impact on efficiency, barriers
Business Traits
Revenue, employment, industry, revenue growth,
customer churn
Function Traits
Employee size, budget, strategy focus, reliance on
outsourcing, decision-making process

9. Index Development
9
 Goal was to create an instrument that was concise, provided
diagnostic detail, and met thresholds for reliability and validity
 Input consisted of the battery of 53 success attributes (49 after
dropping 4 items with high item non-response)
 Exploratory factor analysis used to identify underlying
dimensions of success, account for redundancy, and eliminate
items with low communality
 Resulting index included:
 25 items
 6 dimensions (3 to 7 items per dimension)
 Explained 67% of variance
 Cronbach Alphas ranged from .74 to .89
 Aggregated index had a .69 correlation with perceived overall success of
the product development function

10. 10
7359 87
Highly
Successful
(19%)
Somewhat
Successful
(34%)
Marginally
Successful
(28%)
Rarely
Successful
(19%)
The Product Development Success Index (PDSI) is
calibrated to a scale of 0 to 100
• The PDSI correlates with success in meeting (1) software product
development goals and (2) hard business outcomes, such as
revenue, internal efficiency and customer retention.
• Companies are divided into four tiers for purposes of studying the
correlates and consequences of success.

11. The overall PDSI score is comprised of six dimensions
11
 Culture: attract and retain top talent, have creative leadership,
encourage interaction, provide career growth, and reward and
recognize innovation
 Feedback: degree of input on products collected from customers
and end-users
 Communication: interaction between business and technical
team members as well as within the development groups
 Staffing: appropriate mix of staff from in-house, outsourced,
domestic and foreign workers
 Collaboration: extent to which team members assist each other
and can cut across hierarchy to interact
 Time/Budget Focus: succeeding on traditional project metrics

12. The 6 sub-indices vary in importance
12
Sub-Index Importance to
Product Development Success Index
 Importance is based on correlation with overall perceived success
(normalized pairwise estimates)
 Dimension scores are weighted accordingly
Percentages do not add up to 100% due to rounding. See notes for more exact figures.

13. Across the sample, the biggest pain-point is improving
culture
13
Performance on Sub-Indices vs. Importance
Top Challenge:
Recognizing and
Rewarding Innovation

14. Correlates and Consequences of
Product Development Success

15. What do the successful PD functions look like?
15
Most Successful Least Successful
Large Divisions (1,000+ Staff) Small (<100)
R&D Budget $161M R&D Budget $39M
Larger technology budgets Smaller budgets
Information Services Education and Hospitality
Revenue > Cost Cutting Cost Cutting > Revenue
Sell high margin products Not a focus
Select high quality vendors Not a focus
Business execs & IT
collaboratively drive strategy
Business execs drive strategy
Business execs & IT
collaboratively drive technical
IT drives technical decisions
Seek to lead their industry Tend to follow or track with peers
Confident in Innovation Strategy Not Confident

16. Measures of Agile Development
Q1: For each one, please indicate how well the statement describes your organization.
Q2a: How frequently does your team review and re-plan the next highest priorities for ongoing software development efforts?
Q2c: How frequently during the development process do end users test working software, including new features?
Q3a: How often do you discuss and/or implement improvements to your software development process?
Q2: How long does it take in your organization to go from an idea to software that is working in a production environment?
Q2b: What percentage of your new software, including new features, is covered by automated tests that run every time a feature is
added or modified? 16
Component Frequency/Rating
Time from idea to working software Three weeks or less
New software covered by automated tests 76%+
Frequency of review and re-planning
priorities
Every three weeks or more often
Frequency of end user testing Every three weeks or more often
Frequency of process improvement Every three weeks or more often
Team Members Exchange Information and
Learn From Each Other
Describes company (6 or 7 on a 7-
point scale)
Business and Technical Teams Collaborate
of Software Development Projects
Describes company (6 or 7 on a 7-
point scale)

17. Agile practices correlate strongly with PD success
17
Number of Agile Practices Used
2% 1%
8%7% 11%
18%14%
8% 7%
17%
24%
18%
10%
23%
16%
21%
26%
13%
25%
34%
26%
20%
28%
32%
14%
3%
14%
41%
14%
6%
Total
(n=204)
Rarely
Successful
(39)
B
Marginally
Successful
(57)
B
Somewhat
Successful
(70)
C
Highly
Successful
(38)
D
0 Agile Practices
1 Agile Practice
2 Agile Practices
3 Agile Practices
4 Agile Practices
5 Agile Practices
6 Agile Practices
7 Agile Practices
Q1: For each one, please indicate how well the statement describes your organization.
Q2a: How frequently does your team review and re-plan the next highest priorities for ongoing software development efforts?
Q2c: How frequently during the development process do end users test working software, including new features?
Q3a: How often do you discuss and/or implement improvements to your software development process?
Q2: How long does it take in your organization to go from an idea to software that is working in a production environment?
Q2b: What percentage of your new software, including new features, is covered by automated tests that run every time a feature is added or
modified?
*Caution, small base size
AB
A
D
D
CD
BC

18. Companies need to cross thresholds on their way to
success – consensus, financial, knowledge
Q34: Why does your organization not move faster in software product development and technology? (Select all that apply)
*Caution, small base size 18
Constraints on Speed of Software Product Development
Total
(n=204)
Rarely
Successful
(n=39*)
A
Marginally
Successful
(n=57)
B
Somewhat
Successful
(n=70)
C
Highly
Successful
(n=38*)
D
Financial
constraints
47% 51% 58%D 41% 34%
Disagreement
among decision
makers
29% 49%BC 23% 23% 29%
Lack of time 29% 31% 30% 30% 24%
Ineffectiveness of
consensus
decision making
25% 51%BCD 18% 20% 18%
Fear of risk 23% 41%CD 25% 19% 11%
Lack of vision 19% 41%BCD 12% 16% 11%
Lack of
knowledge
19% 38%BCD 14% 19%D 5%
Legal constraints 19% 13% 21% 20% 21%
None of these 15% 5% 14% 16% 24%A

19. More successful companies operate industries with more
dynamic competitive activity
Q9: How frequently do the following actions occur in your industry? Seven-point scale from “Never” to “Constantly”
*Caution, small base size 19
33%
33%
18%
37%
23%
7%
56%
27%
21%
68%
55%
32%
Software products with brand new
capabilities are released
There are new entrants to the industry
Firms leave the industry
Rarely Successful
(n=37-38*)
A
Marginally
Successful(n=56-57)
B
Somewhat Successful
(n=69)
C
Highly Successful
(n=36-38*)
D
Industry Changes
(% rated 6-7 on a 7-point frequency scale)
AB
AB
BC
B
B

20. Perceived Development &
Innovation Success

21. PDSI correlates with success on a variety of goals
connected to new software product development
Q16: How successful is your organization in achieving the following goals? *Caution, small base size 21
Index Score versus Software Product Development Goals
(% rated 6-7 on a 7-point success scale)
Total
(n=196-204)
Rarely
Successful
(n=38-39*)
A
Marginally
Successful
(n=52-57)
B
Somewhat
Successful
(n=69-70)
C
Highly
Successful
(n=37*)
D
Overall success of new product
innovation
37% 8% 18% 43% AB 87% ABC
Capabilities meet customer needs 42% 10% 21% 59% AB 74% AB
Stay competitive with dev 42% 5% 24% A 56% AB 84% ABC
Products competitive in market 41% 10% 23% 50%AB 82%ABC
Keep software products current 37% 3% 25% A 46% AB 76% ABC
Products create new revenue 31% 5% 19% A 37% AB 68% ABC
Quickly roll out new products 31% 8% 14% 33% AB 76% ABC
Based on radically new tech 30% 5% 16% 29% A 79% ABC

22. Product development success also tracks with business
outcomes like revenue growth & customer retention
22
Number of Positive Business Outcomes
15%
5%
14% 10%
34%
20%
8%
19% 29%
16%
23%
23%
18%
27%
21%
19%
31%
18%
17% 13%
16%
21% 21%
11% 11%
8% 13% 11% 6% 5%
Total
(n=204)
Rarely Successful
(n=39*)
A
Marginally
Successful
(n=57)
B
Somewhat
Successful
(n=70)
C
Highly Innovative
(n=38*)
D
0 Positive Outcomes
1 Positive Outcome
2 Positive Outcomes
3 Positive Outcomes
4 Positive Outcomes
5 Positive Outcomes
Q18: What impact do new software products developed by your organization have on the reputation of your brand?
Q19: How, if at all, have software products developed by your organization in the past three years affected the internal efficiency of your
organization?
Q20: How would you describe the change in your company’s total revenue over the last three years?
Q21: How would you describe the change in your company’s total number of customers over the last three years?
Q22: Thinking of the last 12 months, which best describes the change (if any) in your company’s customer retention rate?
*Caution, small base size
ABC
A

23. Key Takeaways

24. The overall PDSI score is comprised of six dimensions
24
 Key opportunity is to build a culture of talented people where
innovation is rewarded
 Most successful companies tend to be larger, better funded,
focused on quality, and strive to be leading edge
 Success is related to a dynamic competitive landscape
 Agile development practices truly matter
 Success requires overcoming hurdles: consensus -> financial ->
know-how
 It is possible to score and diagnose success
For more information, visit:
Visit: http://productdevelopmentsuccess.com/

25. Author Contact Information
25
 Charles Colby, Principal, Chief Methodologist and Founder,
Rockbridge Associates, Inc.: ccolby@rockresearch.com
 Sunil Mithas, Professor, Robert H. Smith School of Business &
Research Director, Center for Excellence in Service:
smithas@rhsmith.umd.edu
 Tony Orlando, Executive Vice President, 3 Pillar Global:
tony.orlando@3pillarglobal.com
 Riccardo de Marchi Trevisan, Business Development Executive:
riccardo.demarchi@3pillarglobal.com
 Erin Norman, Director, Rockbridge Associates, Inc.:
ENorman@rockresearch.com

26. Appendix: Detailed Index Scores