Presentation "AI Product Manager" at the Digital Product School (on 10/22/2020) from Datentreiber. Content: • Overview over the AI product innovation cycle • AI Thinking: ideating and prioritizing the right use cases • AI Prototyping: testing critical hypotheses with experiments • AI Engineering: building scalable & user friendly AI applications • AI Management: maintaining AI solutions with DataOps • Outlook: how to become an AI product manager (links & more)

1. Martin Szugat @ Digital Product School on 10/22/2020
AI Product Manager

2. Agenda
Overview over the AI product innovation cycle00:15
AI Thinking: ideating and prioritizing the right use cases00:30
AI Prototyping: testing critical hypotheses with experiments00:45
AI Engineering: building scalable & user friendly AI applications01:00
AI Management: maintaining AI solutions with DataOps01:15
Outlook: how to become an AI product manager (links & more)01:30
Q&A01:45
Welcoming: introduction & agenda00:00

4. 4
1996-2008
IT-Consultant, Author and
Software Developer
Study and Research of
Bioinformatics (Data Science)
2001-2008
Managing Director & Shareholder of SnipClip
GmbH (Marketing Agency)
2008-2013
Program Director of the Predictive
Analytics World & Deep Learning World
(Conference Series)
2014-dato
Managing Director & Founder of
Datentreiber GmbH (Consultancy)
2014-dato
Advisory Board for Media & IT
for DDG AG (AI Company Builder)
2020-dato
Martin Szugat
Shareholder of Digitaltreiber GmbH
(Recruitment Agency)
2016-dato
Chief Data Officer & Shareholder
of 42AI GmbH (AI Market Network)
2018-dato

5. 5

6. Agenda
Welcoming: introduction & agenda00:00
Overview over the AI product innovation cycle00:15
AI Thinking: ideating and prioritizing the right use cases00:30
AI Prototyping: testing critical hypotheses with experiments00:45
AI Engineering: building scalable & user friendly AI applications01:00
AI Management: maintaining AI solutions with DataOps01:15
Outlook: how to become an AI product manager (links & more)01:30
Q&A01:45

7. Agenda
Welcoming: introduction & agenda00:00
AI Thinking: ideating and prioritizing the right use cases00:30
AI Prototyping: testing critical hypotheses with experiments00:45
AI Engineering: building scalable & user friendly AI applications01:00
AI Management: maintaining AI solutions with DataOps01:15
Outlook: how to become an AI product manager (links & more)01:30
Q&A01:45
Overview over the AI product innovation cycle00:15

8. 9
AI
Product
Machine
LearningData x =
What is an AI product?

9. 10
Data ProductAnalyticsData x =
What is a data product?

10. What is an AI (Data + Analytics) Strategy?
Accessible Data
AI
Use Cases
Company’s
Objectives
Data / AI Products with a
Business Case

11. Collection of Analytics
Use Cases
(Problem, Solution, Benefit)
Roadmap for Data-Driven
Business Cases
(Costs, Risks, Profits)
Assumptions
(Analytical, Economical, …)
Learnings
(Data, Business, User, …)
Business Value
(Information → Decision → Action →
Impact → Objective)
Data Sources
(Collection, Acquisition, …)
Data
Thinking
Data
Mining
Data Engineering
Data
Management
Data Strategy
Data Prototypes
Data Product
Data SourcesData AssetsData Product
Innovation Cycle

12. Data
Management
Data Engineering
Data
Mining
Data
Thinking
Data, Model &
Product
Management
Data, Software &
UI Engineering
Data
Mining & User
Experiments
Data & Design
Thinking
2. User Under-
standing
(Desirability)
3. Data Under-
standing
(Feasibility)
1. Business
Under-
standing
(Viability)
2. Modelling &
Visualization
3. Evaluation
1. Data
Exploration &
Preparation
3. Learn
1. Build
2. Measure
3. Monitor
1. Deploy
2. Orchestrate
CRISP-DM
Design
Thinking
Proof of Concept (PoC)?
Proof of Value (PoV)?
Lean
Develop-
ment
DataOps

13. Data, Model &
Product
Management
Operating
Data, Software &
UI Engineering
Engineering
Data & Design
Thinking
Data
Mining & User
Experiments
Designing
Experiment-ing
Data LabData Factory
➔ Exploration to
Learn
➔ Exploitation to
Earn

14. 15
Designing
Experiment-ingEngineering
Operating
Data Strategist, AI Translator, …
Canvas, Mockups, …
Design Thinking, Sprints …
Data Scientist, UX Designer, …
Data Analytics, Modelling, …
CRISP-DM, Kanban, …
Data Steward, Product Manager, …
Monitoring, Audits, …
DataOps, SPC, …
Data Engineer, Developer, …
Cloud, MapReduce, …
Scrum, Lean …
Skills, Tools & Methods

15. 16
Designing
Experiment-ingEngineering
Operating
Data Strategist, AI Translator, …
Canvas, Mockups, …
Design Thinking, Sprints …
Data Scientist, UX Designer, …
Data Analytics, Modelling, …
CRISP-DM, Kanban, …
Data Steward, Product Manager, …
Monitoring, Audits, …
DataOps, SPC, …
Data Engineer, Developer, …
Cloud, MapReduce, …
Scrum, Lean …
AI Product Manager

16. Agenda
Welcoming: introduction & agenda00:00
Overview over the AI product innovation cycle00:15
AI Prototyping: testing critical hypotheses with experiments00:45
AI Engineering: building scalable & user friendly AI applications01:00
AI Management: maintaining AI solutions with DataOps01:15
Outlook: how to become an AI product manager (links & more)01:30
Q&A01:45
AI Thinking: ideating and prioritizing the right use cases00:30

17. Holy Grail Use Cases
Everybody wants it. Nobody has it. Some claim to have it.

18. Moonshots
4% of the US state budget was invested in the Apollo program.

19. Lighthouse Use Cases
If you head for the lighthouse, you'll probably shipwreck.

20. Pet Projects
Bosses are usually furthest away from the actions and thus the
relevant information.

21. Boring is the new Sexy.
Look for use cases that sound boring because they often are very subject-specific.

22. Delegate
Form
Check
Stake-
holders on
board
Business
Plan Check
Proof of Concept
(PoC)
Integration Tests
Proof of Value
(PoV)
Ideas
Use Cases
(Drafts)
Business Cases
(Concepts)
Prototypes
Releases
MVDP
*
Meet-ing
Work-
shop
Designing
Experimenting
Engineering
Operating
Use Case Ideation & Prioritization Process
Count Effort
?
Backlog
* MVP: Minimum Viable (Data) Product
Data (Product
Design)
Sprints
(Agile)
Develop-ment
Sprints

23. From Use Cases to Business Cases
User
Under-
standing
Business
Under-
standing
Data
Under-
standing
Users
Problems
Solutions
Benefits
?
Use
Cases
Costs
Risks
Profits
Business
Cases
?
Object-
ives
Results
Actions
Decisions
?
Diverge Converge Diverge Converge Diverge Converge
Viability Desirability Feasibility

24. 1st Day: Overview of Actual
Status & Outlook on Target
Status.
2nd Day: In-depth Look &
Check into the Details.

25. Martin Szugat & Martijn Baker @ Data Brain Meetup:
➔ https://www.slideshare.net/Datentreiber/presentations
➔ https://www.youtube.com/watch?v=U8EbR2gnl_o
Data Strategy Design:
An Open Source Toolbox &
Method for Data Thinking

26. Agenda
Welcoming: introduction & agenda00:00
Overview over the AI product innovation cycle00:15
AI Thinking: ideating and prioritizing the right use cases00:30
AI Engineering: building scalable & user friendly AI applications01:00
AI Management: maintaining AI solutions with DataOps01:15
Outlook: how to become an AI product manager (links & more)01:30
Q&A01:45
AI Prototyping: testing critical hypotheses with experiments00:45

27. Cross Industry Standard
Process for Data Mining
https://en.wikipedia.org/wiki/Cross_Industry_Sta
ndard_Process_for_Data_Mining
LEARN
BUILD
MEASURE

28. Hypotheses-Driven Innovation
Build
Experi-
ments
Measure
Metrics
Learn
from
Obser-
vations
Hypotheses(In-)Validated
Viability
Feasi-
bility
Desir-
ability

30. Assum-
ption

31. Assum-
ption
A demand forecast of accuracy x% will decrease out
of stock situations by y% and thus save the company
z% euros per year.
12/2020
Martin Szugat 2 Month
Build a simple machine learning model
and test it with n users (demand
planners).
Model performance as RMSE as well
as business performance as OoS delta
rate.
Prediction Performance
RMSE < e.g. current estimation
OoS rate > -10% → Saved costs per year = 1M €
➔ Positive estimated ROI for project

32. Assum-
ption

33. Assum-
ption

34. Assum-
ption

35. Assum-
ption Business performance doesn’t scale
with model performance
10.12.2020
Martin Szugat
A better demand forecast prediction will reduce out of
stock situations.
That even if the RSME is improved by 10% the
OoS rate is only decreased by 2%.
Model performance and business performance
doesn’t scale the same level.
Test other machine learning approaches to improve RSME by
x%.

36. Assum-
ption

37. Assum-
ption

38. Assum-
ption

39. Assum-
ption
Canvas
Research
Prototype
Pilot
…
Product
Explo-ration

41. Hypothesis, Experiments & Learnings Database (HELD):
➔ https://dtbr.de/held

42. Agenda
Welcoming: introduction & agenda00:00
Overview over the AI product innovation cycle00:15
AI Thinking: ideating and prioritizing the right use cases00:30
AI Prototyping: testing critical hypotheses with experiments00:45
AI Management: maintaining AI solutions with DataOps01:15
Outlook: how to become an AI product manager (links & more)01:30
Q&A01:45
AI Engineering: building scalable & user friendly AI applications01:00

43. PoC Trap
• Data & technology faith
• “Throwing over the fences” phenomena
• “Not thought through to the end” mindset

44. Value Pipeline
Business
Value
Analytics in
Production
Data Sources
PoC
Concept
Idea
InnovationPipeline
Exploration vs. Exploitation / Learn vs. Earn
Operation: Maintain
Innovation: Change
Clash of Interests &
Culture!

45. Titel
Text
Zoom in

46. Experiment vs. Test
Source: https://medium.com/data-ops/dataops-is-not-just-devops-for-data-6e03083157b7
Test
Experiment

47. 48
Exploration Stage
Gold Standard Data Sets
Analytics in
Production
Data Lakeland
Validation Stage:
Real World Data Sets
Production Stage
“Real Time” Data Sets
Moni-
toring
Analytics in
Development
Analytics in
Experimentation
Frequent
Exports
Sporadic
Exports
Sandboxes

48. 49
Testing. Testing. Testing.
Source: https://martinfowler.com/articles/cd4ml.html

49. 50
Continuous Deployment & Integration
Source: https://martinfowler.com/articles/cd4ml.html

50. Agenda
Welcoming: introduction & agenda00:00
Overview over the AI product innovation cycle00:15
AI Thinking: ideating and prioritizing the right use cases00:30
AI Prototyping: testing critical hypotheses with experiments00:45
AI Engineering: building scalable & user friendly AI applications01:00
Outlook: how to become an AI product manager (links & more)01:30
Q&A01:45
AI Management: maintaining AI solutions with DataOps01:15

51. 52
Throwing over the fences.

52. Designing
Experiment-ingEngineering
Operating
Innovation Lab
Data LabData Factory
IT

53. You build it.
You fix it.

54. Designing
Experiment-ingEngineering
Operating
DataOps
You design it.
You test it.
You build it.
You fix it.
Data Strategists
Data Scientists
Data Engineers
Data Stewards

55. 56
DataOps is NOT Just DevOps for Data
Source: https://medium.com/data-ops/dataops-is-not-just-devops-for-data-6e03083157b7

56. Eat your own dog food: analytics for analytics.

57. Source: https://www.slideshare.net/PAWDeutschland/data-science-development-lifecycle-everyone-talks-about-it-nobody-really-knows-how-to-doit-and-everyone-else-is-doing-it

58. Agenda
Welcoming: introduction & agenda00:00
Overview over the AI product innovation cycle00:15
AI Thinking: ideating and prioritizing the right use cases00:30
AI Prototyping: testing critical hypotheses with experiments00:45
AI Engineering: building scalable & user friendly AI applications01:00
AI Management: maintaining AI solutions with DataOps01:15
Q&A01:45
Outlook: how to become an AI product manager (links & more)01:30

59. Non-Linear Data Product Innovation Process (Cycle of Cycles)
Designing Engineering Operating
5Prototypes
20Concepts
3Products
100Ideas
1System
Experiment-
ing
Unit Data Strategy Data Lab Data Factory Data Operations
Back to Backlog Back to Backlog
PoC
PoV
Tests
AI Product Manager

60. Raw
Data
Clean Data
Value Pipeline
Anomaly
Detection
PoC
Concept
Idea
InnovationPipeline
XYZ Prediction
PoC
Concept
Idea
InnovationPipeline
Business
Value
Data & Analytics Pipelines

61. From Data Craftsmanship …

62. … to a Data Industry.

63. AI Product Manager
Analytical
Technical
Business
Design
Thinking
Product Design & Management
DataOps
Scrum / Kanban
Data & Software
Architecture
Data Management &
Governance
Machine Learning
Statistics
CRISP-DM
AI Governance
Business
Analyses
Data Visualization &
Storytelling
Soft Skills: Moderation,
Mediation, Negotiation, ..
CI / CD
DevOps
UI / UX
Lean
Management

64. 65
Further literature
1. Data Strategy & Data Thinking
1. Design thinking for data products
2. Data Strategy: Good Data vs. Bad Data
3. How to Define and Execute Your Data and AI Strategy
4. See next slide
2. Data Science Development Process:
1. Data Science at Roche: From Exploration to Productionization
2. Data Science Development Lifecycle
3. DataOps / ModelOps / AIOps
1. DataOps is NOT Just DevOps for Data
2. The DataOps Cookbook
3. Introducing ModelOps To Operationalize AI
4. Monitoring Machine Learning Models in Production
5. Continuous Delivery for Machine Learning
4. AI Product Management
1. A step-by-step guide to becoming a Data Product Manager
2. Managing Data Science as Products
3. What you need to know about product management for AI
4. Practical Skills for The AI Product Manager
5. Bringing an AI Product to Market
5. Other
1. The New Business of AI (and How It’s Different From
Traditional Software)
2. When is AI not AI?

65. Get started.
• Designkit: http://dtbr.de/designkit
• LinkedIn Group: http://dtbr.de/data-thinker
• Video training: http://dtbr.de/ddm
• Interactive trainings: http://dtbr.de/training
• News: http://dtbr.de/twitter
• Presentations: http://dtbr.de/slideshare
• More: https://www.datentreiber.de

66. Source:
https://medium.com/womeninai/can-
artificial-intelligence-solve-my-business-
problem-4ff3bcbffe32

67. Agenda
Welcoming: introduction & agenda00:00
Overview over the AI product innovation cycle00:15
AI Thinking: ideating and prioritizing the right use cases00:30
AI Prototyping: testing critical hypotheses with experiments00:45
AI Engineering: building scalable & user friendly AI applications01:00
AI Management: maintaining AI solutions with DataOps01:15
Outlook: how to become an AI product manager (links & more)01:30
Q&A01:45

69. datentreiber.deWir treiben Ihr Unternehmen voran.
Web: www.datentreiber.de
Blog: www.datentreiber.de/blog/
Martin Szugat
Geschäftsführer
Telefon: +49 [0]881 12 88 46 53
Email: ms@datentreiber.de