Agile Mumbai 2022 - Ashwinee Singh | Agile in AI or AI in Agile?

Agile in AI
OR
AI in Agile
Ashwinee Singh – India Head of Business Agility
ashwinee.singh@ust.com
November 2022
Agile Mumbai 2022
www.agilemumbai.com

Proprietary © 2022 UST Inc
2
INTRODUCTION
 Ashwinee is a seasoned Digital Transformation Leader
with 23 years of IT Industry experience. Ashwinee has
progressed in his career being a Software Engineer, Dev
Lead, Technical Manager, Project Manager, Program
Manager, Delivery Manager, Transformation Manager,
Transformation Director, Agile Transformation Coach and
Director.
 He has been leading and contributing to some of large
scale Agile-DevOps transformations for Fortune clients
across the globe (USA, UK, Australia, Switzerland,
France, Germany and India). He has been employed
with IBM, Capgemini, Cognizant, and Infosys in past.
 Ashwinee currently heads UST’s Business Agility
Practice for India and Asia as Practice Director.

3
SESSION TOPICS
 The AI Revolution
 Challenges building AI applications
 Does Agile have a solution
 Building ML Model
 Adopting Agile for building ML Model
 Challenges applying Agile for ML
 Industry approaches of applying Agile for ML
 Evolving an Agile approach for ML
 DevOps / MLOps for Model Lifecycle Management
 How AI is helping with Agile-DevOps implementation

5
AI ADOPTION INDUSTRY TREND
Source: DataRobot Source: IBM’s Global AI Adoption Index 2021

6
Source: www.forbes.com
AI PROJECTS FAILURE RATE !
Gartner: 85% of all AI projects will fail to
deliver outcomes in 2022
MIT-SMR: 70% of companies report minimal
to no impact from AI
MIT-SMR: Less than 2 out of 5 companies
reported business gain in last 3 years
VentureBeat: 87% of DS projects never make
it to production
Tom Siebel: 99% of internal AI projects fail

7
WHAT IS CAUSING AI PROJECTS TO FAIL ?
Source : www.cognilytica.com
Applying application development approaches to data-centric AI
Lack of sufficient quantity of data
Lack of sufficient quality of data
Underestimating time and cost of the data component of AI projects
Lack of planning for continued AI, model, data iteration and lifecycle
Misalignment of real world data and interaction against training data and models
Applying proof of concept thinking to real-world pilots
ROI Misalignment of AI solution to problem
1.Vendor misalignment on promise vs. reality
1.Overpromising AI capabilities and underdelivering projects

8
WHAT ARE MAIN AI MODELLING ISSUES FACED BY BUSINESSES ?
66%
Lack of clarity on
provenance of training
data
64%
Lack of
collaboration across roles
involved in AI model
development and
deployment
63%
Lack of AI policies
63%
Monitoring AI across cloud
and AI environments.
Source: IBM’s Global AI Adoption Index 2021

9
WHATS SO DIFFERENT WITH DEVELOPING AI SOLUTION ?
1. Usually many additional layers of unpredictability and unknowns
2. The key focus remains data exploration and insight generation and not just the application development
3. Typical application development solution is built around business rules however data science solution
solves problems differently (i.e., not dependent on business rules)
4. Highly evolving and exploratory nature makes it difficult to predict timelines
5. Most of the time goes into data engineering, feature exploration, etc., which is hard to measure tangibly
6. Change in data quality and sufficiency have a significant impact on the end result / outcome
7. Higher complexities of skill set requirements
Data Analyst, Expert Analyst (SMEs), Data Engineer, Data Scientists, Analytics / DS lead, Product
Manager / Owner, UX designer, ML engineer, etc.
8. The result of Proof of Concept/prototype may vary from the result of a real-world project due to many
factors, and interdependencies involved.
9. Defining problem and approach is a complex and iterative process and hard to define clearly upfront
10. Success of AI projects requires planning for continuous management (i.e., monitoring, evaluation,
continuous training, deployment, etc.)
You may not need AI if the solution can be described in a flowchart or with a set of simple
heuristics

10
IMPORTANCE OF WORKING WITH DATA
Source - COGNILYTICA
“Over 80% of the time enterprises spend on AI
projects goes toward preparing, cleaning and
labeling data.”
Specifically, the report finds that the many steps involved in
collecting, aggregating, filtering, cleaning, deduping,
enhancing, selecting and labeling data far outnumber the
steps on the data science, model building and deployment
sides.
A recent report from AI research and advisory firm Cognilytica
finds that –

11
THE MACHINE LEARNING MODEL
Machine
Learning
Model
Algorithm
Training
Data
+
“How to learn from Data”

12
APPLYING DATA MODELLING METHOD FOR ML DEVELOPMENT
CRISP-DM 1.0 Method

13
CRISP-DM PHASES FOR DEVELOPING MODEL
Courtesy: IBM

14
THE MACHINE LEARNING LIFECYCLE
Courtesy: IBM DES

15
MICROSOFT’S TDSP LIFECYCLE FOR ML DEVELOPMENT
Microsoft’s Team Data Science Project (TDSP) Lifecycle

16
CHALLENGES ADOPTING AGILE IN BUILDING AI SOLUTION
Could not leverage empirical data for estimation and general decision
making
1.AI projects do not follow a linear, predictable path
Could not forecast number of tasks or size of task / activity hence could not
really apply traditional Scrum by estimating and planning Sprint
Fixed duration Sprint timeboxing does not wok due to varying degree of
logical work breakups requiring constant exploration and experimentation,
hypothesis testing, requires research and learning
ML modeling is stochastic where the outcomes are characterized by
probabilities so often it is not possible to define end deliverable / outcome
The measure of success is hard to define upfront. AI / ML models and their
components (code, trained data, parameters, hyperparameters, etc.) are not
end objective, but they are just enablers in delivery of a suitable ML solution
The AI-ML Model isn’t built around expectation of satisfying customer / user
needs rather to see how given data could be best utilized to help with any
business needs which often is not known in advance

17
ADAPTING SCRUM TO DATA DRIVEN SCRUM
Source: www.datadrivenscrum.com

18
Generic Tasks of the Improved Process Model
EVOLVING AGILE AI PROJECT MANANGEMENT APPROACH

19
DESIGNING PRINCIPLES FOR BLENDED AGILE METHOD
Flexible length Sprints working in Kanban fashion:
o Sprints may be too short to work out a messy modeling algorithm, but too long if exploration quickly indicates a need to pivot
o Small increments with systematic, frequent validation are required to assess the degree to which it is addressing the business
problem.
o If necessary, rapidly abandon one concept and pursue another, communicate and set the expectation from the solution,
including the degree to which it is addressing the problem
o Much of the work is far removed from end-users, so feedback must be gathered in many forms, not just sprint demos
All the phases of the project should be mutually iterative; progression
backward or forwards should be allowed (Unlike CRISP-DM and other
approaches)
o (e.g., it is highly likely that unknown problems of data
understanding are identified while making the model, the approach
should allow going back to the data understanding phase)
AI data projects should be approached from the top-down and bottom-up
directions. The outcomes of AI efforts depend on finding the overlap
between
o What is possible based on the information that can be extracted from
the underlying data?
o What is desirable, based on identifying actionable, timely decision
support for business stakeholders?
 We suggest a blended methodology that adopts the relevant and fitting aspects of Agile and other approaches
 We can use phases and iterative approaches of the data-centric methodologies like CRISM-DM and extend it for each business
requirement / customer need
 Use of DataOps, MLOps for automation and management of data sets and algorithmic models and the entire process

20
EVOLVING AGILE METHOD FOR BUILDING ML MODEL
Product Owner
Data Scientist
Agile-ML Process Lead
Data Engineers
Data Analysts
Software Engineers
Business Analysts
Definition of
Ready (DoR)
Definition of
Done (DoD)

21
EVOLVING AGILE METHOD FOR BUILDING ML MODEL
Product Owner
Data Scientist
& Data Engineers
Agile-ML
Process Lead
 Defines all data related backlog items
 Defines acceptance criteria for data
related PBIs
 Defines all non-data related backlog
items
 Defines Business Value and
prioritization
 Works as process orchestrator for the
entire Team
 Manages flow of work and maintains
Kanban Task Board Work-in-Progress
(WIP) limits
 Organises and moderates all team
events and maintains team cohesion
Definition of
Done (DoD)
Definition of
Ready (DoR)
Kanban Task Board

22
KEY PRINCIPLES AND PRACTICES OF AGILE METHOD
 Team to practice Agile ScrumBan approach based on
milestone activities which varies very much in terms of effort
and time (Example – Data Cleaning, Data Labelling, Model
Evaluation, etc) without any fixed timeboxed sprints
 Team work focus effectively managed by close
collaboration across Troika roles of Data Scientist, Product
Owner and Agile-ML Process Lead
 Integrated Product Item Backlog is
managed by team which has both Data
related items as well Business and other
Technical task items added and tracked
through Kanban Task Board
 Data driven activities are performed
more as exploratory / research activities
with lots of unknowns under Data
Scientist Leadership
 Agile-Scrum events of daily stand-up and Retrospectives
are followed with a cadence
 Agile-Scrum events of Demo and Review are done on
event basis
 Business activities are planned and
worked by team under Product Owner
Leadership with clear focus on
applicability of ML Solution for business
 Model Definition of Done (DoD) is
established which ensures that model is
trained properly, data is correct to
accurately predict the future or in a state
to get a correct answer for the given
business problem it is trying to solve
Product Owner
Data Engineers
& Scientist
Agile-ML
Process Lead
Role Troika

23
COMORBIDITY PREDICTION ML SOLUTION USING AGILE METHOD
Client Context
A large healthcare player based in Indiana, US
Solution Objectives
• Predict progression of
comorbidities in members
with:
• Type-2 Diabetes Mellitus
(T2DM),
• Chronic Kidney Disease
(CKD)
• End Stage Renal
Disease (ESRD)
• Identify members at high risk
of developing comorbidities
Solution Approach
• Solution developed using historical claims, member diagnosis conditions, drug refill
history, member laboratory tests, and member demographics information to
understand the risk of developing a comorbidity in different time periods
• For each primary disease, set of relevant comorbidities are chosen based on
exploratory data analysis and clinical research inputs
• Developed 85 models to access risk of developing comorbidities for each
combination of primary disease, comorbidity and time period
• Solution provides top contributing factors towards risk score
• Solution Development team comprised of Data Scientist, Product Owner, Process
Lead, Data Engineers and Data Analysts
• Solution built in about 16 weeks timeframe leveraging Agile-Kanban ways of working
Benefits Realization - The developed solution helped our client to :
• Accurately identify members with high risk of developing comorbidities and the underlying factors with 90%
accuracy
• Implement targeted focused interventions at a member-level
• Reduce cost of expensive procedures Courtesy: Abzooba

24
FRAMEWORK FOR BUILDING ML SOLUTION
Courtesy: xpresso.ai
1. Define the problem and get the data
needed
2. Figure out how you will measure success
3. Figure out how you will evaluate that
success
4. Prepare your data
5. Develop a model incrementally
6. Scale up and improve your model
7. Tune your parameters and regularize your
model

25
DEVOPS / MLOPS APPROACH OF DEPLOYING ML SOLUTION
Courtesy: xpresso.ai

26
AI TO HELP AGILE-DEVOPS IMPLEMENTATION
Courtesy: ourcodeworld.com

27
APPLYING AI FOR AGILE-DEVOPS IMPLEMENTATION
Product
Development
Product
Discovery
1
2
3
4
1
2
3
4
NLP being used to write User Stories and
establish initial Product Backlog
(https://userstorygenerator.ai/).
Initial Product Backlog
word2vec, paragraph2vec, Long Short-Term
Memory (used in Google Translate), or
Convolutional Neural Networks (used in
Facebook’s DeepText engine) can generate
dense vector representations that produce
superior results on various NLP tasks
Deep Learning as Tool
NLP component which performs automatic
analysis on textual artifacts and then generates
vector representations of those artifacts
NLP as Tool
AI generated data can be compiled and
summarised to provide product owners and
other business stakeholders insights,
planning and prioritizing features and bug
fixes for future releases
Insights for PO
Code Modelling component is responsible
for learning vector representations which
reflect the semantic and syntactic structure
of source code and is used often in IDEs
Code Modelling
Machine learning applications can absorb data
streams from various DevOps Telemetry Tools
to find correlations & generating more
insightful view of the application’s overall
health and useful foresights
Telemetry Analysis
Applying AI or machine learning algorithms
to these different type of testing results could
identify patterns of poor coding practices that
result in too many errors caught by the tests
Testing Augmentation
Sophisticated Code Generator tools like
GitHub’s Copilot are generating as much as
30% of new code by using AI for some
programming languages
Code Generation
Product Backlog

28
Summary
KEY TAKEAWAYS ON COUPLING AI AND AGILE
 AI-ML Applications development is too data-centric and hence does not get
benefitted by just following the regular Agile methods which have worked
well for Software Development
 Evolving Agile model around ScrumBan approach inspired from CRISP-DM,
Data-Driven-Scrum, IBM-DES and Microsoft’s TDSM seems promising and
works better with contextualization
 Formalizing key Troika roles of Data Scientist, Product Owner and Agile-ML
Process Lead helps keeping the right focus and balance across Data and
Application Development activities thus increasing the business benefits
 Figuring out your measures of success while developing ML models is crucial
 The practices - AIOps and MLOps - both play a significant role in aiding
businesses in achieving operational efficiency. MLOps brings agility by
bringing machine learning model into production and managing it, whereas
AIOps brings agility by using AI-ML and big data to automate IT / project
operations. Hence, they both contribute to “Agile in AI and AI in Agile.”)
 Increasing number of AI-ML techniques like NLP are proving very useful in
increasing the effectiveness of Agile-DevOps implementations

Thank you
ust.com
Ashwinee.Singh@ust.com

30
01
About UST

31
At UST, we believe in the power of
technology to engineer a better future
22+
30+
30K
Years in business
We have been
bringing technology
to life for our clients
for over 20 years
Countries
We operate in 30+
countries with over
34 delivery centers and
42 operating centers
Employees
We have over 30,000
associates committed
to your success
140+
7+
We are privately held with
an investment from one of
the largest PE funds in the
world
Clients
We serve over 140 Global
1000 clients
Industries
Healthcare
Life Sciences
Retail & CPG
Semiconductor
Manufacturing
Financial Services
Technology, Media
& Telecom
Travel & Hospitality

32
Together, we create
successful outcomes
for our customers
• We believe in building long term partnerships
with clients
• More attention paid to client success at all levels
• Commitment beyond contract – to deliver
business outcomes
• Flexible contractual models
• Consistent executive attention
• Dedicated account team
• Client feedback drives investments up front and
throughout the journey
32
13Average tenure of client relationships

33
A digital leader with a robust portfolio of solutions
Leader in digital services focusing on tangible outcomes through meaningful innovation
CUSTOMER
EXPERIENCE & AGILE
• Business Agility Services
along with distributed Agile
Development
• Digital solutions with
human-centered design
• Intelligent process
automation
ANALYTICS
• World class data
engineering skills
• Artificial Intelligence
powered solutions for deep
insights and predictive
capabilities
• Machine Learning for
automating functions and
conversion of data to
actionable insights
CYBERSECURITY &
BUSINESS RISK
• AI driven vulnerability
assessment for threat
detection and scoring
• Rule driven threat
remediation
recommendation and
automated resolution
• Selective business risk
analysis and solutions
LEGACY
MODERNIZARION &
CLOUD ENABLEMENT
• AI powered application
analysis and mapping tools
• Infrastructure and
application cloud
migration capabilities as
well as tools to assess and
optimize efficiency of an
environment
• Methodology and tools for
Cloud Native for rapid
application delivery
INNOVATION &
EMERGING SOLUTIONS
• Innovation Pods for rapid
and repeatable innovation
• Use of emerging
technologies (blockchain,
quantum computing,
AR/VR) to address business
needs
• Comprehensive digital
product design solutions
for new products and
services
Comprehensive portfolio of services for DESIGN, BUILD and OPERATE (“DBO”)

34
Global business agility services
• UST Agility Consulting
centers are located in the
US, Europe, Australia, and
Latin America with
engagements around the
world
• Agile Solution centers
operate out 15+ countries
• 150+ Agile/DevOps
Coaches, 3500+ Scrum
Master and thousands of
practitioners
• Agile leadership in each
region works through a
unified vision and integrated
approach
Aliso Viejo
Calabasas
New York
Norfolk
Atlanta
Dallas
Chicago
Pittsburgh
Bentonville
Austin Toronto
London Ireland Geneva Madrid
Copenhagen Oldenburg
Singapore Shanghai
Penang Taiwan Hong
Kong Tokyo
Trivandrum
Kochi
Coimbatore
Bangalore
Chennai
Pune
Mumbai
Delhi
Sydney
Active Agile
engagements in:
Chile Argentina
Venezuela Peru
Colombia
Active Agile engagements in:
Amsterdam London Leeds Glasgow
Madrid Barcelona Budapest
Copenhagen
Active Agile engagement
in: Trivandrum Kochi
Chennai Bangalore
Active Agile
engagement in:
Mexico City
Guadalajara Leon
Costa Rica
Active Agile
engagement
across the US
Turkey
Costa Rica
Active Agile engagement
in : Sydney
USA
India
China
Spain
Germany
Mexico
Singapore
Philippines
Malaysia
Taiwan
Turkey
Colombia
Australia
UK

35
Copyright and confidentiality notice
Copyright © 2022 by UST Global Inc. All rights reserved.
This document is protected under the copyright laws of United States, India,
and other countries as an unpublished work and contains information that shall
not be reproduced, published, used in the preparation of derivative works, and/or
distributed, in whole or in part, by the recipient for any purpose other than to
evaluate this document. Further, all information contained herein is proprietary
and confidential to UST Global Inc and may not be disclosed to any third party.
Exceptions to this notice are permitted only with the express, written permission
of UST Global Inc.
UST® is a registered service mark of UST Global Inc.
UST
5 Polaris Way
Aliso Viejo, CA 92656
T +1 949 716 8757
F +1 949 716 8396
ust.com

36
Appendix

37
WADING THROUGH COMPLEXITY
From Stacey’s Complexity Matrix
Simple Domain
• Everything is Known
• Sense – Categorize-Respond
Complicated Domain
• More known than unknown
• Sense – Analyze-Respond
Complex Domain
• More unknowns than known
• Probe – Sense – Respond
Chaotic Domain
• Very little is known
• Act-Sense-Respond
Requirements
Technology
Known
Predictable
Certain
Uncertain
Simple
Complicated
Complicated
Complex
Empirical Process
Emergent
Practices
Agile
Servant Leadership
Good Practices
Lean – Six
Sigma
Supervision
Best
Practices
Waterfall
C&C
Unpredictable
Chaotic
Novel Practices
Lean – Build-Measure-
Learn
Entrepreneurship

Agile Mumbai 2022 - Ashwinee Singh | Agile in AI or AI in Agile?

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à Agile Mumbai 2022 - Ashwinee Singh | Agile in AI or AI in Agile?

Similaire à Agile Mumbai 2022 - Ashwinee Singh | Agile in AI or AI in Agile? (20)

Plus de AgileNetwork

Plus de AgileNetwork (20)

Dernier

Dernier (20)

Agile Mumbai 2022 - Ashwinee Singh | Agile in AI or AI in Agile?

Notes de l'éditeur