Recommended
Recommended
More Related Content
Similar to Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and AI for better Products, Customer experience and Resilient Platform
Similar to Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and AI for better Products, Customer experience and Resilient Platform (20)
More from AgileNetwork
More from AgileNetwork (20)
Recently uploaded
Recently uploaded (20)
Agile Mumbai 2022 - Balvinder Kaur & Sushant Joshi | Real-Time Insights and AI for better Products, Customer experience and Resilient Platform
- 1. © 2021 Thoughtworks | Confidential Real time insights and AI for better products, customer experience and resilient platform Balvinder Khurana and Sushant Joshi www.agilemumbai.com
- 2. Who we are 2 Balvinder Khurana Principal Consultant Data Architect and Global data community lead Sushant Joshi Product Principal @sushantjoshi https://sushant-joshi.medium.com Balvinder has 15 years of experience in building large- scale custom software and big data platform solutions for complicated client problems. She has extensive experience in Analysis, Design, Architecture, and Development of Web based Enterprise systems and Analytical systems using Agile practices like Scrum and XP. Balvinder currently works as a Data Architect and Global Data Community Lead for Thoughtworks Sushant is a Product Principal at ThoughtWorks. His work includes working with clients to assess product-market-fit, create goal aligned roadmaps and product delivery. He brings in his ever-curious mindset, business knowledge, and interdisciplinary thinking to solve problems that form our surroundings. His primary focus area is - product discovery - through which he helps address key product risks in the early stages of the product Sushant is passionate about Indian digital ecosystems. He is working with Indian companies to create better products.
- 3. Start your day with the your business dashboard! 3
- 4. Start your day with the your business dashboard! 4
- 5. Problem Landscape 5 Retail Bank KYC - Compliance to Customer Service The Ask
- 6. Few examples: How much amount was disbursed yesterday in Mumbai? How many car loans were sanctioned this week? What’s the health of APIs and underlying systems for last 3 hours? What are we trying to achieve 6 “Data intelligence delivering business value” What are the reasons for drop offs? Is it system or user? Are my APIs overloaded? Average time for disbursement Opportunity loss in the Funnel at different stages Login Offers Risk Checks Sanction 2FA Disbursement Which offers are attractive? How many customers are moving further after viewing offers How long it takes to send the OTP and customer action? Do users need alternate mechanism?
- 7. Explosion of personas and explosion of requirements 7 Revenue Generated Price Sensitivity Number of Users Customer micro- Segment Business Deployment status Load on a service Downtime for Service Developers Service Availability Service Traceability Routing Security team Insights What can I understand? Data Scientists Customer 360 Customer Propensity Customer-product fit Customer facing executives 7
- 8. 8 Isolated Solutions Web application Mobile application User click stream Social media Market data System level metrics Customer support Competitor data Logs Real time monitoring of all infrastructure components and service issues monitored using Prometheus and charts created on Grafana. Many other tools like EFK stack, kafka etc. are used. Developers/ IT Support Understand system health and avoid failures Periodic data is provided (sometimes manually) after pulling out of tools like Kafka, GTM in form of excel. Product Owners No drop-outs, all journeys should be completed Business/ C-level execs Data pulled out on-demand (manually) and shared via email/excel sheets. How is my business performing, where are the leakages
- 9. 9 Business ● Siloed ○ Systems ○ Tools ● Different ○ Targets ○ Maturity ○ Objectives People & Process ● Dependent ○ On a central data team ● Manual effort ○ Lack of standard processes ○ Duplicate effort ● Different ○ Tech Stack, Architecture & tools ○ Data granularity, formats & architectures ○ Data exploration Scopes ● Siloed ○ Data (Storage) ○ Business Units Data & Technology Limitations and pain points
- 10. 10 ● Consolidation ● Coordination ● Manual synthesis ● Low confidence Piecemeal Solutions
- 11. 11 Shift in mindset Solutions of Yesterday created the problems of Today Photo by Garidy Sanders on Unsplash
- 12. 12 Persona and business domains, wants/requirement s Levers of mindset shift Traditionally data is looked from the weekly or for monthly leadership review. So someone requests and presents what needs to be presented From requests to predefined datasets serving insights & exploration Design, Data discoverability & presentability Product Mindset 12 © 2021 Thoughtworks
- 13. Bringing it all together - The “Superdata” Solution 13
- 14. 14 14 What it is really? Make it quick and easy to explore a hypothesis (business or technical), accept or disprove it, and move on to find the root cause. Platform which enables people to use their skills, extend their senses, support their intuitions. The Superdata solution a.k.a Command center
- 15. 15 Data Platform Cloud DW Data Lake (Cloud Object Storage) arts Data Marts ODS Tech Solution Dashboards Data Service API’s Reports Data Ingestion & Integration Batch Ingestion Unstructured Source Ingestion API Ingestion Streaming Ingestion Orchestration Service 15 Source Systems Bank applications data Tele channels data Physical data from Branches Social media data Partners data 15 15 DevOps/DataOps Data Governance Data Catalog Data Quality Security Business Events Data Transformation & Ml ELT Stream Processing ML Toolkits* Deep Learning* ETL *Future Scope
- 16. ©ThoughtWorks 2021 Commercial in Confidence SOURCE SYSTEMS Tele channels data Physical data from branches Social media data Partners data Bank applications data Whitelisted, one way hashed data CONFIGURABLE SECURITY LAYERS Data filter service GEMALTO CCKM GEMALTO HSM TOKEN LOOKUP STORE DATA INGESTION / INTEGRATION ON LAYER DATA TRANSFORMATION LAYER DATA CONSUMPTION LAYER Cloud DW Data Lake (Cloud Object Storage) arts Data Marts ODS ODS Cloud Native Encryption at Storage DATA PLATFORM DATA INGESTION / INTEGRATION ON LAYER DATA TRANSFORMATION LAYER DATA CONSUMPTION LAYER PII / SPDI Layer (Only for PII fields) Data Encryption Component Data Decryption Component Tokenization Layer (Only for PII fields) Tokenizer De Tokenizer Gemalto Virtual HSM KMS VPC Security in Layers
- 17. Data Quality Framework Intermediate Data Quality Fit for Purpose Data Quality Ad-hoc Analysis Data Discovery Metadata Service Repository & Indexing Service Ownership of DQ Fit for Purpose Data Quality Purpose Fit for Purpose Data Quality Purpose Baseline Data Quality / Sensible Defaults Metrics Definition Rules Authoring Rules Execution Engine KPIs and Dashboards Metrics Definition Continuous data quality improvement*
- 18. Dashboards Data Service API’s Reports 18 Tech Stack Data Platform Cloud DW Data Lake (Cloud Object Storage) Data Marts ODS Data Ingestion & Integration Batch Ingestion Unstructured Source Ingestion API Ingestion Streaming Ingestion Orchestration Service Data Transformation & Ml ELT Stream Processing ML Toolkits Deep Learning ETL Source Systems Bank Applications Data Tele Channels Data Physical Store Data Social Media Data Partners data DevOps/DataOps Data Governance Data Catalog Data Quality Security Business Events
- 19. 19 19 Data Platform Cloud DW Data Lake (Cloud Object Storage) arts Data Marts ODS Dashbo ards Data Service API’s Report s Data Ingestion & Integration Batch Ingestion Unstructured Source Ingestion API Ingestion Streaming Ingestion Orchestration Service Data Transformation & Ml ELT Stream Processi ng ML Toolkits Deep Learning ETL Source Systems Bank Applications Data Tele Channels Data Physical Store Data Social Media Data Partners data DevOps/DataOps Data Governance Data Catalog Data Quality Security Business Events Serve Data as a Product Auto Loan/Cam paign Customer Personal Loan /Network analysis Credit Card/Finan ce Social Media Customer Profile Domain driven data boundaries The boundaries cut across the platform - from source to consumption!
- 20. Principles guiding building blocks of Data Mesh 20 Domain ownership Data as a product Self-serve data infrastructure Federated computational governance {G} {G} {G}
- 21. 21 Use cases served through platform ● Increased self-service ● Anomaly detection and alerting ● Personalization and Nudges ● Domain driven boundaries for data ● Adaptive journey completions
- 23. 23 Business ● Responsive ○ Real Time ○ Self service ○ Responding to customer behavior quicker ● Insights ○ Stimulating ○ Proactive Process & People ● Transparent ○ Democratization ○ Standardized processes ○ Data driven process planning ● Empowered ○ Own, create and analyse ○ Touch multiple business aspects ● Governance & Quality ○ Accurate ○ Secure ● Technology ○ Resilient ○ Rapidly evolving ○ Loosely coupled ○ Configuration driven Data & Technology Impact “Can we ask for this data” to “Can we pull up this data” “we are not able to track numbers, since last couple of hours dashboard is showing the same numbers”
- 24. 24 Our Learning ● Data-as-a-first-class-citizen ● Extend domain boundaries to data platform ● Avoid tight coupling with upstream and downstream ● Timely scaling considerations
- 25. Thank you Sushant Joshi Twitter: @sushantjoshi https://sushant-joshi.medium.com Balvinder Khurana LinkedIn: https://www.linkedin.com/in/balvinder-khurana/ https://khuranabalvinder.medium.com/ 25 www.agilemumbai.com
- 26. Size, Scale and Complexity 26 Web & Apps Tele channels Physical stores Digital Ecosystem partners Kiosks, agents etc Sales & marketing operations Products Technology Risk & compliance Strategy Enterprise functions Social media Retail Loans Retail Deposits ... Regulators Investors Aggregators Partners Product based Customer segment based Channel Corporate Customer risk Fraud Underwriting Liquidity HR Employee Facilities Legal How customers interact with the organisation Constraining and driving forces
- 27. Data Product Data Platform Architecture Quantum Fundamental unit of architecture Self-Serve Data Product Domain Polyglot Data Output Ports Polyglot Data Input Ports Discoverable Addressable Self-describing Trustworthy Interoperable governed by global open standard Secure enforce globally configured access control at each data product output port Control Ports Stats Logs, metrics Self discovery Management Defined and Monitored silos 27
Editor's Notes
- Sushant & Balvinder 1 min : Intro : Slide 4 n 4 min : Business context : Slide 5 to 8 : quick passthrough 5 min : Explain the situation and mindset shift : slide 8 to 12 2 min : slide for mindset shift : Slide 13 & 14 1 min : ask and summersing the approach in our own words : slide 17 10 mins min : solution + tech + security + quality + tech stack upto Slide 22 2 min : domain data product : Slide 23 2 min : Impact on people and processes : slide 21 2 min : Impact and learnings Closure 29 min / 35 min 10 min : Q&A
- Sushant Every executive likes information at their fingertips. In the form which will help them do real time probing and take decisions in time. We hear this advice from everyone, know where you stand. They look for Actionable Insights available real time rather than monthly or periodic reports
- Sushant Imagine you needing to catch a flight, first thing you would want to know is how long it will take you to reach airport, traffic is unpredictable. They look for Actionable Insights available real time rather than monthly or periodic reports
- Sushant & Balvinder (Techview of The Ask) THE BANK and the operating environment Pre Covid days - Digital lending is on everyone’s agenda / Some are exploring , toying with the idea Situation at the banking world Engagement is a problem and Fintechs are vying for the pie Payment infrastructure is coming to an age / wallets Customer engagement is coming at the center of the strategy KYC Resulting into KYC not for compliance but for acquiring, retaining and serving right THE ASK Define data strategy and roadmap for a data platform on cloud Self-service data platform which can onboard multiple products and systems in future To know the customer you need to know your systems wells CLIENT BACKGROUND Leading Bank in India who had embarked on the ambitious digital journey to bring in all retail loans under one roof, provide better customer experience and eliminate waste in the proces This was also the time startups have started making inroads in banks’ business quite well. GOAL Unified and clear view of customer actions Self-service insights into customer and system behavior in real-time (at scale) to Identify value pockets through behaviour based segments Create business ecosystem to power real-time offers based on data insights Provide clear view of business for timely actions and course corrections to optimize identified metric such as Risk, Account Profitability
- Sushant Actionable Insights through Real Time, Self Serviced Information of everything that’s happening on the platform WHAT Generalise. We are directly jumping on the domain oriented
- Parameters - Data democratization, self-service Sushant WHO Double click on objective through lenses of stakeholders Balvinder will come on this slide
- Balvinder HOW Because of all the limitations mentioned earlier, each stakeholder group started attempting to solve their problems individually Organisation goal was not aligned and individuals were opting for solutions which would make sense to them and feasible with in the limited resources they have. The intent (and hence call to action), granularity and scope of information needed is different. So not just the tools, but also the data that is consumed - is isolated For each group: Stakeholder - ask - solution - data Add data stakeholder group
- Balvinder But it was not easy to reach to answers to the questions for each stakeholder group. There are so many hurdles on the way. Business Siloed and fragmented systems Different targets and no common agreed goals for building data world view Varying maturity of business and tech orgs Who is focussed on Customer happiness Do we want 99.99% availability but still pissed off customer Data Siloed and locked useful data into various tools and owned business orgs operating in silos Disparate data sources Difficult to Correlate Quickly for Monitoring or finding Business relevance Limited scope for exploration Unified architectures (of consuming platform) are not possible Technology Competing or non-compatible tech stacks Learn individual tools
- Balvinder Human/emotional dimension What happens because of the frustration What got you here wont take you far - change in the mindset is needed Consolidated insights was still a problem It needed co-ordinating for information availability and then synthesis by someone who may not have the best of understanding of how that information is collected Patchy solution for a group of people was still serving only limited section and broad based acceptance and hence data availability was a challenge
- Sushant https://unsplash.com/photos/-X1CDIau79o
- Sushant
- Sushant & Balvinder
- Balvinder Inline with regulated environment Scalability Configurability Self-service consumption Security Banking and regulated world Financial data Approach to security Data Quality
- Data Quality framework hierarchical data quality framework from the perspective of data users. This framework consists of big data quality dimensions, quality characteristics, and quality indexes ROI of data quality Define, Measure, Analyze, Design/Improve, and Verify/Control
- Balvinder Inline with regulated environment Scalability Configurability Self-service consumption Security Banking and regulated world Financial data Approach to security
- Balvinder Bootstrap the platform so easily, scale to the sources and scale to the consumers and kept on incrementally materializing data driven value which was differentiating Decompose data products around domains, distribute the ownership. The principle we have been applying to web services world to create microservices.
- Balvinder & sushant A decentralised socio technical approach in managing and accessing analytical data at scale Getting value from data at scale, in complex organization in an environment thats constantly changing, looks at both the organizational responsibilities and the architecture Domain Ownership The genesis of the first principle is pushing towards the Domain expertise continuum. Domain teams to manage and own not only their operational data but also the analytical data. “No longer a by-product of that domain but an first class product of that domain” which they share with other domains to enable the value driven outcomes for the organization So that the ecosystem creating and consuming data can scale out as the number of sources of data, number of use cases, and diversity of access models to the data increases; simply increase the autonomous nodes on the mesh. Data as product For a distributed data platform to be successful, domain data teams must apply product thinking with similar rigor to the datasets that they provide; considering their data assets as their products and the rest of the organization's data scientists, ML and data engineers as their customers. So that data users can easily discover, understand and securely use high quality data with a delightful experience; data that is distributed across many domains. Self serve infrastructure So that the Portfolio teams can create and consume data products autonomously using the platform abstractions, hiding the complexity of building, executing and maintaining secure and interoperable data products. Infrastructure is centrally managed, yet it is provisioned per data product to support its autonomous operation in a multi-tenancy fashion. It’s important that deployment or update of one data product doesn’t impact other data products, from an infrastructure perspective. Federated computational governance So that data users can get value from aggregation and correlation of independent data products - the mesh is behaving as an ecosystem following global interoperability standards; standards that are baked computationally into the platform.
- Sushant Slide 1- 7 : 8-9 mins Slide 8-10: 5-6 mins Slide 11-14: 3-4 mins Slide 15-20: 7 mins Slide 21-22: 3 min QnA : 15 mins
- Sushant(business, process & people) and Balvinder (Data & tech) How engineering and data platforms can be used to derive real-time business and system insights that help in proactive decisions (Data intelligence delivering business value). Approach towards creating self-service data analysis and visualization platform (Impact of Data Intelligence in Software Development Life Cycle). Artificial intelligence can help you respond to customer behavior quicker. (Automated intelligence delivering business value). Understand how holistic data view can help into multiple aspects of business - operation, process, and delivery (Data intelligence delivering business value).
- Sushant Data has a better Idea
- Sushant How does a large enterprise look like General picture Thought process Complexity of channels X complexity of Products Each product has it’s own way of selling and operations Customer segments also need specific handling such as HNI, premium, priority sector etc Each product type takes it’s own shape in terms of strategy, risks and balance Large enterprise have multi speed departments, that dictates inherent need for different systems and customised processes for suitability. This leads to each business group optimising people, tech and processes based on their goals Which results in Silos Fragmented systems Disparate data sources owned business orgs (operating in silos) Competing tech stacks // non compatible tech stacks Different targets and no common agreed goals for building data world view Varying maturity of business and tech orgs Siloed attempts to solve the challenges made it a further big crises as big picture was missing No one talked about it explicitly and what it means for the data solution
- Explain and lead to definition - Six dimensions of data product