Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Explainable AI in Industry (FAT* 2020 Tutorial)

Artificial Intelligence is increasingly playing an integral role in determining our day-to-day experiences. Moreover, with proliferation of AI based solutions in areas such as hiring, lending, criminal justice, healthcare, and education, the resulting personal and professional implications of AI are far-reaching. The dominant role played by AI models in these domains has led to a growing concern regarding potential bias in these models, and a demand for model transparency and interpretability. In addition, model explainability is a prerequisite for building trust and adoption of AI systems in high stakes domains requiring reliability and safety such as healthcare and automated transportation, as well as critical industrial applications with significant economic implications such as predictive maintenance, exploration of natural resources, and climate change modeling.

As a consequence, AI researchers and practitioners have focused their attention on explainable AI to help them better trust and understand models at scale. The challenges for the research community include (i) defining model explainability, (ii) formulating explainability tasks for understanding model behavior and developing solutions for these tasks, and finally (iii) designing measures for evaluating the performance of models in explainability tasks.

In this tutorial, we will first motivate the need for model interpretability and explainability in AI from societal, legal, customer/end-user, and model developer perspectives. [Note: Due to time constraints, we will not focus on techniques/tools for providing explainability as part of AI/ML systems.] Then, we will focus on the real-world application of explainability techniques in industry, wherein we present practical challenges / implications for using explainability techniques effectively and lessons learned from deploying explainable models for several web-scale machine learning and data mining applications. We will present case studies across different companies, spanning application domains such as search and recommendation systems, sales, lending, and fraud detection. Finally, based on our experiences in industry, we will identify open problems and research directions for the research community.

  • Soyez le premier à commenter

Explainable AI in Industry (FAT* 2020 Tutorial)

  1. 1. Explainable AI in Industry FAT* 2020 Tutorial Krishna Gade (Fiddler Labs), Sahin Cem Geyik (LinkedIn), Krishnaram Kenthapadi (Amazon AWS AI), Varun Mithal (LinkedIn), Ankur Taly (Fiddler Labs) https://sites.google.com/view/fat20-explainable-ai-tutorial 1
  2. 2. Introduction and Motivation 2
  3. 3. Explainable AI - From a Business Perspective 3
  4. 4. What is Explainable AI? Data Black-Box AI AI product Confusion with Today’s AI Black Box ● Why did you do that? ● Why did you not do that? ● When do you succeed or fail? ● How do I correct an error? Black Box AI Decision, Recommendation Clear & Transparent Predictions ● I understand why ● I understand why not ● I know why you succeed or fail ● I understand how to correct errors Explainable AI Data Explainable AI Explainable AI Product Decision Explanation Feedback
  5. 5. Internal Audit, Regulators IT & Operations Data Scientists Business Owner Can I trust our AI decisions? Are these AI system decisions fair? Customer Support How do I answer this customer complaint? How do I monitor and debug this model? Is this the best model that can be built? Black-box AI Why I am getting this decision? How can I get a better decision? Poor Decision Black-box AI creates confusion and doubt
  6. 6. Black-box AI creates business risk for Industry
  7. 7. Explainable AI - From a Model Perspective 7
  8. 8. Why Explainability: Debug (Mis-)Predictions 8 Top label: “clog” Why did the network label this image as “clog”?
  9. 9. 9 Why Explainability: Improve ML Model Credit: Samek, Binder, Tutorial on Interpretable ML, MICCAI’18
  10. 10. Why Explainability: Verify the ML Model / System 10 Credit: Samek, Binder, Tutorial on Interpretable ML, MICCAI’18
  11. 11. 11 Why Explainability: Learn New Insights Credit: Samek, Binder, Tutorial on Interpretable ML, MICCAI’18
  12. 12. 12 Why Explainability: Learn Insights in the Sciences Credit: Samek, Binder, Tutorial on Interpretable ML, MICCAI’18
  13. 13. Explainable AI - From a Regulatory Perspective 13
  14. 14. Immigration Reform and Control Act Citizenship Rehabilitation Act of 1973; Americans with Disabilities Act of 1990 Disability status Civil Rights Act of 1964 Race Age Discrimination in Employment Act of 1967 Age Equal Pay Act of 1963; Civil Rights Act of 1964 Sex And more... Why Explainability: Laws against Discrimination 14
  15. 15. Fairness Privacy Transparency Explainability 15
  16. 16. Fairness Privacy Transparency Explainability 16
  17. 17. Fairness Privacy Transparency Explainability 17
  18. 18. Why Explainability: Growing Global AI Regulation ● GDPR: Article 22 empowers individuals with the right to demand an explanation of how an automated system made a decision that affects them. ● Algorithmic Accountability Act 2019: Requires companies to provide an assessment of the risks posed by the automated decision system to the privacy or security and the risks that contribute to inaccurate, unfair, biased, or discriminatory decisions impacting consumers ● California Consumer Privacy Act: Requires companies to rethink their approach to capturing, storing, and sharing personal data to align with the new requirements by January 1, 2020. ● Washington Bill 1655: Establishes guidelines for the use of automated decision systems to protect consumers, improve transparency, and create more market predictability. ● Massachusetts Bill H.2701: Establishes a commission on automated decision-making, transparency, fairness, and individual rights. ● Illinois House Bill 3415: States predictive data analytics determining creditworthiness or hiring decisions may not include information that correlates with the applicant race or zip code.
  19. 19. 19 SR 11-7 and OCC regulations for Financial Institutions
  20. 20. Model Diagnostics Root Cause Analytics Performance monitoring Fairness monitoring Model Comparison Cohort Analysis Explainable Decisions API Support Model Launch Signoff Model Release Mgmt Model Evaluation Compliance Testing Model Debugging Model Visualization Explainable AI Train QA Predict Deploy A/B Test Monitor Debug Feedback Loop “Explainability by Design” for AI products
  21. 21. AI @ Scale - Challenges for Explainable AI 21
  22. 22. LinkedIn operates the largest professional network on the Internet 645M+ members 30M+ companies are represented on LinkedIn 90K+ schools listed (high school & college) 35K+ skills listed 20M+ open jobs on LinkedIn Jobs 280B Feed updates
  23. 23. 23© 2019 Amazon Web Services, Inc. or its affiliates. All rights reserved | The AWS ML Stack Broadest and most complete set of Machine Learning capabilities VISION SPEECH TEXT SEARCH NEW CHATBOTS PERSONALIZATION FORECASTING FRAUD NEW DEVELOPMENT NEW CONTACT CENTERS NEW Amazon SageMaker Ground Truth Augmented AI SageMaker Neo Built-in algorithms SageMaker Notebooks NEW SageMaker Experiments NEW Model tuning SageMaker Debugger NEW SageMaker Autopilot NEW Model hosting SageMaker Model Monitor NEW Deep Learning AMIs & Containers GPUs & CPUs Elastic Inference Inferentia FPGA Amazon Rekognition Amazon Polly Amazon Transcribe +Medical Amazon Comprehend +Medical Amazon Translate Amazon Lex Amazon Personalize Amazon Forecast Amazon Fraud Detector Amazon CodeGuru AI SERVICES ML SERVICES ML FRAMEWORKS & INFRASTRUCTURE Amazon Textract Amazon Kendra Contact Lens For Amazon Connect SageMaker Studio IDE NEW NEW NEW NEW NE W
  24. 24. Achieving Explainable AI Approach 1: Post-hoc explain a given AI model ● Individual prediction explanations in terms of input features, influential examples, concepts, local decision rules ● Global prediction explanations in terms of entire model in terms of partial dependence plots, global feature importance, global decision rules Approach 2: Build an interpretable model ● Logistic regression, Decision trees, Decision lists and sets, Generalized Additive Models (GAMs) 24
  25. 25. Slide credit: https://twitter.com/chandan_singh96/status/1138811752769101825?s=20 Integrated Gradients
  26. 26. Case Studies on Explainable AI 26
  27. 27. Case Study: Explainability for Debugging ML models Ankur Taly** (Fiddler labs) (Joint work with Mukund Sundararajan, Kedar Dhamdhere, Pramod Mudrakarta) 27 **This research was carried out at Google Research
  28. 28. Achieving Explainable AI Approach 1: Post-hoc explain a given AI model ● Individual prediction explanations in terms of input features, influential examples, concepts, local decision rules ● Global prediction explanations in terms of entire model in terms of partial dependence plots, global feature importance, global decision rules Approach 2: Build an interpretable model ● Logistic regression, Decision trees, Decision lists and sets, Generalized Additive Models (GAMs) 28
  29. 29. Top label: “fireboat” Why did the model label this image as “fireboat”? 29
  30. 30. Top label: “clog” Why did the model label this image as “clog”?
  31. 31. Attribute a model’s prediction on an input to features of the input Examples: ● Attribute an object recognition network’s prediction to its pixels ● Attribute a text sentiment network’s prediction to individual words ● Attribute a lending model’s prediction to its features A reductive formulation of “why this prediction” but surprisingly useful :-) The Attribution Problem
  32. 32. Feature*Gradient Attribution to a feature is feature value times gradient, i.e., xi* 𝜕y/𝜕xi ● Gradient captures sensitivity of output w.r.t. feature ● Equivalent to Feature*Coefficient for linear models ○ First-order Taylor approximation of non-linear models ● Popularized by SaliencyMaps [NIPS 2013], Baehrens et al. [JMLR 2010] 32
  33. 33. Feature*Gradient Attribution to a feature is feature value times gradient, i.e., xi* 𝜕y/𝜕xi ● Gradient captures sensitivity of output w.r.t. feature ● Equivalent to Feature*Coefficient for linear models ○ First-order Taylor approximation of non-linear models ● Popularized by SaliencyMaps [NIPS 2013], Baehrens et al. [JMLR 2010] 33 Gradients in the vicinity of the input seem like noise
  34. 34. score intensity Interesting gradients uninteresting gradients (saturation) 1.0 0.0 Baseline … scaled inputs ... … gradients of scaled inputs …. Input
  35. 35. IG(input, base) ::= (input - base) * ∫0 -1▽F(𝛂*input + (1-𝛂)*base) d𝛂 Original image Integrated Gradients Integrated Gradients [ICML 2017] Integrate the gradients along a straight-line path from baseline to input
  36. 36. ● Ideally, the baseline is an informationless input for the model ○ E.g., Black image for image models ○ E.g., Empty text or zero embedding vector for text models ● Integrated Gradients explains F(input) - F(baseline) in terms of input features Aside: Baselines (or Norms) are essential to explanations [Kahneman-Miller 86] E.g., A man suffers from indigestion. Doctor blames it to a stomach ulcer. Wife blames it on eating turnips. ● Both are correct relative to their baselines. What is a baseline?
  37. 37. Original image “Clog” Why is this image labeled as “clog”?
  38. 38. Original image Integrated Gradients (for label “clog”) “Clog” Why is this image labeled as “clog”?
  39. 39. ● Deep network [Kearns, 2016] predicts if a molecule binds to certain DNA site ● Finding: Some atoms had identical attributions despite different connectivity Detecting an architecture bug
  40. 40. ● There was an implementation bug due to which the convolved bond features did not affect the prediction! ● Deep network [Kearns, 2016] predicts if a molecule binds to certain DNA site ● Finding: Some atoms had identical attributions despite different connectivity Detecting an architecture bug
  41. 41. ● Deep network predicts various diseases from chest x-rays Original image Integrated gradients (for top label) Detecting a data issue
  42. 42. ● Deep network predicts various diseases from chest x-rays ● Finding: Attributions fell on radiologist’s markings (rather than the pathology) Original image Integrated gradients (for top label) Detecting a data issue
  43. 43. Tabular QA Visual QA Reading Comprehension Peyton Manning became the first quarterback ever to lead two different teams to multiple Super Bowls. He is also the oldest quarterback ever to play in a Super Bowl at age 39. The past record was held by John Elway, who led the Broncos to victory in Super Bowl XXXIII at age 38 and is currently Denver’s Executive Vice President of Football Operations and General Manager Kazemi and Elqursh (2017) model. 61.1% on VQA 1.0 dataset (state of the art = 66.7%) Neural Programmer (2017) model 33.5% accuracy on WikiTableQuestions Yu et al (2018) model. 84.6 F-1 score on SQuAD (state of the art) 43 Q: How many medals did India win? A: 197 Q: How symmetrical are the white bricks on either side of the building? A: very Q: Name of the quarterback who was 38 in Super Bowl XXXIII? A: John Elway Three Question Answering Tasks
  44. 44. Tabular QA Visual QA Reading Comprehension Peyton Manning became the first quarterback ever to lead two different teams to multiple Super Bowls. He is also the oldest quarterback ever to play in a Super Bowl at age 39. The past record was held by John Elway, who led the Broncos to victory in Super Bowl XXXIII at age 38 and is currently Denver’s Executive Vice President of Football Operations and General Manager Kazemi and Elqursh (2017) model. 61.1% on VQA 1.0 dataset (state of the art = 66.7%) Neural Programmer (2017) model 33.5% accuracy on WikiTableQuestions Yu et al (2018) model. 84.6 F-1 score on SQuAD (state of the art) 44 Q: How many medals did India win? A: 197 Q: How symmetrical are the white bricks on either side of the building? A: very Q: Name of the quarterback who was 38 in Super Bowl XXXIII? A: John Elway Three Question Answering Tasks Robustness question: Do these network read the question carefully? :-)
  45. 45. Visual Question Answering Q: How symmetrical are the white bricks on either side of the building? A: very 45 Kazemi and Elqursh (2017) model (ResNet + multi-layer LSTM + Attention) Accuracy: 61.1% (state of the art: 66.7%)
  46. 46. Visual Question Answering Q: How symmetrical are the white bricks on either side of the building? A: very How do we probe the model’s understanding of the question? 46 Kazemi and Elqursh (2017) model (ResNet + multi-layer LSTM + Attention) Accuracy: 61.1% (state of the art: 66.7%)
  47. 47. Visual Question Answering Q: How symmetrical are the white bricks on either side of the building? A: very How do we probe the model’s understanding of the question? Enter: Attributions ● Baseline: Empty question, but full image ● By design, attribution will not fall on the image 47 Kazemi and Elqursh (2017) model (ResNet + multi-layer LSTM + Attention) Accuracy: 61.1% (state of the art: 66.7%)
  48. 48. Attributions Q: How symmetrical are the white bricks on either side of the building? A: very How symmetrical are the white bricks on either side of the building? red: high attribution blue: negative attribution gray: near-zero attribution 48 Kazemi and Elqursh (2017) model (ResNet + multi-layer LSTM + Attention) Accuracy: 61.1% (state of the art: 66.7%)
  49. 49. Attacking the Model Q: How symmetrical are the white bricks on either side of the building? A: very 49 Kazemi and Elqursh (2017) model (ResNet + multi-layer LSTM + Attention) Accuracy: 61.1% (state of the art: 66.7%)
  50. 50. Attacking the Model Q: How symmetrical are the white bricks on either side of the building? A: very Q: How asymmetrical are the white bricks on either side of the building? A: very 50 Kazemi and Elqursh (2017) model (ResNet + multi-layer LSTM + Attention) Accuracy: 61.1% (state of the art: 66.7%)
  51. 51. Attacking the Model Q: How symmetrical are the white bricks on either side of the building? A: very Q: How asymmetrical are the white bricks on either side of the building? A: very Q: How big are the white bricks on either side of the building? A: very 51 Kazemi and Elqursh (2017) model (ResNet + multi-layer LSTM + Attention) Accuracy: 61.1% (state of the art: 66.7%)
  52. 52. Attacking the Model Q: How symmetrical are the white bricks on either side of the building? A: very Q: How asymmetrical are the white bricks on either side of the building? A: very Q: How big are the white bricks on either side of the building? A: very Q: How fast are the white bricks speaking on either side of the building? A: very 52 Kazemi and Elqursh (2017) model (ResNet + multi-layer LSTM + Attention) Accuracy: 61.1% (state of the art: 66.7%)
  53. 53. Attacking the Model Q: How symmetrical are the white bricks on either side of the building? A: very Q: How asymmetrical are the white bricks on either side of the building? A: very Q: How big are the white bricks on either side of the building? A: very Q: How fast are the white bricks speaking on either side of the building? A: very 53 Attributions help assess model robustness and uncover blind spots! Kazemi and Elqursh (2017) model (ResNet + multi-layer LSTM + Attention) Accuracy: 61.1% (state of the art: 66.7%)
  54. 54. Low-attribution nouns 'tweet', 'childhood', 'copyrights', 'mornings', 'disorder', 'importance', 'critter', 'jumper', 'fits' What is the man doing? → What is the tweet doing? How many children are there? → How many tweet are there? VQA model’s response remains the same 75.6% of the time on questions that it originally answered correctly 54 Replace the subject of a question with a low-attribution noun from the vocabulary ● This ought to change the answer but often does not! Attack: Subject Ablation
  55. 55. ● Visual QA ○ Prefix concatenation attack (accuracy drop: 61.1% to 19%) ○ Stop word deletion attack (accuracy drop: 61.1% to 52%) ● Tabular QA ○ Prefix concatenation attack (accuracy drop: 33.5% to 11.4%) ○ Stop word deletion attack (accuracy drop: 33.5% to 28.5%) ○ Table row reordering attack (accuracy drop: 33.5 to 23%) ● Paragraph QA ○ Improved paragraph concatenation attacks of Jia and Liang from [EMNLP 2017] Paper: Did the model understand the question? [ACL 2018] Many Other Attacks
  56. 56. Case Study: Talent Platform “Diversity Insights and Fairness-Aware Ranking” Sahin Cem Geyik, Krishnaram Kenthapadi 56
  57. 57. Guiding Principle: “Diversity by Design”
  58. 58. Insights to Identify Diverse Talent Pools Representative Talent Search Results Diversity Learning Curriculum “Diversity by Design” in LinkedIn’s Talent Solutions
  59. 59. Plan for Diversity
  60. 60. Plan for Diversity
  61. 61. Identify Diverse Talent Pools
  62. 62. Inclusive Job Descriptions / Recruiter Outreach
  63. 63. Representative Ranking for Talent Search S. C. Geyik, S. Ambler, K. Kenthapadi, Fairness-Aware Ranking in Search & Recommendation Systems with Application to LinkedIn Talent Search, KDD’19. [Microsoft’s AI/ML conference (MLADS’18). Distinguished Contribution Award] Building Representative Talent Search at LinkedIn (LinkedIn engineering blog)
  64. 64. Intuition for Measuring and Achieving Representativeness Ideal: Top ranked results should follow a desired distribution on gender/age/… E.g., same distribution as the underlying talent pool Inspired by “Equal Opportunity” definition [Hardt et al, NIPS’16] Defined measures (skew, divergence) based on this intuition
  65. 65. Desired Proportions within the Attribute of Interest Compute the proportions of the values of the attribute (e.g., gender, gender-age combination) amongst the set of qualified candidates ● “Qualified candidates” = Set of candidates that match the search query criteria ● Retrieved by LinkedIn’s Galene search engine Desired proportions could also be obtained based on legal mandate / voluntary commitment
  66. 66. Fairness-aware Reranking Algorithm (Simplified) Partition the set of potential candidates into different buckets for each attribute value Rank the candidates in each bucket according to the scores assigned by the machine-learned model Merge the ranked lists, balancing the representation requirements and the selection of highest scored candidates Representation requirement: Desired distribution on gender/age/… Algorithmic variants based on how we achieve this balance
  67. 67. Validating Our Approach Gender Representativeness ● Over 95% of all searches are representative compared to the qualified population of the search Business Metrics ● A/B test over LinkedIn Recruiter users for two weeks ● No significant change in business metrics (e.g., # InMails sent or accepted) Ramped to 100% of LinkedIn Recruiter users worldwide
  68. 68. Lessons learned • Post-processing approach desirable • Model agnostic • Scalable across different model choices for our application • Acts as a “fail-safe” • Robust to application-specific business logic • Easier to incorporate as part of existing systems • Build a stand-alone service or component for post-processing • No significant modifications to the existing components • Complementary to efforts to reduce bias from training data & during model training • Collaboration/consensus across key stakeholders
  69. 69. Engineering for Fairness in AI Lifecycle Problem Formation Dataset Construction Algorithm Selection Training Process Testing Process Deployment Feedback Is an algorithm an ethical solution to our problem? Does our data include enough minority samples? Are there missing/biased features? Do we need to apply debiasing algorithms to preprocess our data? Do we need to include fairness constraints in the function? Have we evaluated the model using relevant fairness metrics? Are we deploying our model on a population that we did not train/test on? Are there unequal effects across users? Does the model encourage feedback loops that can produce increasingly unfair outcomes? Credit: K. Browne & J. Draper
  70. 70. Acknowledgements LinkedIn Talent Solutions Diversity team, Hire & Careers AI team, Anti-abuse AI team, Data Science Applied Research team Special thanks to Deepak Agarwal, Parvez Ahammad, Stuart Ambler, Kinjal Basu, Jenelle Bray, Erik Buchanan, Bee-Chung Chen, Patrick Cheung, Gil Cottle, Cyrus DiCiccio, Patrick Driscoll, Carlos Faham, Nadia Fawaz, Priyanka Gariba, Meg Garlinghouse, Gurwinder Gulati, Rob Hallman, Sara Harrington, Joshua Hartman, Daniel Hewlett, Nicolas Kim, Rachel Kumar, Nicole Li, Heloise Logan, Stephen Lynch, Divyakumar Menghani, Varun Mithal, Arashpreet Singh Mor, Tanvi Motwani, Preetam Nandy, Lei Ni, Nitin Panjwani, Igor Perisic, Hema Raghavan, Romer Rosales, Guillaume Saint-Jacques, Badrul Sarwar, Amir Sepehri, Arun Swami, Ram Swaminathan, Grace Tang, Ketan Thakkar, Sriram Vasudevan, Janardhanan Vembunarayanan, James Verbus, Xin Wang, Hinkmond Wong, Ya Xu, Lin Yang, Yang Yang, Chenhui Zhai, Liang Zhang, Yani Zhang
  71. 71. Case Study: Talent Search Varun Mithal, Girish Kathalagiri, Sahin Cem Geyik 71
  72. 72. LinkedIn Recruiter ● Recruiter Searches for Candidates ○ Standardized and free-text search criteria ● Retrieval and Ranking ○ Filter candidates using the criteria ○ Rank candidates in multiple levels using ML models 72
  73. 73. Modeling Approaches ● Pairwise XGBoost ● GLMix ● DNNs via TensorFlow ● Optimization Criteria: inMail Accepts ○ Positive: inMail sent by recruiter, and positively responded by candidate ■ Mutual interest between the recruiter and the candidate 73
  74. 74. Feature Importance in XGBoost 74
  75. 75. How We Utilize Feature Importances for GBDT ● Understanding feature digressions ○ Which a feature that was impactful no longer is? ○ Should we debug feature generation? ● Introducing new features in bulk and identifying effective ones ○ An activity feature for last 3 hours, 6 hours, 12 hours, 24 hours introduced (costly to compute) ○ Should we keep all such features? ● Separating the factors for that caused an improvement ○ Did an improvement come from a new feature, or a new labeling strategy, data source? ○ Did the ordering between features change? ● Shortcoming: A global view, not case by case 75
  76. 76. GLMix Models ● Generalized Linear Mixed Models ○ Global: Linear Model ○ Per-contract: Linear Model ○ Per-recruiter: Linear Model ● Lots of parameters overall ○ For a specific recruiter or contract the weights can be summed up ● Inherently explainable ○ Contribution of a feature is “weight x feature value” ○ Can be examined in a case-by-case manner as well 76
  77. 77. TensorFlow Models in Recruiter and Explaining Them ● We utilize the Integrated Gradients [ICML 2017] method ● How do we determine the baseline example? ○ Every query creates its own feature values for the same candidate ○ Query match features, time-based features ○ Recruiter affinity, and candidate affinity features ○ A candidate would be scored differently by each query ○ Cannot recommend a “Software Engineer” to a search for a “Forensic Chemist” ○ There is no globally neutral example for comparison! 77
  78. 78. Query-Specific Baseline Selection ● For each query: ○ Score examples by the TF model ○ Rank examples ○ Choose one example as the baseline ○ Compare others to the baseline example ● How to choose the baseline example ○ Last candidate ○ Kth percentile in ranking ○ A random candidate ○ Request by user (answering a question like: “Why was I presented candidate x above candidate y?”) 78
  79. 79. Example 79
  80. 80. Example - Detailed 80 Feature Description Difference (1 vs 2) Contribution Feature………. Description………. -2.0476928 -2.144455602 Feature………. Description………. -2.3223877 1.903594618 Feature………. Description………. 0.11666667 0.2114946752 Feature………. Description………. -2.1442587 0.2060414469 Feature………. Description………. -14 0.1215354111 Feature………. Description………. 1 0.1000282466 Feature………. Description………. -92 -0.085286277 Feature………. Description………. 0.9333333 0.0568533262 Feature………. Description………. -1 -0.051796317 Feature………. Description………. -1 -0.050895940
  81. 81. Pros & Cons ● Explains potentially very complex models ● Case-by-case analysis ○ Why do you think candidate x is a better match for my position? ○ Why do you think I am a better fit for this job? ○ Why am I being shown this ad? ○ Great for debugging real-time problems in production ● Global view is missing ○ Aggregate Contributions can be computed ○ Could be costly to compute 81
  82. 82. Lessons Learned and Next Steps ● Global explanations vs. Case-by-case Explanations ○ Global gives an overview, better for making modeling decisions ○ Case-by-case could be more useful for the non-technical user, better for debugging ● Integrated gradients worked well for us ○ Complex models make it harder for developers to map improvement to effort ○ Use-case gave intuitive results, on top of completely describing score differences ● Next steps ○ Global explanations for Deep Models 82
  83. 83. Case Study: Model Interpretation for Predictive Models in B2B Sales Predictions Jilei Yang, Wei Di, Songtao Guo 83
  84. 84. Problem Setting ● Predictive models in B2B sales prediction ○ E.g.: random forest, gradient boosting, deep neural network, … ○ High accuracy, low interpretability ● Global feature importance → Individual feature reasoning 84
  85. 85. Example 85
  86. 86. Revisiting LIME ● Given a target sample 𝑥 𝑘, approximate its prediction 𝑝𝑟𝑒𝑑(𝑥 𝑘) by building a sample-specific linear model: 𝑝𝑟𝑒𝑑(𝑋) ≈ 𝛽 𝑘1 𝑋1 + 𝛽 𝑘2 𝑋2 + …, 𝑋 ∈ 𝑛𝑒𝑖𝑔ℎ𝑏𝑜𝑟(𝑥 𝑘) ● E.g., for company CompanyX: 0.76 ≈ 1.82 ∗ 0.17 + 1.61 ∗ 0.11+… 86
  87. 87. xLIME Piecewise Linear Regression Localized Stratified Sampling 87
  88. 88. Piecewise Linear Regression Motivation: Separate top positive feature influencers and top negative feature influencers 88
  89. 89. Impact of Piecewise Approach ● Target sample 𝑥 𝑘=(𝑥 𝑘1, 𝑥 𝑘2, ⋯) ● Top feature contributor ○ LIME: large magnitude of 𝛽 𝑘𝑗 ⋅ 𝑥 𝑘𝑗 ○ xLIME: large magnitude of 𝛽 𝑘𝑗 − ⋅ 𝑥 𝑘𝑗 ● Top positive feature influencer ○ LIME: large magnitude of 𝛽 𝑘𝑗 ○ xLIME: large magnitude of negative 𝛽 𝑘𝑗 − or positive 𝛽 𝑘𝑗 + ● Top negative feature influencer ○ LIME: large magnitude of 𝛽 𝑘𝑗 ○ xLIME: large magnitude of positive 𝛽 𝑘𝑗 − or negative 𝛽 𝑘𝑗 + 89
  90. 90. Localized Stratified Sampling: Idea Method: Sampling based on empirical distribution around target value at each feature level 90
  91. 91. Localized Stratified Sampling: Method ● Sampling based on empirical distribution around target value for each feature ● For target sample 𝑥 𝑘 = (𝑥 𝑘1 , 𝑥 𝑘2 , ⋯), sampling values of feature 𝑗 according to 𝑝𝑗 (𝑋𝑗) ⋅ 𝑁(𝑥 𝑘𝑗 , (𝛼 ⋅ 𝑠𝑗 )2) ○ 𝑝𝑗 (𝑋𝑗) : empirical distribution. ○ 𝑥 𝑘𝑗 : feature value in target sample. ○ 𝑠𝑗 : standard deviation. ○ 𝛼 : Interpretable range: tradeoff between interpretable coverage and local accuracy. ● In LIME, sampling according to 𝑁(𝑥𝑗 , 𝑠𝑗 2). 91 _
  92. 92. Summary 92
  93. 93. LTS LCP (LinkedIn Career Page) Upsell ● A subset of churn data ○ Total Companies: ~ 19K ○ Company features: 117 ● Problem: Estimate whether there will be upsell given a set of features about the company’s utility from the product 93
  94. 94. Top Feature Contributor 94
  95. 95. 95
  96. 96. Top Feature Influencers 96
  97. 97. Key Takeaways ● Looking at the explanation as contributor vs. influencer features is useful ○ Contributor: Which features end-up in the current outcome case-by-case ○ Influencer: What needs to be done to improve likelihood, case-by-case ● xLIME aims to improve on LIME via: ○ Piecewise linear regression: More accurately describes local point, helps with finding correct influencers ○ Localized stratified sampling: More realistic set of local points ● Better captures the important features 97
  98. 98. Case Study: Integrated Gradients for Explaining Diabetic Retinopathy Predictions Ankur Taly** (Fiddler labs) (Joint work with Mukund Sundararajan, Kedar Dhamdhere, Pramod Mudrakarta) 98 **This research was carried out at Google Research
  99. 99. Prediction: “proliferative” DR1 ● Proliferative implies vision-threatening Can we provide an explanation to the doctor with supporting evidence for “proliferative” DR? Retinal Fundus Image 1Diabetic Retinopathy (DR) is a diabetes complication that affects the eye. Deep networks can predict DR grade from retinal fundus images with high accuracy (AUC ~0.97) [JAMA, 2016].
  100. 100. Retinal Fundus Image Integrated Gradients for label: “proliferative” Visualization: Overlay heatmap on green channel
  101. 101. Retinal Fundus Image Integrated Gradients for label: “proliferative” Visualization: Overlay heatmap on green channel Lesions Neovascularization ● Hard to see on original image ● Known to be vision-threatening
  102. 102. Retinal Fundus Image Integrated Gradients for label: “proliferative” Visualization: Overlay heatmap on green channel Lesions Neovascularization ● Hard to see on original image ● Known to be vision-threatening Does attributions help improve doctors better diagnose diabetic retinopathy?
  103. 103. Assisted-read study 9 doctors grade 2000 images under three different conditions A. Image only B. Image + Model’s prediction scores C. Image + Model’s prediction scores + Explanation (Integrated Gradients)
  104. 104. 9 doctors grade 2000 images under three different conditions A. Image only B. Image + Model’s prediction scores C. Image + Model’s prediction scores + Explanation (Integrated Gradients) Findings: ● Model’s predictions (B) significantly improve accuracy vs. image only (A) (p < 0.001) ● Both forms of assistance (B and C) improved sensitivity without hurting specificity ● Explanations (C) improved accuracy of cases with DR (p < 0.001) but hurt accuracy of cases without DR (p = 0.006) ● Both B and C increase doctor ↔ model agreement Paper: Using a deep learning algorithm and integrated gradients explanation to assist grading for diabetic retinopathy --- Journal of Ophthalmology [2018] Assisted-read study
  105. 105. Case Study: Fiddler’s Explainable AI Engine Ankur Taly (Fiddler labs) 105
  106. 106. All your data Any data warehouse Custom Models Fiddler Modeling Layer Explainable AI for everyone APIs, Dashboards, Reports, Trusted Insights Fiddler’s Explainable AI Engine Mission: Unlock Trust, Visibility and Insights by making AI Explainable in every enterprise
  107. 107. Credit Line Increase Fair lending laws [ECOA, FCRA] require credit decisions to be explainable Bank Credit Lending Model Why? Why not? How? ? Request Denied Query AI System Credit Lending Score = 0.3 Example: Credit Lending in a black-box ML world
  108. 108. How Can This Help… Customer Support Why was a customer loan rejected? Bias & Fairness How is my model doing across demographics? Lending LOB What variables should they validate with customers on “borderline” decisions? Explain individual predictions (using Shapley Values)
  109. 109. How Can This Help… Customer Support Why was a customer loan rejected? Bias & Fairness How is my model doing across demographics? Lending LOB What variables should they validate with customers on “borderline” decisions? Explain individual predictions (using Shapley Values)
  110. 110. How Can This Help… Customer Support Why was a customer loan rejected? Bias & Fairness How is my model doing across demographics? Lending LOB What variables should they validate with customers on “borderline” decisions? Explain individual predictions (using Shapley Values) Probe the model on counterfactuals
  111. 111. How Can This Help… Customer Support Why was a customer loan rejected? Why was the credit card limit low? Why was this transaction marked as fraud? Integrating explanations
  112. 112. Classic result in game theory on distributing gain in a coalition game ● Coalition Games ○ Players collaborating to generate some gain (think: revenue) ○ Set function v(S) determining the gain for any subset S of players ● Shapley Values are a fair way to attribute the gain to players based on their contributions ○ Shapley value of a player is a specific weighted aggregation of its marginal contribution to all possible subsets of other players ○ Axiomatic guarantee: Unique method under four very simple axioms Shapley Value [Annals of Mathematical studies,1953]
  113. 113. ● Define a coalition game for each model input X ○ Players are the features in the input ○ Gain is the model prediction (output), i.e., gain = F(X) ● Feature attributions are the Shapley values of this game Shapley Values for Explaining ML models
  114. 114. ● Define a coalition game for each model input X ○ Players are the features in the input ○ Gain is the model prediction (output), i.e., gain = F(X) ● Feature attributions are the Shapley values of this game Challenge: What does it mean for only some players (features) to be present? Shapley Values for Explaining ML models
  115. 115. ● Define a coalition game for each model input X ○ Players are the features in the input ○ Gain is the model prediction (output), i.e., gain = F(X) ● Feature attributions are the Shapley values of this game Challenge: What does it mean for only some players (features) to be present? Approach: Randomly sample the (absent) feature from a certain distribution ● Different approaches choose different distributions and yield significantly different attributions! Preprint: The Explanation Game: Explaining Machine Learning Models with Cooperative Game Theory Shapley Values for Explaining ML models
  116. 116. How Can This Help… Global Explanations What are the primary feature drivers of the dataset on my model? Region Explanations How does my model perform on a certain slice? Where does the model not perform well? Is my model uniformly fair across slices? Slice & Explain
  117. 117. Model Monitoring: Feature Drift Investigate Data Drift Impacting Model Performance Time slice Feature distribution for time slice relative to training distribution
  118. 118. How Can This Help… Operations Why are there outliers in model predictions? What caused model performance to go awry? Data Science How can I improve my ML model? Where does it not do well? Model Monitoring: Outliers with Explanations Outlier Individual Explanations
  119. 119. Model Diagnostics Root Cause Analytics Performance monitoring Fairness monitoring Model Comparison Cohort Analysis Explainable Decisions API Support Model Launch Signoff Model Release Mgmt Model Evaluation Compliance Testing Model Debugging Model Visualization Explainable AI Train QA Predict Deploy A/B Test Monitor Debug Feedback Loop An Explainable Future
  120. 120. Explainability Challenges & Tradeoffs 120 User PrivacyTransparency Fairness Performance ? ● Lack of standard interface for ML models makes pluggable explanations hard ● Explanation needs vary depending on the type of the user who needs it and also the problem at hand. ● The algorithm you employ for explanations might depend on the use-case, model type, data format, etc. ● There are trade-offs w.r.t. Explainability, Performance, Fairness, and Privacy.
  121. 121. Explainability in ML: Broad Challenges Actionable explanations Balance between explanations & model secrecy Robustness of explanations to failure modes (Interaction between ML components) Application-specific challenges Conversational AI systems: contextual explanations Gradation of explanations Tools for explanations across AI lifecycle Pre & post-deployment for ML models Model developer vs. End user focused
  122. 122. Reflections ● Case studies on explainable AI in practice ● Need “Explainability by Design” when building AI products 122
  123. 123. Thanks! Questions? ● Feedback most welcome :-) ○ krishna@fiddler.ai, sgeyik@linkedin.com, kenthk@amazon.com, vamithal@linkedin.com, ankur@fiddler.ai ● Tutorial website: https://sites.google.com/view/fat20-explainable-ai- tutorial ● To try Fiddler, please send an email to info@fiddler.ai 123

×