Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Evaluating Machine Learning Models: A Case Study

143 vues

Publié le

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2ya9hxV.

Nelson Ray talks about on how to estimate the business impact of launching various machine learning models, in particular, those Opendoor uses for modeling the liquidity of houses. Filmed at qconnewyork.com.

Nelson Ray manages the Risk Science group at Opendoor in San Francisco. His team is responsible for per-home liquidity estimation and developing responsive risk models. Prior to joining Opendoor, he was a data scientist at Google and a software engineer at Metamarkets.

Publié dans : Technologie
  • Soyez le premier à commenter

  • Soyez le premier à aimer ceci

Evaluating Machine Learning Models: A Case Study

  1. 1. A Case Study in Real Estate QCon New York June 2017 Nelson Ray Business Backtesting of ML Models:
  2. 2. InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ opendoor-machine-learning
  3. 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
  4. 4. Who has run an A/B test before?
  5. 5. Did it go off without a hitch?
  6. 6. Unguided A/B Testing
  7. 7. Focus of this Talk A/B Test Quasi-experiments Simulation-Based Inference Observational Analysis CostConfidence
  8. 8. Talk Structure • Real Estate 101 for Home Buyers and Sellers • The Opendoor Way • Resale Risk • Problems with A/B Testing • How Simulation Helped in Real Estate • A General Recipe for Simulation • Team Info.
  9. 9. A/B Testing Support Group Hi, I’m Nelson! I’m a recovering A/B testing user from such places as… • Facebook • Metamarkets • Google • Opendoor I’ll cover how to perform a business backtest of your ML models using simulations! Introduction
  10. 10. M I S S I O N Empower everyone with the freedom to move $25T of assets 63.5% of Americans are homeowners #1 consumer expenditure ($17,798/yr) $1.4T of annual transaction volume $100B in fees
  11. 11. Research online Receive bids Interview Choose Decide to move 100+ day process with 14% failure rate Seller Improvements Yard work Cleaning Photographs MLS, Zillow, Trulia Open houses Showings Maintenance Offer Counter, Acceptance Inspection Financing period Final walkthrough Offline signatures Title records Offer Inspection Financing Realtor Sale Ready List Contract Closing Contract C U R R E N T P R O C E S S 5.5M Americans per year buy and sell through this process Decides to move Buyer Finances Location Timing Discovery Research online Receive bids Interview Choose Open houses Showings Viewings RealtorSearch 100+ day process with friction at each step S E L L E R S B U Y E R S DAY 0 DAY 121$1000’s in upfront costs 90+ Days 4-5% in price drops and concessions 14% of deals fall- through 6-7% in fees Months of research and gathering data Months of viewing suboptimal listings Fears of the home condition and financing DAY 0
  12. 12. Research online Receive bids Interview Choose Decide to move 100+ day process with 14% failure rate Seller Improvements Yard work Cleaning Photographs MLS, Zillow, Trulia Open houses Showings Maintenance Offer Counter, Acceptance Inspection Financing period Final walkthrough Offline signatures Title records Offer Inspection Financing Realtor Sale Ready List Contract Closing Contract C U R R E N T P R O C E S S 5.5M Americans per year buy and sell through this process Decides to move Buyer Finances Location Timing Discovery Research online Receive bids Interview Choose Open houses Showings Viewings RealtorSearch 100+ day process with friction at each step S E L L E R S B U Y E R S DAY 0 DAY 121$1000’s in upfront costs 90+ Days 4-5% in price drops and concessions 14% of deals fall- through 6-7% in fees Months of research and gathering data Months of viewing suboptimal listings Fears of the home condition and financing DAY 0
  13. 13. toexperienceanautomated, hassle-freesalesprocess. Simplyenteryouraddress toensurewecanaccuratelyprice yourhome. Filloutashorthomeprofile withafullreportofyourhome’s value. Andreceiveanofferinminutes Sellingyourhomeisaseasyas clickingnext S E L L E R S
  14. 14. Research online Receive bids Interview Choose Decide to move 100+ day process with 14% failure rate Seller Improvements Yard work Cleaning Photographs MLS, Zillow, Trulia Open houses Showings Maintenance Offer Counter, Acceptance Inspection Financing period Final walkthrough Offline signatures Title records Offer Inspection Financing Realtor Sale Ready List Contract Closing Contract C U R R E N T P R O C E S S 5.5M Americans per year buy and sell through this process Decides to move Buyer Finances Location Timing Discovery Research online Receive bids Interview Choose Open houses Showings Viewings RealtorSearch 100+ day process with friction at each step S E L L E R S B U Y E R S DAY 0 DAY 121$1000’s in upfront costs 90+ Days 4-5% in price drops and concessions 14% of deals fall- through 6-7% in fees <50% have a bachelor’s degree Months of research and gathering data Months of viewing suboptimal listings Fears of the home condition and financing DAY 0
  15. 15. B U Y E R S Thousandsofbuyersshop withusmonthly Allhomescomewitha money-backguarantee anda2-yearwarranty Searchingandshowingsare self-service,on-demand Buyingahomeisaseasyas clickingnext Ourbuyershaveexclusive accesstoourinventory
  16. 16. What is our risk in reselling a home?
  17. 17. • Listed ~$800k • 6+ months on market Home 1
  18. 18. Home 2 • Listed ~$300k • 1 month on market
  19. 19. • Opendoor bears risk in reselling the house • Costs vary substantially by house • Fair to each seller to charge based on their expected cost Our Philosophy
  20. 20. Framing the problem
  21. 21. House Economics Conversion Fee Profit Fee
  22. 22. Formalization • Infinite number of pricing models • Assuming we even had a candidate f’, how do we test this? • A/B testing approach • randomize on offers: f vs f’ • evaluate {# of houses, profit}
  23. 23. Metric Measurement Lag • Time to observe # • days • Time to observe $ • months
  24. 24. Formalization • Infinite number of pricing models • Assuming we even had a candidate f’, how do we test this? • A/B testing approach • randomize on offers: f vs f’ • evaluate {# of houses, profit} • Many months of measurement lag
  25. 25. Simulating Offers • Historical transaction data • House lists on the market • Simulate our buying process • Estimate our costs • Observe actual outcome for house
  26. 26. Simulating Offers Actual resale cost: $50k Expected resale costs • funder: $10k -> {Paccept: .9, $: -40k} • fbase: $55k -> {Paccept: .1, $: 5k} Actual resale cost: $10k Expected resale costs • funder: $5k -> {Paccept: .9, $: -5k} • fbase: $8k -> {Paccept: .7, $: -2k}
  27. 27. Simulating Offers $ # fbase funder
  28. 28. Simulating Offers $ # fbase funder f2base
  29. 29. Simulating Offers $ # fbase funder f2base f3base
  30. 30. Simulating Offers $ # fbase funder f2base f3base f4base
  31. 31. Simulating Offers $ # fbase funder f2base f3base f4base
  32. 32. Simulating Offers $ # fbase funder f2base f3base f4base
  33. 33. Simulating Offers $ # fbase funder f2base f3base f4base $ # fbase funder
  34. 34. Simulating Offers $ # fbase funder f2base f3base f4base $ # fbase funder
  35. 35. Understanding Current Trade-Offs $ # fbase funder f2base f3base f4base $ # fbase funder • Clarity into trade-offs • Identify suitable candidates • Backtesting with business metrics • Seconds vs months • Only cost is computational • Though quality dependent on simulation models
  36. 36. Estimating Future Trade-Offs $ # fbase funder f2base f3base f4base
  37. 37. Estimating Future Trade-Offs $ # fbase funder f2base f3base f4base Cdesired
  38. 38. Estimating Future Trade-Offs $ # fbase funder f2base f3base f4base Cdesired Coracle
  39. 39. Oracle Performance Actual resale cost: $50k Expected resale costs • funder: $10k -> {Paccept: .9, $: -40k} • fbase: $55k -> {Paccept: .1, $: 5k} • foracle: $50k -> {Paccept: .15, $: 0k} Actual resale cost: $10k Expected resale costs • funder: $5k -> {Paccept: .9, $: -5k} • fbase: $8k -> {Paccept: .7, $: -2k} • foracle: $10k -> {Paccept: .65, $: 0k}
  40. 40. Estimating Future Trade-Offs $ # fbase funder f2base f3base f4base Cdesired Coracle
  41. 41. Estimating Future Trade-Offs $ # fbase funder f2base f3base f4base Cdesired CoracleCBayes
  42. 42. $ # fbase funder f2base f3base f4base C20% CoracleCBayes • Estimate what is theoretically achievable • Set ML improvement goal • Translation into business trade-offs • “Easy” part is to hit ML target Estimating Future Trade-Offs
  43. 43. Simulation Accuracy
  44. 44. Unguided A/B Testing
  45. 45. The Guide
  46. 46. Guided A/B Testing
  47. 47. Pyramid of Causal Inference A/B Test Quasi-experiments Simulation-Based Inference Observational Analysis CostConfidence
  48. 48. Recipe for guided testing
  49. 49. Simulating Offers • Historical transaction data • House lists on the market • Simulate our buying process • Estimate our costs • Observe actual outcome for house Data generating process User model
  50. 50. Recipe: Data Generating Process Simple version: replay historical data • Home buying and selling • Past housing transactions • Ridesharing services • Passenger app sessions • Search engine ad auctions • Stock of potential ads
  51. 51. Recipe: User Model • Home buying and selling • P(sell | cost) • Ridesharing services • P(accept ride | price, ETA) • Search engine ad auctions • P(click | user features, ad features)
  52. 52. A/B test responsibly
  53. 53. Simulate before testing
  54. 54. • Founded: March 2014 • Transactions / month: 500 • Number of employees: 300 • 50 data scientists and engineers • We’re hiring! • E-mail: nelson@opendoor.com Opendoor by the numbers
  55. 55. Title Text Title Text Acknowledgements
  56. 56. Q&A
  57. 57. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ opendoor-machine-learning

×