"You can't just turn the crank": Machine learning for fighting abuse on the consumer web
•
1 j'aime•812 vues
Fighting fake registrations, phishing, spam and other types of abuse on the consumer web appears at first glance to be an application tailor-made for machine learning: you have lots of data and lots of features, and you are looking for a binary response (is it an attack or not) on each request. However, building machine learning systems to address these problems in practice turns out to be anything but a textbook process. In particular, you must answer such questions as: - How do we obtain quality labeled data? - How do we keep models from "forgetting the past"? - How do we test new models in adversarial environments? - How do we stop adversaries from learning our classifiers? In this talk I will explain how machine learning is typically used to solve abuse problems, discuss these and other challenges that arise, and describe some approaches that can be implemented to produce robust, scalable systems.
Recommandé
Recommandé
Contenu connexe
Similaire à "You can't just turn the crank": Machine learning for fighting abuse on the consumer web
Similaire à "You can't just turn the crank": Machine learning for fighting abuse on the consumer web (20)
Dernier
Dernier (20)
"You can't just turn the crank": Machine learning for fighting abuse on the consumer web
- 1. "You can't just turn the crank" Machine learning for fighting abuse on the consumer web David Freeman Research Scientist/Engineer, Facebook Inc. ScAINet 2018 Atlanta, GA USA, 11 May 2018
- 3. What do they try to do? Malware Payment Fraud Scraping Click Fraud Phishing Spam Social Engineering Fake Products Scams "Like" FraudPromotion Fraud Identity Theft What do we see? Fake Reviews Misinfor- mation Financial Theft Account Resale Fundamental question: Which requests are bad? • Perfect for machine learning!
- 4. What could possibly go wrong? Machine learning workflow Label Train Validate Launch Measure Profit!Lots!
- 5. How do we obtain labeled data? (hint: not from your users) Machine learning workflow Label
- 6. • Human labeling of random samples. • Labelers don't always know what they're looking for • Labelers are inconsistent (with themselves and each other) • Labelers get tired (esp. if most samples are good) • Apply crowdsourcing best practices: • Precise definitions, multiple labeling, ML-assisted sampling • But will it scale? Labeling: Gold standard Objective measurement
- 7. • Find high-precision signals of badness • Examples: unusual user-agent, malformed header • DO NOT BLOCK ON THESE SIGNALS • They are controlled by the adversary • When the adversary adapts you will lose visibility • Automatically generate signals using anomaly detection. Labeling: Silver standard Automatic labeling
- 8. • Use whatever you have! • CS data, rules, other models • Mitigate risks of blindness and feedback loops: • Oversample manually labeled examples • Oversample false positives and false negatives when retraining. • Undersample positive examples from previous iterations of this model. • Sample and label examples near the decision boundary Labeling: Bronze standard Be scrappy
- 9. • Users are terrible at reporting. • Product flows bias reporting. • Reports can be gamed. • Reports can serve as an directional measure. Labeling: Iron standard Have your users do the work
- 10. • Segment the problem • e.g. status with link from country X • Downsample intelligently • if your distribution is lumpy, sample from all the lumps • Learning the prior vs. focusing on the bad stuff • no golden rule here -- you have to experiment Assembling a training set Labeling is just the beginning
- 11. {Training set 2 { Model v2 Refreshing your data Don't forget the past! {Training set 1 { Model v1 Mitigation: • Keep old attacks around (exponential decay?) • Keep old models around (raise thresholds?) { Training set N { Model vN
- 12. How do you know your model is ready to go? Machine learning workflow Train Validate
- 13. • Labels aren't perfect • Often miss on recall • Models interact with each other • Use offline P-R and ROC to stack-rank model candidates Validating Performance Don't trust offline replay Model B FP Model A
- 14. • Fundamental A/B testing assumption: Experiment effects are independent of the cohorts chosen The Perils of A/B Testing
- 15. The Perils of A/B Testing A B X • Looks good so far.... Start with a small experiment
- 16. The Perils of A/B Testing A B X • Did the adversary give up or iterate? Roll it out to (almost) everyone — Option 1
- 17. The Perils of A/B Testing A B • Now your experiment is a vulnerability Roll it out to (almost) everyone — Option 2
- 18. • Run new model online in "log-only" mode • Evaluate performance where the new model disagrees with the old one. • ideally via sampling & labeling • Push based on FP/FN tradeoff Using Shadow Mode Prod model FP New model
- 19. How do you figure out if it worked? Machine learning workflow Launch Measure
- 20. True Positives Don't Matter What's happening here? Time Precision
- 21. • Really want # of good users affected • Solution: use one minus specificity (aka FPR) True Positives Don't Matter What's happening here? Time TP Time FP vs. Time FP Time TP 1 TN FP + TN<latexit sha1_base64="FfpNUyvhzZ48h+ueQ2Hy3cgzX50=">AAACDXicbZDLSgMxFIYz9VbrbdSlm2AVBLHMiKDLoiCupEJv0BlKJs20oUlmSDJiGeYF3Pgqblwo4ta9O9/GtJ2Ftv4Q+PKfc0jOH8SMKu0431ZhYXFpeaW4Wlpb39jcsrd3mipKJCYNHLFItgOkCKOCNDTVjLRjSRAPGGkFw6txvXVPpKKRqOtRTHyO+oKGFCNtrK594MIT6IUS4dTjQfSQ1m+zLMfrGjyG43vXLjsVZyI4D24OZZCr1rW/vF6EE06Exgwp1XGdWPspkppiRrKSlygSIzxEfdIxKBAnyk8n22Tw0Dg9GEbSHKHhxP09kSKu1IgHppMjPVCztbH5X62T6PDCT6mIE00Enj4UJgzqCI6jgT0qCdZsZABhSc1fIR4gE402AZZMCO7syvPQPK24TsW9OytXL/M4imAP7IMj4IJzUAU3oAYaAINH8AxewZv1ZL1Y79bHtLVg5TO74I+szx8ZwZry</latexit><latexit sha1_base64="FfpNUyvhzZ48h+ueQ2Hy3cgzX50=">AAACDXicbZDLSgMxFIYz9VbrbdSlm2AVBLHMiKDLoiCupEJv0BlKJs20oUlmSDJiGeYF3Pgqblwo4ta9O9/GtJ2Ftv4Q+PKfc0jOH8SMKu0431ZhYXFpeaW4Wlpb39jcsrd3mipKJCYNHLFItgOkCKOCNDTVjLRjSRAPGGkFw6txvXVPpKKRqOtRTHyO+oKGFCNtrK594MIT6IUS4dTjQfSQ1m+zLMfrGjyG43vXLjsVZyI4D24OZZCr1rW/vF6EE06Exgwp1XGdWPspkppiRrKSlygSIzxEfdIxKBAnyk8n22Tw0Dg9GEbSHKHhxP09kSKu1IgHppMjPVCztbH5X62T6PDCT6mIE00Enj4UJgzqCI6jgT0qCdZsZABhSc1fIR4gE402AZZMCO7syvPQPK24TsW9OytXL/M4imAP7IMj4IJzUAU3oAYaAINH8AxewZv1ZL1Y79bHtLVg5TO74I+szx8ZwZry</latexit><latexit sha1_base64="FfpNUyvhzZ48h+ueQ2Hy3cgzX50=">AAACDXicbZDLSgMxFIYz9VbrbdSlm2AVBLHMiKDLoiCupEJv0BlKJs20oUlmSDJiGeYF3Pgqblwo4ta9O9/GtJ2Ftv4Q+PKfc0jOH8SMKu0431ZhYXFpeaW4Wlpb39jcsrd3mipKJCYNHLFItgOkCKOCNDTVjLRjSRAPGGkFw6txvXVPpKKRqOtRTHyO+oKGFCNtrK594MIT6IUS4dTjQfSQ1m+zLMfrGjyG43vXLjsVZyI4D24OZZCr1rW/vF6EE06Exgwp1XGdWPspkppiRrKSlygSIzxEfdIxKBAnyk8n22Tw0Dg9GEbSHKHhxP09kSKu1IgHppMjPVCztbH5X62T6PDCT6mIE00Enj4UJgzqCI6jgT0qCdZsZABhSc1fIR4gE402AZZMCO7syvPQPK24TsW9OytXL/M4imAP7IMj4IJzUAU3oAYaAINH8AxewZv1ZL1Y79bHtLVg5TO74I+szx8ZwZry</latexit><latexit sha1_base64="FfpNUyvhzZ48h+ueQ2Hy3cgzX50=">AAACDXicbZDLSgMxFIYz9VbrbdSlm2AVBLHMiKDLoiCupEJv0BlKJs20oUlmSDJiGeYF3Pgqblwo4ta9O9/GtJ2Ftv4Q+PKfc0jOH8SMKu0431ZhYXFpeaW4Wlpb39jcsrd3mipKJCYNHLFItgOkCKOCNDTVjLRjSRAPGGkFw6txvXVPpKKRqOtRTHyO+oKGFCNtrK594MIT6IUS4dTjQfSQ1m+zLMfrGjyG43vXLjsVZyI4D24OZZCr1rW/vF6EE06Exgwp1XGdWPspkppiRrKSlygSIzxEfdIxKBAnyk8n22Tw0Dg9GEbSHKHhxP09kSKu1IgHppMjPVCztbH5X62T6PDCT6mIE00Enj4UJgzqCI6jgT0qCdZsZABhSc1fIR4gE402AZZMCO7syvPQPK24TsW9OytXL/M4imAP7IMj4IJzUAU3oAYaAINH8AxewZv1ZL1Y79bHtLVg5TO74I+szx8ZwZry</latexit>
- 22. Not so fast! Machine learning workflow Profit!Adapt!
- 23. What not to Do (I) Show the adversary what your limits are Message 500 people Message 400 people Message 300 people 🛑 🛑 ✅
- 24. • Introduce delay in blocking response (and/or) • Undo the damage without telling the user. What to do (I) Don't give immediate feedback
- 25. "We don't want to be the ones solving the CAPTCHAs" What not to Do (II) Look for specific content to block
- 26. What to Do (II) Focus on bad behavior, not only bad content
- 27. What to Do (III) Use data the adversary doesn't know/control
- 28. Scoring at Entry Points prevent access to accounts Clustering, Anomaly Detection prevent accounts from doing damage User Reporting find false negatives Behavioral Analysis detect bad activityIncreasing speed More information available What to Do (IV) Defense in depth
- 29. • Think about each step of the ML process. • It's hard to build a good training set. • Adversarial adaptation breaks many assumptions. • Control the data & the response. Take aways Thanks to: Hervé Robert, Isaac Fullinwider, Henry Lu, Sagar Patel, Hongyang Li, Nektarios Leontiadis