15. Why is the ANTISPAM PROBLEM hard Scale of the problem; 25B Connections, 5B deliveries, 450M mailboxes User feedback is often late, noisy and not always actionable Large, diverse stream of legitimate traffic that looks like spam Slow adoption of authentication technologies like DKIM and SPF Spammers are clever; target and specialize attacks Rapidly changing spam campaigns with a large bot controlled IP base; large variations even within a single campaign A significant percentage of spam comes from large ESPs like Hotmail, Google and Yahoo 15
16. Generation 1: Manual management layer Heuristics, blocks, blacklists Provide attack mitigation and operational flexibility, highly explainable. Not durable, expensive to keep pace with fast morphing spam Ad hoc queries Proprietary implementations, not very scalable, steep learning curve Reactive and usually late 16
17. Generation 2: Machine Management Layer Online reputation models Simple, mostly scoring/counter/ratio based models Highly scalable due the absence of any state/memory Generalize too broadly, lack expressive power Batch trained reputation models Typically digested memory based hashing or machine learning models Difficult to implement and due to the need for labeled examples scale well only moderately Slow to update and learn, lack explainability, limited operational control 17
27. Lets REVIEW OUR DESIGN GOALs AGAIN Classifiers are notorious for lack of explainability Engineers and analysts needs to know what the classifier is missing Engineers and analysts need to know about emerging threats Analysts need “canned” reports along interesting dimensions Machines need smart feature engineering Develop a scalable system to provide deep insight into spammer campaigns Double up as a platform for standard reporting Also double up as a platform for adhoc analysis and data probing Signal amplification and smart feature extraction platform 26
28. Our ANTISPAM ANALYTIC PLATFORM Hadoop: Implements map reduce, written in Java but supports many other languages including Perl and C++ using the streaming interface Feature engineering with small simple Perl programs for data extraction and transformation SQL-like “Pig” programming language for data analysis and management Mahout: data mining libraries that provide shrink- wrapped, scalable, sophisticated algorithms Other proprietary algorithms and frameworks for specialized tasks 27
29. Various ASPECTS of A GRID DRIVEN SOLUTION Standard reporting Ad hoc querying Campaign discovery from spam feedback using frequent item set mining “Gaming” detection in notspam feedback using connected components 28
31. AD HOC queries for ANTISPAM research Identify domains that had few spam votes in the previous time window but have a high number of spam votes today All IPs in the last hour that sent a particular URL pattern…or that sent any unknown URL >500 times Which domains/IPs suddenly increased their sending volume after a positive reputation change Which FROM addresses exhibit low message size entropy All messages that had nothing but a URL and the domain of the URL had low page rank 30
32. AD HOC QUERIES - Anatomy of a PIG QUERY --- This includes some basic string functions, including splitting a string on the '@' character register /homes/jpujara/pig_scripts/string.jar; define splitEmail string.Tokenize('2','@'); --- Load up some data - incoming messages at a date and time, and our trusted user database MESSAGES = load '/projects/antispam/mta_feature_logs/$date*/*/*-$time*' using com.yahoo.ymail.pigfunctions.AsStorage('__record_key__,firstrcpt,mailfrom') as (mid:chararray,to:chararray,from:chararray); USERS = load '/projects/antispam/TrustedUser.bz2' using com.yahoo.ymail.pigfunctions.AsStorage('user,t') as (user:chararray,trusted:int); --- Split the e-mail addresses into user+domain and generate the appropriate user-id for yahoo users and partners EXPLODED_MESSAGES = FOREACH MESSAGES GENERATE to,FLATTEN(splitEmail(to)) as (user,udomain),FLATTEN(splitEmail(from)) as (sender,sdomain); YAHOO_MESSAGES = FOREACH EXPLODED_MESSAGES GENERATE (udomain MATCHES '.*yahoo.*' ? user : to ) as yuser,sdomain; 31 --- Combine the message and sender domains with the trusted user data and select only trusted messages YAHOO_MESSAGES_TRUST = JOIN YAHOO_MESSAGES by yuser, USERS by user; TRUSTED_MESSAGES = FILTER YAHOO_MESSAGES_TRUST by trusted > 0; --- Group by domain, and generate a count, order by descending count DOMAIN_GROUPS = GROUP TRUSTED_MESSAGES by sdomain; DOMAIN_GROUPS_COUNT = FOREACH DOMAIN_GROUPS GENERATE group,COUNT(TRUSTED_MESSAGES) as count; DOMAIN_GROUPS_ORDER = ORDER DOMAIN_GROUPS_COUNT by count DESC; --- Output the results STORE DOMAIN_GROUPS_ORDER into '$targetdir/topDomains';
33. CAMPAIGN Discovery in SPAM Feedback Frequent Itemset Mining Classical method Research interesting relationships between variables in a large database Primarily applied for market basket analysis Many good implementations APRIORI Easy to implement Parallelizes moderately well but bottlenecks for extremely large data sets Not very efficient with the number scans ECLAT Parallelizes easily Amenable to a good grid implementation Fewer scans of the dataset Parallel FP GROWTH Designed explicitly for systems like hadoop Implemented in Mahout 0.2 32
36. Frequent itemset MINING on ONE DAY’s SPAM REPORTS 9 2595 (IPTYPE:none,FROMUSER:sales,SUBJ:It's Important You Know,FROMDOM:dappercom.info,URL:dappercom.info,ip_D:66.206.14.77,) 9 2457 (IPTYPE:none,FROMUSER:sales,SUBJ:Save On Costly Repairs,FROMDOM:aftermoon.info,URL:aftermoon.info,ip_D:66.206.14.78,) 9 2447 (IPTYPE:none,FROMUSER:sales,SUBJ:Car-Dealers-Compete-On-New-Vehicles,FROMDOM:sherge.info,URL:sherge.info,ip_D:66.206.25.227,) 9 2432 (IPTYPE:none,FROMUSER:sales,SUBJ:January 18th: CreditReport Update,FROMDOM:zaninte.info,URL:zaninte.info,ip_D:66.206.25.227,) 9 2376 (IPTYPE:none,FROMUSER:health,SUBJ:Finally. Coverage for the whole family,FROMDOM:fiatchimera.com,URL:articulatedispirit.com,ip_D:216.218.201.149,) 9 2184 (IPTYPE:none,FROMUSER:health,SUBJ:Finally. Coverage for the whole family,FROMDOM:fiatchimera.com,URL:stratagemnepheligenous.com,ip_D:216.218.201.149,) 9 1990 (IPTYPE:none,FROMUSER:sales,SUBJ:Closeout 2008-2009-2010 New Cars,FROMDOM:sastlg.info,URL:sastlg.info,ip_D:66.206.25.227,) 9 1899 (IPTYPE:none,FROMUSER:sales,FROMDOM:brunhil.info,SUBJ:700-CreditScore-What-Is-Yours?,URL:brunhil.info,ip_D:66.206.25.227,) 9 1743 (IPTYPE:none,FROMUSER:sales,SUBJ:Now exercise can be fun,FROMDOM:accordpac.info,URL:accordpac.info,ip_D:66.206.14.78,) 9 1706 (IPTYPE:none,FROMUSER:sales,SUBJ:Closeout 2008-2009-2010 New Cars,FROMDOM:rionel.info,URL:rionel.info,ip_D:66.206.25.227,) 9 1693 (IPTYPE:none,FROMUSER:sales,SUBJ:January 18th: CreditReport Update,FROMDOM:astroom.info,URL:astroom.info,ip_D:66.206.25.227,) 9 1689 (IPTYPE:none,FROMUSER:sales,SUBJ:eBay: Work@Home w/Solid-Income-Strategies,FROMDOM:stamine.info,URL:stamine.info,ip_D:66.165.232.203,) 35 2432 (IPTYPE:none,FROMUSER:sales,SUBJ:January 18th: CreditReportUpdate,FROMDOM:zaninte.info,URL:zaninte.info, ip_D:66.206.25.227,) 2447 (IPTYPE:none,FROMUSER:sales,SUBJ:Car-Dealers-Compete-On-New-Vehicles,FROMDOM:sherge.info,URL:sherge.info, ip_D:66.206.25.227,)
37.
38. Delays classification of spamming IP addressesThrows off the classifiers if the feedback is not filtered well Model the problem as a bipartite graph Well known model for matching algorithms Broadly applied in various fields like coding theory A graph whose vertices are disjoint form disjoint sets U,V There is an edge connecting every U to a vertex in V 36
39. Connected COMPONETS - EXPLAINED Y1 = Yahoo user 1, Y2 = Yahoo user 2 IP1 = IP address of the host Y1 “voted” notspam from 37 y1 IP1 y1 SQUARING weight = 2 y1 IP2 y1
40. Connected COMPONENTS for “GAMING” DETECTION 38 Set of IPs/YIDs used exclusively for voting notspam Set of (likely new) spamming IPs which are “worth” voting for y1 IP3 IP1 y2 IP4 IP2 y3 Set of “voted on” IPs Set of “voted from” IPs Set of Yahoo IDs voting notspam
41. Connected Components - RESULTS 39 - Connnected components for IPsnotspam was voted from
43. CONCLUSIONS We have had success leveraging parallel, stateful algorithms on grid systems to keep pace with polymorphic spam that evade traditional analysis and algorithms Frequent Itemset Mining rapidly identifies cohesive campaigns in ISSPAM feedback Connected Components amplifies weak signals in gamed NOTSPAM feedback and helps separate signal from noise in the feedback Grid system based analysis platforms may be broadly applicable across the security domain 41
44. Apply Slide Download Hadoop distribution http://hadoop.apache.org Try out Pig on standalone, single Linux box Identify source data to aggregate Start simple: IP patterns across web access logs Begin with offline aggregation; yesterday’s attacks still interesting Read Connected Components and Frequent Itemset Mining papers Stop looking for a single, invariant “tell” – far too costly Start thinking about co-occurrence of innocuous features 42
45. Resources for implementers Hadoop setup, documentation and resources http://hadoop.apache.org/ Pig documentation and resources http://hadoop.apache.org/pig/ Mahout documentation and resources http://lucene.apache.org/mahout/ Frequent itemset mining implementation repository http://fimi.cs.helsinki.fi/src/ Connected components description [link not yet live] Ranger, Raghuraman, Penmetsa, Bradski, and Kozyrakis. Evaluating MapReduce for Multi-core and Multiprocessor Systems. In HPCA 2007 43
Who knows what Yokai are? <audience poll>Shape-shifters from Japanese mythology. Many other examples, e.g. Proteus, who would tell you the future, but first you had to capture him. Just like the gods, Change shape to avoid capture* vary over IP, vary over content, vary over template features (e.g. document structure, subjects, size entropy)
In abuse, these are “shape shifters.”They vary many aspects of the message to avoid detection: IPSubjectContentFor example, these four messages are obviously built from a single template, but changing its shape to avoid capture. How to catch?In the past: + Heuristics & Regex + Dictionary (URLdb) + Invariant metadataChallenges: + slow to write+ difficult to write+ easy to evade
Here is a third type of shape-shifting spamFor all of these: attackers have distinct advantage, because they can change most aspects and still get through
1.3 sextillion (1.3e21) variations, almost all can be recognized by human being in milisecondsspammers learned they can change any variable to hide from bulk filtershttp://cockeyed.com/lessons/viagra/viagra.html
These bastards… the most despised doctors on the InternetAlmost all pages resolve through numerous HTML/Javascript redirectors to this page
Daniel Geer said, there are targets of CHANCE and targets of CHOICE. Small businesses are in the former camp, catching the miscellaneous attacks out there.Increasingly, larger companies are TARGETS OF CHOICE, meaning the bad guys a) specifically tailor their attacks based on known vulnerabilities, and b) use feedback loops to improve the effectiveness of them.
This is what a targetted attack profile looks like: After you patch, they almost stop trying
One example of such a clearly targeted attack: 400KB of style gibberish embedded in a style sheet, completely throws out our parsersMaybe ASCII art spam, or something else that couldn’t be caught by simple pattern matchingThis is what our filters see: a stream of ASCII that is deliberately using multiple layerse.g. here, a TinyURL redirector, further obfuscated with non-printing HTML, spaces, and CSS chaffTo fight in olden days, hand-written regex to identify a patternOR heuristic on some invariant part of the message. But what is invariant? dozens of TinyURL clonesdozens of HTML and CSS tricks2^32 IP addressesinfinite FROM addressesinfinite SUBJECT lines…
Sent by botnetsThis is Reactor Mailer; controlled Srizbi from the McColo datacenters until Nov 2008This is the template for Stormbot; notice it has control variables for all the settingsWhile most of these came in through SMTP port 25, now they are increasingly hitting HTTP and port 80
Historically, POINT SOLUTIONS address each problem individuallyregexheuristicWouldn’t this be better if this guy could use more than one finger at a time?Something is *almost over the limit* along one dimension and *almost over the limit* along another.Message from IP that sends 80% good mail, with tinyurl that we don’t recognize, that was addressed to 40 people.*PRIOR PROBABILITY**COMPOSITE SCORE*
Scale forces simplistic architectures; Feedback based architectures always lag behind the spam campaignFeedback also has many segments;- Personal preference spam: “I didn’t like this week’s Amazon gold box deals but I liked last week’s messages from Amazon”- Annoyance emails from legitimate bulk mailers: “This coupon is coming far too often these days”-Listserver spam: “This finance group - Newsletter messages that are no longer interesting to the user: “Gosh I am so not into that band any more”sometimes sends me stock spam”Traffic to a small enterprise domains can be restricted with firewall rules etc but large free mail provider traffic is full of corner casesCompounding the problem is the fact that adoption of DKIM and SPF has been slow, especially internationally and in emerging economies.But make no mistake, some of these spammers are very cleverIts more fruitful to target yahoo or google than to build a generic spam engine
Lets looks at what is in place right now in terms of an architecture; Most large scale systems have some components from gen1 technologiesProvide attack mitigation and operational flexibility, highly explainable. Not durable, expensive to keep pace with fast morphing spamProprietary implementations, not very scalable, steep learning curveReactive and usually late
Two ways this has been solved in the past: Machine management…Both systems, because of scale, were limited to looking at small pieces of data – an IP, a URL, etc.
In this talk we’ll introduce Hadoop, an open-source grid computing environment with applications to fighting abuse. We’ll talk about how Hadoop can be applied to polymorphic spam and abuseAbout three years ago, Doug Cutting released version 0.15 of Hadoop, an open-source platform inspired by Google’s proprietary Map:Reduce algorithm“Supercomputer” – petabytes of storage, terabytes of RAM allow “needle in the haystack” even at Y!Mail scalehundred of featureshundreds of billions of recordstrends buried in global data
Hadoop is the most prevalentAlso “Ngrid” and “Sun’s GridEngine” are other alternatives
Input data format is application-specific, specified by the user Output is a set of <key,value> pairs User expresses algorithm using two functionsMap is applied on the input data and produces a list of intermediate <key,value> pairs Reduce is applied to all intermediate pairs with the same key. It typically performs some kind of merging operation and produces zero or more output pairs Finally, output pairs are sorted by their key value
Toy exampleProvide some insight into what a map reduce program looks like, looks very much like unix command line
Java code to highlight the mapper, mapper simply adds each word to a set and emits a count of 1 for each time the word is seen
The reducer simply sums the values for each word, draw attention to line 32While this is a toy example, it should give a fair idea about how to structure a problem to be solvable by map reduce. The key takeaway is that writing even native map reduce programs can be quite simple and executing it even simpler
Take the audience progressively through more and more sophisticated applications, starting from basic reporting and ending in outbound spammer analysis based on SWARM features
Knowing the accurancy of your SVM/Bayes classifier puts you in no better situation to ask and answer what type of spam is leaking; and we know spammers are constantly probing80% of the spam/content classification problem is in smart feature engineering
Lets looks at how our/Yahoo’s platform looks like Perl programs for feature engineering make it very easy and flexibleHadoop with its pig support is already well suited as a platform for adhoc data analysisFor deep data mining, open source mahout
We will look at the hadoop is four different settings;
* In antispam, these basic reports combined with human review form a barrier against highly directed attacks that exploit system weaknesses* Note how easy it is to slice and dice your data and write fairly sophisticated reports using pig/streaming. It is critical in antispam systems that the reporting platform be flexible and provide a lot of expressive power, hadoop and pig achieve that.*
Previous such queries were against small samples, now we can do it against the full data set and get highly accurate results in a very short amount of timeAlternate architectures such as OLAP are too expensive at this scale
* Pig is a data flow specification language. Its like SQL but unlike SQL it is better suited for data flow control. * In antispam, these basic reports combined with human review form a barrier against highly directed attacks that exploit system weaknessesNote how easy it is to slice and dice your data and write fairly sophisticated reports using pig. It is critical in antispam systems that the reporting platform be flexible and provide a lot of expressive power, hadoop and pig achieve that.*
-- People who bought eggs also bought bread
* We ran frequent itemset on one day’s spam votes, the results are striking.* Notice in the above example how the same campaign [the same FROMUSER] is being managed with different templates for subjects and URLs and is also originating from different IPs* Others records in the background are the result of the freq itemset mining algorithm as well and map very closely with spam campaigns.
Develop a bipartite graph of users and the IPs they vote fromSquaring of the graph give rise to connected componentWeight of the connected component is a measured by the number of vertices that share the component.
GamingIPs are IPs that the spammers try to whitelist in advanceDetected them by extending the connected component view on Ips the notspam is voted on
The results are quite spectacular!! There is a massive amount of “gaming” going on with “notspam feedback” and there are only a handful of Ips that are doing this. There are a large number of smaller components not shown in the results above
The results are less stronger – notice the two smaller weaker clusters in row 3 and 4The big takeway is that such unsupervised matching algorithms are going to be extremely power amplifiers of signals and can be used to rapidly separate out noise from signal.Imagine this being applied on traffic with more items such as IPs, message subjects, size of messages, fuzzy signatures etc.
We encourage and invite others to try hadoop in anti spam and anti abuse architectures and share their experiences with us.
Three users known badsame IP leads to new cookiesame cookie leads to new birthdayetc.*AMPLIFICATION OF SMALL SIGNAL*