Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Orchestrating the Intelligent Web with Apache Mahout

4 908 vues

Publié le

Presentation on Apache Mahout at Linux Conference Australia 2011

Publié dans : Technologie, Formation

Orchestrating the Intelligent Web with Apache Mahout

  1. 1. Orchestrating the Intelligent Web with Apache Mahout Presented by Aneesha Bakharia Twitter: aneesha Email: aneesha.bakharia@gmail.com
  2. 2. What is Apache Mahout? <ul><li>Open source </li></ul><ul><li>Machine Learning Java library </li></ul><ul><li>Scalable (Apache Hadoop) </li></ul><ul><li>Framework for developing, testing and deploying large-scale algorithms http://mahout.apache.org/ </li></ul>
  3. 3. What’s in a Name? <ul><li>Mahout is Hindi for Elephant Driver </li></ul>
  4. 4. What is Apache Mahout? <ul><li>Framework </li></ul><ul><ul><li>Vector Math/Matrices (eg SVD) </li></ul></ul><ul><ul><li>Collections </li></ul></ul><ul><ul><li>Hadoop </li></ul></ul><ul><li>Algorithms </li></ul><ul><ul><li>Classification, Clustering, etc </li></ul></ul><ul><li>Your Application??? </li></ul><ul><ul><li>You can orchestrate the intelligent web!!! </li></ul></ul>
  5. 5. A New Breed of Developer <ul><li>Key Skills </li></ul><ul><ul><li>Databases </li></ul></ul><ul><ul><li>Programming </li></ul></ul><ul><ul><li>Networking </li></ul></ul><ul><ul><li>Security </li></ul></ul><ul><li>… but now also </li></ul><ul><ul><li>distributed data processing is fast becoming an essential part the developer’s toolbox. </li></ul></ul>
  6. 6. <ul><li>You never know where you will use Probability and Statistics!!!! Video snippet from Equilibrium: http://en.wikipedia.org/wiki/Equilibrium_%28film%29 </li></ul>
  7. 7. <ul><li>You never know what you will discover!!!! </li></ul>
  8. 8. Where people swear in the United States? http://flowingdata.com/2011/01/25/where-people-swear-in-the-united-states/
  9. 9. Algorithms is Apache Mahout <ul><li>Recommendation (collaborative filtering) </li></ul><ul><li>Clustering </li></ul><ul><li>Classification </li></ul><ul><li>Evolutionary Algorithms </li></ul>
  10. 10. Algorithms is Apache Mahout <ul><li>Top 10 algorithms in data mining Wu, X., Kumar, V., Ross Quinlan, J., Ghosh, J., Yang, Q., Motoda, H., et al. (2008). Top 10 algorithms in data mining. Knowledge and Information Systems, 14(1), 1-37. </li></ul><ul><li>k-Means, Apriori (fp-growth), kNN, Naive Bayes , SVM (coming) Already supported </li></ul>
  11. 11. Requirements <ul><li>Java 1.6 java -version </li></ul><ul><li>Maven 2.2 mvn -- version </li></ul><ul><li>Hadoop 0.2 </li></ul>
  12. 12. Running Mahout <ul><li>Command line launcher bin/mahout (This shows the list of algorithms) Valid program names are: </li></ul><ul><li>canopy: : Canopy clustering </li></ul><ul><li>cleansvd: : Cleanup and verification of SVD output </li></ul><ul><li>clusterdump: : Dump cluster output to text </li></ul><ul><li>dirichlet: : Dirichlet Clustering </li></ul><ul><li>fkmeans: : Fuzzy K-means clustering </li></ul><ul><li>fpg: : Frequent Pattern Growth </li></ul><ul><li>itemsimilarity: : Compute the item-item-similarities for item-based collaborative filtering </li></ul><ul><li>kmeans: : K-means clustering </li></ul><ul><li>lda: : Latent Dirchlet Allocation </li></ul><ul><li>ldatopics: : LDA Print Topics </li></ul><ul><li>lucene.vector: : Generate Vectors from a Lucene index </li></ul><ul><li>matrixmult: : Take the product of two matrices </li></ul><ul><li>meanshift: : Mean Shift clustering </li></ul><ul><li>recommenditembased: : Compute recommendations using item-based collaborative filtering </li></ul><ul><li>… .. </li></ul>
  13. 13. Running Mahout <ul><li>Run any algorithm eg kmeans locally bin/mahout kmeans –help Job-Specific Options: --input (-i) input --output (-o) output --distanceMeasure (-dm) eg SquaredEuclidean --numClusters (-k) k </li></ul>
  14. 14. Running Mahout <ul><li>Scale out Runs on cluster as per conf files in Hadoop directory </li></ul><ul><li>export HADOOP_HOME = /pathto/hadoop-0.20.2/ </li></ul><ul><li>Need to use the driver classes KMeansDriver.runjob(Path input, Path output ...) </li></ul>
  15. 15. Clustering <ul><li>Unsupervised Machine Learning technique </li></ul><ul><li>Organise items in to clusters/groups based upon similarity </li></ul><ul><li>Good for finding patterns and exploring data </li></ul>
  16. 16. Clustering <ul><li>Lots of Algorithms: k-means, Fuzzy K-means, Mean Shift, Canopy, Dirichlet Process, Latent Dirichlet Allocation </li></ul><ul><li>Similarity Distance Measures </li></ul><ul><ul><li>Euclidean </li></ul></ul><ul><ul><li>Cosine </li></ul></ul><ul><ul><li>Tanimoto </li></ul></ul><ul><ul><li>Manhattan </li></ul></ul>
  17. 17. Vectors <ul><li>Documents Bag of words word1 => 10 word2 => 2 word3 => 4 Resulting vector [10.0, 2.0, 4.0, .... ] </li></ul>
  18. 18. Range of Vectorization Tools <ul><li>Collate multiple words (n-grams) </li></ul><ul><li>Normalization </li></ul><ul><li>TF-IDF </li></ul><ul><li>Stop word removal </li></ul>
  19. 19. kmeans Example <ul><li>Set of text files in a directory </li></ul><ul><li>Use seqdirectory to convert files to vectors bin/mahout seqdirectory -i <input> -o <seq-output> </li></ul><ul><li>Use seq2sparse to convert to sparse vector bin/mahout seq2sparse -i seq-output -o <vector-output> </li></ul><ul><li>Run kmeans with k=5 bin/mahout kmeans -i<vector-output> -c <cluster-temp> -o <cluster-output> -k 5 </li></ul><ul><li>View output bin/mahout clusterdump </li></ul>
  20. 20. Easy enough, but <ul><li>How do you know k? </li></ul><ul><li>Data Exploration is required to find the </li></ul><ul><ul><li>k for your purposes </li></ul></ul><ul><ul><li>Similarity distance for your purpose </li></ul></ul><ul><li>Role for the Data Scientist </li></ul><ul><ul><li>Explore, Model, Test and Evaluate </li></ul></ul>
  21. 21. Recommender Engines <ul><li>Encounter the most </li></ul><ul><li>Recommend products (books, movies, etc) based upon past actions </li></ul><ul><li>Infer tastes and preferences to identify unknown items of interest </li></ul>
  22. 22. Recomendation <ul><li>Algorithms: user and item recommendation </li></ul><ul><li>Framework for storage, online and offline computation </li></ul><ul><li>Similarity Measures </li></ul><ul><ul><li>Cosine </li></ul></ul><ul><ul><li>Tanimoto </li></ul></ul><ul><ul><li>Pearson </li></ul></ul>
  23. 23. Frequent Pattern Mining <ul><li>Discover interesting patterns based upon how items occur in a sequence </li></ul><ul><li>Example Sales Transactions (Bread, Milk and Eggs) (Nappies, Beer) </li></ul><ul><li>Parallel FPGrowth Algorithm </li></ul>
  24. 24. Classification <ul><li>Set of classes/categories (observed pattern) </li></ul><ul><li>Decide if a new input matches a category </li></ul><ul><li>Supervised technique – need training </li></ul><ul><li>Eg spam or not </li></ul>
  25. 25. Classification <ul><li>Algorithms: Naive Bayes, Random Forest Decision Tree, SVM coming </li></ul><ul><li>Learn a model from a manually trained dataset </li></ul><ul><li>Predict the class of an unseen object based on features </li></ul>
  26. 26. Latent Dirichlet Allocation <ul><ul><li>Convert text to term-document matrix </li></ul></ul><ul><ul><li>LDA produces </li></ul></ul><ul><ul><ul><li>word-theme mapping </li></ul></ul></ul><ul><ul><ul><li>theme-document mapping </li></ul></ul></ul><ul><ul><ul><li>Allows topic overlap </li></ul></ul></ul><ul><ul><li>Need to specify number of Topics (k) </li></ul></ul>
  27. 27. Latent Dirichlet Allocation <ul><li>LDA </li></ul><ul><ul><li>Tweet 1 </li></ul></ul><ul><ul><li>Tweet 2 </li></ul></ul><ul><ul><li>Tweet 3 </li></ul></ul>Term-Document Matrix Specify No Themes (k) Topic to Word Mapping X Tweet to Topic Mapping Word 1 Word 2 Word n Doc 1 1 0 2 Doc 2 0 1 0 Doc 3 0 1 1 Word 1 Word 2 Word n Topic 1 0.5 0 1 Topic 2 0 0.5 0 Topic 1 Topic 2 Doc 1 1 0 Doc 2 0 1 Doc 3 0 1
  28. 28. Latent Dirichlet Allocation <ul><ul><li>Run LDA bin/mahout  lda  -input <PATH>  ‐output <PATH>  –numTopics  20 </li></ul></ul><ul><ul><li>View Topics bin/mahout  LDAPrintTopics  ‐input  <PATH> ‐output  <PATH> ‐dictonaryType  sequencefile </li></ul></ul>
  29. 29. Suggesting Twitter Lists <ul><ul><li>Twitter introduced Lists group people you follow so you can see only their timeline of tweets </li></ul></ul><ul><ul><li>Build an application that could recommend people that should be grouped in the same list. </li></ul></ul><ul><ul><li>LDA because it will allow for overlapping list membership - this is great because people talk about multiple topics. </li></ul></ul>
  30. 30. Suggesting Twitter Lists <ul><ul><li>Twitter API Tasks </li></ul></ul><ul><ul><ul><li>Get list of people that a user follows </li></ul></ul></ul><ul><ul><ul><li>Retrieve tweets for each person </li></ul></ul></ul><ul><ul><ul><li>Save Lists back to Twitter </li></ul></ul></ul><ul><ul><li>Data Processing </li></ul></ul><ul><ul><ul><li>Combine all tweets for a person </li></ul></ul></ul><ul><ul><ul><li>Remove stop words </li></ul></ul></ul><ul><ul><ul><li>Stem words </li></ul></ul></ul><ul><ul><ul><li>Create a user-word matrix </li></ul></ul></ul>
  31. 31. Suggesting Twitter Lists <ul><ul><li>Web UI </li></ul></ul><ul><ul><ul><li>Authenticate to Twitter </li></ul></ul></ul><ul><ul><ul><li>Display suggested lists (based on estimate of k) (Could also display the important tweets that place the person in the group?) </li></ul></ul></ul><ul><ul><ul><li>Allow users to change k ie decide on the number of Lists </li></ul></ul></ul><ul><ul><ul><li>Allow group re-organisation with jquery sortables </li></ul></ul></ul>
  32. 32. Gently Getting into Machine Learning and Data Mining <ul><li>Programming Collective Intelligence by Toby Segaram </li></ul><ul><li>Mahout in Action by Owen, Anil, Dunning and Friedman </li></ul>
  33. 33. Summary <ul><li>Mahout offers good abstraction for building intelligent web applications </li></ul><ul><li>Skills in data analysis and exploration are now more important than ever </li></ul><ul><li>Mahout is a good platform for distributed algorithm development </li></ul>
  34. 34. Fascinating Algorithms <ul><li>My Top 3 algorithms </li></ul><ul><ul><li>Some interesting and some disturbing and interesting at the same time </li></ul></ul>
  35. 35. Fascinating Algorithms <ul><li>No 3 – Identifying Manipulated Images http://www.technologyreview.com/computing/20423/page1/ </li></ul>
  36. 36. Fascinating Algorithms <ul><li>No 2 – Seam Carving Content Aware Resizing Example http://swieskowski.net/carve/ </li></ul>
  37. 37. Disturbing Algorithms <ul><li>No 1 – Digital Face Beautification http://leyvand.com/research/beautification/dfb_sketch.pdf </li></ul>
  38. 38. Disturbing Algorithms <ul><li>No 1 – Digital Face Beautification http://leyvand.com/research/beautification/dfb_sketch.pdf </li></ul>
  39. 39. Disturbing Algorithms <ul><li>No 1 – Digital Face Beautification http://leyvand.com/research/beautification/dfb_sketch.pdf </li></ul>Image from Shrek Copyright Dreamworks
  40. 40. Discussion/Questions <ul><li>What will you build? </li></ul>