Sparking Science up with Research Recommendations by Maya Hristakeva

1. Sparking Science up with
Research Recommendations
Maya Hristakeva
@mayahhf

2. Overview
• What is Mendeley Suggest?
• Computation Layer
• Conclusions

3. Read
&
Organize
Search
&
Discover
Collaborate
&
Network
Experiment
&
Synthesize
Mendeley builds tools to help researchers …

4. Being the best researcher you can be!
• Good researchers are on top of their game
• Large amount of research produced
• Takes time to get what you need
• Help researchers by recommending relevant research

5. Mendeley Suggest
Personalized Article
Recommender

6. Recommender System Components
information flow (components often built in parallel)
Data
(Feature
Engineering)
Algorithms Business Logic
and Analytics
User Experience

7. Mendeley Suggest Components (Past)
information flow (components often built in parallel)
Data
(Feature
Engineering)
Algorithms Business Logic
and Analytics
User Experience

8. Mendeley Suggest Components (Present)
information flow (components often built in parallel)
Data
(Feature
Engineering)
Algorithms Business Logic
and Analytics
User Experience

9. Mendeley Suggest Components (Goal)
information flow (components often built in parallel)
Data
(Feature
Engineering)
Algorithms Business Logic
and Analytics
User Experience

10. Overview
• What is Mendeley Suggest?
• Computation Layer
– Algorithms
– Evaluation
– Implementations & Performance
• Conclusions

11. Personalized Article Recommendations
Input:
User libraries
Output:
Suggested
articles to read
Algorithms:
• Collaborative Filtering
– Item-based
– User-Based
– Matrix Factorization
• Content-based

12. Item-based Collaborative Filtering
Recommend articles that are similar to the ones you read
– Similarity is based on article co-occurrences in users’ libraries
– “Users who read x also read y”

13. User-based Collaborative Filtering
Find users who have similar appreciation for articles as you
– Similarity is based on users’ libraries overlap
Recommend new articles based on what the users similar to
you read
– “Users similar to you (based on a, b, c) also read x”

14. Matrix Factorization CF
2 4 5
5 4 1
5 ? 2
1 5 4
4 2
4 5 1
U
n x k
V
k x m
fij= <Ui*,V*j>
E(U,V) = L(Xij, fij) + R(U,V)
X
n x m

15. Overview
• What is Mendeley Suggest?
• Computation Layer
– Algorithms
– Evaluation
– Implementations
• Conclusions

16. Performance
Costly & GoodCostly & Bad
Cheap & GoodCheap & Bad

17. Costly & GoodCostly & Bad
Cheap & GoodCheap & Bad
Performance

18. Costly & GoodCostly & Bad
Cheap & GoodCheap & Bad
Performance

19. How to measure quality?
• Offline Evaluation
– Parameter sweep is quick
– Don’t offend real users
• Methodology
– n-fold cross-validation
– time-based validation
• Metrics
– precision, recall and f-measure
– AUC (area under roc curve), NDCG (normalized discounted cumulative gain)

20. Overview
• What is Mendeley Suggest?
• Computation Layer
– Algorithms
– Evaluation
– Implementations
• Conclusions

21. Implementations
Mahout
(Hadoop)
Mendeley
(Hadoop)
Mahout
(Spark)
Mendeley
(Spark)
MLlib
(Spark)
Item-based CF
User-based CF
Matrix
Factorization

22. Setup
• EMR Cluster
– Master: 1 x r3.xlarge instance (4 core, 32GB)
– Core: 10 x r3.2xlarge instances (8 core, 64GB)
• Data: user libraries
– 15mil documents >>> 1mil users
– 150mil interactions
• Offline Evaluation
– Methodology: time-based evaluation
– Metric: precision@10

23. Implementations
Mahout
(Hadoop)
Mendeley
(Hadoop)
Mahout
(Spark)
Mendeley
(Spark)
MLlib
(Spark)
Item-based CF
User-based CF
Matrix
Factorization

24. Apache Mahout
• Mahout (out-of-the-box)
– Item-based CF
• org.apache.mahout.cf.taste.hadoop.item.RecommenderJob
– ALS Matrix Factorization
• org.apache.mahout.cf.taste.hadoop.als.ParallelALSFactorizationJob
• org.apache.mahout.cf.taste.hadoop.als.RecommenderJob
• Implemented User-based CF on top of Mahout at Mendeley

25. Costly & GoodCostly & Bad
Cheap & GoodCheap & Bad
Orig. item-based
mahout
Tuned item-based
mahout
-0.5K
(-60%
)
Performance
~$125

26. Costly & GoodCostly & Bad
Cheap & GoodCheap & Bad
Orig. item-based
mahout
Tuned item-based
mahout
-0.5K
(-60%
)
Orig. user-based
mahout
Tuned user-based
mahout
-0.1K
(-40%
)
Performance
~$125

27. Costly & GoodCostly & Bad
Cheap & GoodCheap & Bad
Orig. item-based
mahout
Tuned item-based
mahout Orig. user-based
mahout
Tuned user-based
mahout
+150%
-0.2K
(-55%
)
-0.7K
(-82%
)
Performance
~$125

28. Mahout Performance
• Mahout’s recommender is already efficient
– But your data may have unusual properties
• We’ve got improvements by
– Tuning Hadoop’s mapper and reducer allocation over the Recommender Job steps
– Using an appropriate partitioner
• Improve quality
– Mahout provides Item-based CF
– We have many more items than users
– Typically, user-based is more appropriate

29. Implementations
Mahout
(Hadoop)
Mendeley
(Hadoop)
Mahout
(Spark)
Mendeley
(Spark)
MLlib
(Spark)
Item-based CF
User-based CF
Matrix
Factorization

30. Mahout Spark
• Co-occurrence Recommenders with Spark
– Item-Item similarity
• mahout spark-itemsimilarity
SimilarityAnalysis.cooccurrencesIDSs(ratings, …)
– User-User similarity
• mahout spark-rowsimilarity
SimilarityAnalysis.rowSimilarityIDSs(ratings, …)
• Only supports Boolean data and log-likelihood similarity
• Does not generate actual recommendations

31. Mahout Spark
• Could not get to run successfully on our data
• Got further by tuning parameters but still failed with OOM
– spark.driver.maxResultSize
– spark.kryoserializer.buffer.max
– spark.default.parallelism
– spark.storage.memoryFraction
• Gave best runtime performance on MovieLens datasets

32. Implementations
Mahout
(Hadoop)
Mendeley
(Hadoop)
Mahout
(Spark)
Mendeley
(Spark)
MLlib
(Spark)
Item-based CF
User-based CF
Matrix
Factorization

33. Mendeley Spark
• Started as hack-day project
– Implement Item-based and User-based CF in Spark
• Can be implemented in two steps
1. Compute Item-Item or User-User Similarities
• given user preferences
2. Compute Recommendations
• given similarities and user preferences

34. Spark: Item-Item Similarity

35. Spark: Item-Item Similarity

36. Spark: Item-Item Similarity

37. Spark: Item-Item Similarity

38. Spark: Item-Based Recs

39. Spark: Item-Based Recs

40. Costly & GoodCostly & Bad
Cheap & GoodCheap & Bad
Tuned IB Mahout
Tuned UB Mahout
Orig. UB Spark
Performance
~$50

41. Costly & GoodCostly & Bad
Cheap & GoodCheap & Bad
Tuned IB Mahout
Tuned UB Mahout
Orig. UB Spark
Tuned UB Spark
Tuned IB Spark
-0.1K
(-40%
)
Performance
~$50

42. Mendeley Spark Performance
• Spark implementation of User-based CF performs well
• Managed to run variation of Item-based CF
– Uses fewer items per user to recommend similar items to
– Quality not impacted much
• We’ve got improvements by tuning
– Resource allocation
– Parallelism
– http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-
2/

43. Implementations
Mahout
(Hadoop)
Mendeley
(Hadoop)
Mahout
(Spark)
Mendeley
(Spark)
MLlib
(Spark)
Item-based CF
User-based CF
Matrix
Factorization

44. Spark MLlib DimSum
• DimSum: efficient algorithm for computing all-pairs similarity
– “Dimension Independent Matrix Square using MapReduce”
– Contributed by Twitter
• Replace similarity computation with DimSum
– Only supports cosine similarity
• Does not generate actual recommendations
– Compute recommendations as before

45. MLlib DimSum Item-Item Similarity

46. MLlib DimSum User-User Similarity

47. Costly & GoodCostly & Bad
Cheap & GoodCheap & Bad
Tuned IB Mahout
Tuned UB Mahout
Tuned UB Spark
Tuned IB Spark
UB DimSum
Spark MLlib
Performance
~$50

48. Spark MLlib Matrix Factorization
Implements alternating least squares (ALS)
1. Compute Model
2. Compute Recommendations

49. Costly & GoodCostly & Bad
Cheap & GoodCheap & Bad
Tuned IB Mahout
Tuned UB Mahout
Tuned UB Spark
Tuned IB Spark
UB DimSum
Spark MLlib
ALS Matrix Fact.
Spark MLlib
-50%
Performance
~$50

50. MLlib Performance
• Provides good alternative for computing user-user similarities
– Due to data sparsity, not getting big gains in runtime
– Only supports cosine similarity
• Failed to compute item-item similarities
– Exceeds maximum allowed value of 2G for spark.kryoserializer.buffer.max
• User-based CF outperforms ALS CF
• Need scalable solution for generating recommendations based on ALS
CF model

51. Implementations
Mahout
(Hadoop)
Mendeley
(Hadoop)
Mahout
(Spark)
Mendeley
(Spark)
MLlib
(Spark)
Item-based CF
User-based CF
Matrix
Factorization

52. Overview
• What is Mendeley Suggest?
• Computation Layer
• Conclusions

53. Costly & GoodCostly & Bad
Cheap & GoodCheap & Bad
Tuned IB Mahout
Tuned UB Mahout
Tuned UB Spark
Tuned IB Spark
UB DimSum
Spark MLlib
ALS Matrix Fact.
Spark MLlib
Performance
+100%
+150%
~$50

54. Mendeley Suggest Components (Future)
information flow (components often built in parallel)
Data
(Feature
Engineering)
Algorithms Business Logic
and Analytics
User Experience

55. Conclusions
• Mendeley Suggest is a personalized article recommender
• Spark is good alternative to Mahout as computation layer
– Needs some love and tuning
– Much fewer lines of code – easier to maintain and extend
• User-based can outperform item-based and matrix factorization
• Save resources and money by understanding your data
• Test offline before deploying
– but also need online tests to get real performance

56. Thank you!
mendeley.com/suggest