Developed by Chao Shi, Sam O'Mullane, Sean Kickham, Reza Rad and Andrew Rubino Watch the project presentation: https://youtu.be/gkKGnnBenyk This project was completed by students from NYC Data Science Academy's 12-Week Bootcamp. Learn more about the bootcamp: http://nycdatascience.com/data-science-bootcamp/ People make decisions on where to eat based on friends’ recommendations. Since they know you, their suggestions matter more than those of strangers. For the capstone project, we built a hybrid Yelp recommendation system that can provide individualized recommendations based on your friend’s reviews on the social network. We built the machine learning models using Spark, and set up a Flask-Kafka-RDS-Databricks pipeline that allows a continuous stream of user requests. During the presentation, we will talk about the development framework and technical implementation of the pipeline. Read on their project posts and code: https://blog.nycdatascience.com/student-works/capstone/yelp-recommender-part-1/ https://blog.nycdatascience.com/student-works/yelp-recommender-part-2/

1. A Hybrid Recommender with
Yelp Challenge Data
S. Kickham, A. Rubino, R. Rad, S. O’Mullane, C. Shi

2. ● Intro
● Data Preparation & Exploratory Analysis
● Hybrid Recommendation System
● Data Pipeline
● Demo
● Experience & Future Work
Outline
Background
EDA
Recommendation
Algorithms
App Pipeline
Demo

3. Motivation
Business Need
Big Data
Machine
Learning
Yelper
Helper

4. Project Architecture

5. Data Source
5 Gb text data
json format

6. Amazon RDS - MySQL Database
● RDS = Relational Database Service
● Main tables include:
○ Reviews - 4.1M
○ Tips - 947K
○ Businesses - 144K
○ Checkins - 125K
○ Users - 1M
● The main tables allow for simple subtable creation, which allows for fast
requests and data loads.

7. MySQL Schema Viz
● Primary Keys for all tables:
○ Businesses: business_id
○ Users: user_id
○ Reviews: review_id
○ Tips: user_id
○ Checkins: business_id
● RDS allowed us to create
relationships between tables.

8. Word Frequency per Review Type

9. Sentiment Analysis
Pre-
Processing
Neural
Network
Sentiment
bit.ly/sent_analysis
1. Binarize
2. Tokenize
3. Texts_to_Seq
1. Supervised
2. Convolution
3. Pooling
1. 94% Accuracy
2. Spot Check
3. Base Rec.

10. Convolution and Pooling

11. Content-Based
Recommendations
● Item-to-item based on user profile
● Keyword soft filtering
● Average Word2Vec
http://i65.tinypic.com/2ebah6c.png

12. Cosine Similarity using Word2Vec
Pre-
Processing
Neural
Network
Similarity
bit.ly/word_2_vec
1. Tokenize
2. Stop Word
Removal
1. Unsupervised
2. Skip-gram
3. Hyperparameters
1. Cosine
Similarity
2. Rank

13. Word Similarity

14. Word Similarity
https://adriancolyer.files.wordpress.com/2016/04/word2vec-king-queen-vectors.png?w=600

15. Word Algebra
https://adriancolyer.files.wordpress.com/2016/04/word2vec-king-queen-composition.png?w=600

16. Word Algebra: Food
https://adriancolyer.files.wordpress.com/2016/04/word2vec-king-queen-composition.png?w=600

17. Word Algebra: Food

18. Word Algebra: Examples
'filet_mignon' - 'beef' + 'seafood' =
'fork' + 'soup' =
'drink' + 'barley' =

19. Word Algebra: Examples
'filet_mignon' - 'beef' + 'seafood' = lobster tail
'fork' + 'soup' = spoon
'drink' + 'barley' = beer

20. NLP Summary
Sentiment
Analysis
Cosine
Similarity
Base Rec. Content Rec.

21. Collaborative Filtering
https://en.wikipedia.org/wiki/Collaborative_filtering

22. Collaborative Filtering
Matrix
Factorization
Cost Function

23. Yelp Social Network

24. Review-level adjustments
How relevant is a 10 year old review?
Newest 60% of reviews unaffected by time-weighting
Reviews binned by years from present

25. Business-level adjustments
Log
Log transformations for popularity measures
Check Ins
Reviews
Multiplicative factor based
on NLP sentiment

26. Stars Percentile
99th (top 40)
95th (top 300)
83rd
60th
31st
5th
Final Location-only Ratings

27. Recommendation Engine
Radial Filter:
Haversine Distance
Location
Latitude
Longitude
User Info
User ID
User Interest
Keyword
User Requests
Required
Optional
Optional
Optional
Dynamic generation of up to 40 recommendations within 20s
Social Network Data Mining

28. Data Pipeline

29. Data Pipeline: 1. Flask
User ID
Keyword
Longitude
Latitude
Search Radius

30. Components:
1. Brokers
2. Topics
3. Producers
4. Consumers
Data Pipeline: 2. Kafka

31. Data Pipeline: 2. Kafka
Consumer
Producer Consumer
Producer
Topic 1
Topic 2
Kafka Server (Running on Zookeeper)

32. Data Pipeline: 3. Spark
Spark’s Structured Streaming

33. Data Pipeline: Overall

34. Demo

35. NYC Data Science Academy
Thank You