Presentation slides for KDD15'

1. Aug 11st, 2015
Xu Miao, Lijun Tang, Yitong Zhou, Joel Young LinkedIn
Chun-te Chu, Microsoft
Anmol Bhasin Groupon
Distributed Personalization

2. Motivation
Distributed Learning
Personalization
Experiments

3. Recommendation

4. Recommendation

5. Recommendation

6. Common Solution
Apps Tracking ETL
DM
Delivering

7. Common Solution -- Cold Start
Apps Tracking ETL
DM
Delivering
minutes
hours
days
Apps
seconds
seconds
seconds

8. Common Solution -- Warm Start
Apps Tracking ETL
DM
Delivering
minutes
hours
days
seconds
seconds
seconds

9. seconds
seconds
seconds
Bring ML Closer to Users
Apps Tracking ETL
DM
Delivering
minutes
hours
days

10. Distributed Online Learning
▪ Definition:
– Agent presents an example
– User responses with a reward r
– Agent updates the model w

11. Distributed Online Learning
▪ Definition:
– Agent presents an example
– User responses with a reward r
– Agent updates the model w
▪ Challenges:
– Users’ feedback data too few
▪ Distributed Learning

12. Distributed Online Learning
▪ Definition:
– Agent presents an example
– User responses with a reward r
– Agent updates the models
▪ Challenges:
– Users’ feedback data too few
▪ Distributed Learning
– Everyone has different preferences
▪ Personalization

13. Motivation
Distributed Learning
Personalization
Experiments

14. ▪ Bulk Synchronous Parallel (Hadoop & Spark)
– ~ Thousands of interactions to converge
Distributed Gradient Descent

15. ▪ Stale Synchronous Parallel [Ho and etc. 13’]
– For some users, staleness is forever
Distributed Gradient Descent
What did I do?

16. ▪ Blessing
– It is one of the key reasons for PGDs to converge
fast
▪ Challenge
– It goes diminished, and the data comes later has
smaller and smaller impact
– Restart? Residue constant? Hard to manage
Learning Rate

17. Alternating Direction Method of Multipliers (ADMMs)

18. ADMMs -- Bulk Synchronous Parallel

19. ADMMs -- Bulk Synchronous Parallel

20. ADMMs -- Asynchronous Parallel
[Miao, Chu, Tang, Zhou, Young, Bhasin 15’]
timelines
V1
V1’
V1’’
t0
t1
t2

21. ADMMs -- Asynchronous Parallel
[Miao, Chu, Tang, Zhou, Young, Bhasin 15’]
timelines
V1
V1’
V1’’
t0
t1
t2
t3
t4
V2

22. V1’’
ADMMs -- Asynchronous Parallel
[Miao, Chu, Tang, Zhou, Young, Bhasin 15’]
Weighted Merge
1
1
timelines
V1
V1’
t0
t1
t2
V2 V3
t3
t4

23. ADMMs -- Asynchronous Parallel
[Miao, Chu, Tang, Zhou, Young, Bhasin 15’]
Master Versions
timelines

24. ADMMs -- Asynchronous Parallel
[Miao, Chu, Tang, Zhou, Young, Bhasin 15’]
▪ Same convergence rate as Bulk Synchronous Parallel
▪ No learning rate
– Out-of-order sequences of mini-optimizations
– Continuous Learning

25. Motivation
Distributed Learning
Personalization
Experiments

26. Personalized Models

27. Personalized Models

28. Personalized Models
▪ The personalization strength:
– Allow divergence of personal models from the
consensus model
– Improve relevance
– Improve convergence (speed)

29. Motivation
Distributed Learning
Personalization
Experiments

30. Facial Expression Recognition

31. Facial Expression Recognition

32. Facial Expression Recognition

33. Job Recommendation

34. Job Recommendation

35. Speed

36. Conclusion
▪ Asynchronous ADMMs
– Continuous learning
▪ Personalized Models
– Fits users better
– Improves convergence speed

37. Thank You and Questions

38. ADMMs -- Asynchronous Parallel
▪ Delay variations
– Weighted Merge (v.s. Stale Synchronous Parallel)
– Flexible to handle non-stationary distribution
▪ Crazy active users
▪ Passive important users
▪ Spammers