Bayes Law Bayesian Network Latent Dirichlet Allocation References

1. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Topic Modeling
Latent Dirichlet Allocation
Kyunghoon Kim
kyunghoon@unist.ac.kr
http://www.math.unist.ac.kr/~kyunghoon
Mathematical Sciences – CMS
Ulsan National Institude of Science and Technology
2016-1 Graduate Students Pitching
28th May 2016
Kyunghoon Kim Graduate Students Pitching Topic Modeling 1 / 37

2. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Motivation
Figure: from Seyeon Lee
Latent Dirichlet Allocation [1]
Kyunghoon Kim Graduate Students Pitching Topic Modeling 2 / 37

3. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Outline
Bayes Law
Bayesian Network
Latent Dirichlet Allocation
Kyunghoon Kim Graduate Students Pitching Topic Modeling 3 / 37

4. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Bayes Law
P(A) (1)
Kyunghoon Kim Graduate Students Pitching Topic Modeling 4 / 37

5. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Bayes Law
P(A) (1)
P(A|B) =
P(A ∩ B)
P(B)
=
P(A, B)
P(B)
=
P(B|A)P(A)
P(B)
(2)
Kyunghoon Kim Graduate Students Pitching Topic Modeling 4 / 37

6. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Bayes Law
P(A) (1)
P(A|B) =
P(A ∩ B)
P(B)
=
P(A, B)
P(B)
=
P(B|A)P(A)
P(B)
(2)
P(A) =
m
i=1
P[A|Bi ]P[Bi ] (3)
Kyunghoon Kim Graduate Students Pitching Topic Modeling 4 / 37

7. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayes Law
Testing for a rare disease, where 1% of the population is infected.
We have a highly sensitive and specific test, which is not quite
perfect:[2]
• 99% of sick patients test positive.
• 99% of healthy patients test negative.
Given that a patient tests positive, what is the probability that the
patient is actually sick?
Kyunghoon Kim Graduate Students Pitching Topic Modeling 5 / 37

8. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayes Law
Figure: Tree diagram
Kyunghoon Kim Graduate Students Pitching Topic Modeling 6 / 37

9. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayes Law
If you test positive, you’re equally likely to be healthy or sick.
p(sick|+) =
p(+|sick)p(sick)
p(+)
=
0.99 × 0.01
0.99 × 0.01 + 0.01 × 0.99
= 0.50
Kyunghoon Kim Graduate Students Pitching Topic Modeling 7 / 37

10. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayes Law
If you test positive, you’re equally likely to be healthy or sick.
p(sick|+) =
p(+|sick)p(sick)
p(+)
=
0.99 × 0.01
0.99 × 0.01 + 0.01 × 0.99
= 0.50
Law of Total Probability
P(A) =
m
i=1
P[A|Bi ]P[Bi ] (4)
Kyunghoon Kim Graduate Students Pitching Topic Modeling 7 / 37

11. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Bayes Law
P(H|D) =
P(D|H)P(H)
P(D)
(5)
posterior prob =
likelihood ∗ prior prob
evidence
Kyunghoon Kim Graduate Students Pitching Topic Modeling 8 / 37

12. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayes Law - Red Spot
A patient with the red spot face comes to a doctor.[3]
• Chickenpox (Sudoo)
• Smallpox (Chunyeondoo)
• P(RedSpot|Sudoo) = 0.8
• P(RedSpot|Chunyeondoo) = 0.9
P(Sudoo|RedSpot) =
P(RedSpot|Sudoo)P(Sudoo)
P(RedSpot)
(6)
Kyunghoon Kim Graduate Students Pitching Topic Modeling 9 / 37

13. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayes Law - Red Spot
P(RedSpot) = P(RedSpot, Sudoo) + P(RedSpot, ∼ Sudoo)
= P(RedSpot|Sudoo)P(Sudoo)
+ P(RedSpot| ∼ Sudoo)P(∼ Sudoo)
Kyunghoon Kim Graduate Students Pitching Topic Modeling 10 / 37

14. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayes Law - Junho or Chuno
H1 = Junho, H2 = Chuno, D = Voice
P(Junho|Voice) =
P(Voice|Junho)P(Junho)
P(Voice)
P(Chuno|Voice) =
P(Voice|Chuno)P(Chuno)
P(Voice)
• P(Voice|Junho) = 0.9
• P(Voice|Chuno) = 0.8
Kyunghoon Kim Graduate Students Pitching Topic Modeling 11 / 37

15. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayes Law - Junho or Chuno
H1 = Junho, H2 = Chuno, D = Voice
P(Junho|Voice) =
P(Voice|Junho)P(Junho)
P(Voice)
P(Chuno|Voice) =
P(Voice|Chuno)P(Chuno)
P(Voice)
• P(Voice|Junho) = 0.9
• P(Voice|Chuno) = 0.8
• P(Junho) = 0.99
• P(Chuno) = 0.01
Kyunghoon Kim Graduate Students Pitching Topic Modeling 11 / 37

16. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Bayesian Network
A Bayesian network is a probabilistic graphical model that
represents a set of random variables and their conditional
dependencies via a directed acyclic graph.
Kyunghoon Kim Graduate Students Pitching Topic Modeling 12 / 37

17. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayesian Network - Wet Grass or Wet Road
Kyunghoon Kim Graduate Students Pitching Topic Modeling 13 / 37

18. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayesian Network - Wet Grass or Wet Road
Kyunghoon Kim Graduate Students Pitching Topic Modeling 14 / 37
Summer P(Summer)
T 0.3
F 0.7

19. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayesian Network - Wet Grass or Wet Road
Kyunghoon Kim Graduate Students Pitching Topic Modeling 15 / 37
Summer P(Summer)
T 0.3
F 0.7
Summer Rain P(Rain Summer)
T T 0.8
T F 0.2
F T 0.1
F F 0.9

20. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayesian Network - Wet Grass or Wet Road
Kyunghoon Kim Graduate Students Pitching Topic Modeling 16 / 37
Summer P(Summer)
T 0.3
F 0.7
Summer Rain P(Rain Summer)
T T 0.8
T F 0.2
F T 0.1
F F 0.9
Rain WetRoad P(WetRoad Rain)
T T 0.7
T F 0.3
F T 0.0
F F 1.0

21. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayesian Network - Wet Grass or Wet Road
Kyunghoon Kim Graduate Students Pitching Topic Modeling 17 / 37
Summer P(Summer)
T 0.3
F 0.7
Summer Rain P(Rain Summer)
T T 0.8
T F 0.2
F T 0.1
F F 0.9
Rain WetRoad P(WetRoad Rain)
T T 0.7
T F 0.3
F T 0.0
F F 1.0
Summer SpringCooler P(SpringCooler Summer)
T T 0.1
T F 0.9
F T 0.6
F F 0.4

22. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayesian Network - Wet Grass or Wet Road
Kyunghoon Kim Graduate Students Pitching Topic Modeling 18 / 37
Summer P(Summer)
T 0.3
F 0.7
Summer Rain P(Rain Summer)
T T 0.8
T F 0.2
F T 0.1
F F 0.9
Rain WetRoad P(WetRoad Rain)
T T 0.7
T F 0.3
F T 0.0
F F 1.0
Summer SpringCooler P(SpringCooler Summer)
T T 0.1
T F 0.9
F T 0.6
F F 0.4
SCooler Rain WetGrass P(WetGrass SCooler,Rain)
T T T 0.9
T T F 0.1
T F T 0.8
T F F 0.2
F T T 0.7
F T F 0.3
F F T 0.0
F F F 1.0

23. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayesian Network - Wet Grass or Wet Road
What is the probability of Summer, not SpringCooler, Rain,
not Wet Grass, Wet Road?
Kyunghoon Kim Graduate Students Pitching Topic Modeling 19 / 37

24. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayesian Network - Wet Grass or Wet Road
What is the probability of Summer, not SpringCooler, Rain,
not Wet Grass, Wet Road?
P(S, ∼ C, R, ∼ G, P) = P(∼ G|S, ∼ C, R, P)P(S, ∼ C, R, P)
Kyunghoon Kim Graduate Students Pitching Topic Modeling 19 / 37

25. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayesian Network - Wet Grass or Wet Road
What is the probability of Summer, not SpringCooler, Rain,
not Wet Grass, Wet Road?
P(S, ∼ C, R, ∼ G, P) = P(∼ G|S, ∼ C, R, P)P(S, ∼ C, R, P)
P(∼ G| ∼ C, R) by Markovian assumption
Kyunghoon Kim Graduate Students Pitching Topic Modeling 19 / 37

26. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayesian Network - Wet Grass or Wet Road
What is the probability of Summer, not SpringCooler, Rain,
not Wet Grass, Wet Road?
P(S, ∼ C, R, ∼ G, P) = P(∼ G|S, ∼ C, R, P)P(S, ∼ C, R, P)
P(∼ G| ∼ C, R) by Markovian assumption
= P(∼ G| ∼ C, R)P(P|R)P(∼ C|S)P(R|S)P(S)
= 0.3 × 0.7 × 0.9 × 0.8 × 0.3
= 0.04526
Kyunghoon Kim Graduate Students Pitching Topic Modeling 19 / 37

27. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayesian Network - Winter and Snow
Kyunghoon Kim Graduate Students Pitching Topic Modeling 20 / 37

28. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayesian Network - Winter and Snow
P(s, w) = P(s, w, z)+P(s, w, ∼ z)+P(s, ∼ w, z)+P(s, ∼ w,
= P(s|z)P(z|w)P(w) + P(s| ∼ z)P(∼ z|w)P(w)
+P(s|z)P(z| ∼ w)P(∼ w)+P(s, ∼ z)P(∼ z| ∼ w)P(∼ w)
Kyunghoon Kim Graduate Students Pitching Topic Modeling 20 / 37

29. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayesian Network - Winter and Snow
P(s, w) = P(s, w, z)+P(s, w, ∼ z)+P(s, ∼ w, z)+P(s, ∼ w,
= P(s|z)P(z|w)P(w) + P(s| ∼ z)P(∼ z|w)P(w)
+P(s|z)P(z| ∼ w)P(∼ w)+P(s, ∼ z)P(∼ z| ∼ w)P(∼ w)
P(s|w) =
P(s, w)
P(w)
Kyunghoon Kim Graduate Students Pitching Topic Modeling 20 / 37

30. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for Bayesian Network - Winter and Snow
P(s, w) = P(s, w, z)+P(s, w, ∼ z)+P(s, ∼ w, z)+P(s, ∼ w,
= P(s|z)P(z|w)P(w) + P(s| ∼ z)P(∼ z|w)P(w)
+P(s|z)P(z| ∼ w)P(∼ w)+P(s, ∼ z)P(∼ z| ∼ w)P(∼ w)
P(s|w) =
P(s, w)
P(w)
P(w|s) =
P(w, s)
P(s)
Kyunghoon Kim Graduate Students Pitching Topic Modeling 20 / 37

31. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Graphical model representations
Plate notation is a method of representing variables that repeat
in a graphical model.
Kyunghoon Kim Graduate Students Pitching Topic Modeling 21 / 37

32. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Graphical model representations
Plate notation is a method of representing variables that repeat
in a graphical model.
Kyunghoon Kim Graduate Students Pitching Topic Modeling 21 / 37

33. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Notations for LDA
• A word is the basic unit of discrete data {1, · · · , V }.
• A document is a sequence of N words w = (w1, w2, · · · , wN),
where wn is the nth word in the sequence
• A corpus is a collection of M documents
D = {w1, w2, · · · , wM}
Kyunghoon Kim Graduate Students Pitching Topic Modeling 22 / 37

34. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for LDA
Bag of words
{Apple, Banana, Car, Driver, Engine, Fourier, Green}
Apple = [1, 0, 0, 0, 0, 0, 0]T
Banana = [0, 1, 0, 0, 0, 0, 0]T
Kyunghoon Kim Graduate Students Pitching Topic Modeling 23 / 37

35. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for LDA
Bag of words
{Apple, Banana, Car, Driver, Engine, Fourier, Green}
Apple = [1, 0, 0, 0, 0, 0, 0]T
Banana = [0, 1, 0, 0, 0, 0, 0]T
Word probability matrix β is a k × V -dimensional matrix.
Apple Banana Car Driver Engine Fourier Green
Topic1 0 0 0.5 0.4 0.3 0 0.1
Topic2 0.4 0.3 0 0.1 0 0 0.3
Topic3 0 0 0 0 0.1 0.4 0.5
Topics are distribution over fixed vocaburary.
Kyunghoon Kim Graduate Students Pitching Topic Modeling 23 / 37

36. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Example for LDA
Topic assignment variable zm is a k-dimensional multinomial
random vector and word-specific variable.
zm = [0, 1, 0]T
zm contains the selected topic and by combined with β.
Kyunghoon Kim Graduate Students Pitching Topic Modeling 24 / 37

37. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Dirichlet Distribution
Let θ = [θ1, θ2, · · · , θk] be a random pmf, that is θi ≥ 0 for
i = 1, 2, · · · , k and k
i=1 θi = 1.
And suppose α = [α1, α2, · · · , αk], with αi > 0 for each i.
p(θ|α) =
Γ( k
i αi )
k
i Γ(αi )
k
i=1
θαi −1
i
Kyunghoon Kim Graduate Students Pitching Topic Modeling 25 / 37

38. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Dirichlet Distribution
Kyunghoon Kim Graduate Students Pitching Topic Modeling 26 / 37

39. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Dirichlet Distribution
Kyunghoon Kim Graduate Students Pitching Topic Modeling 27 / 37

40. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Dirichlet Distribution
http://parkcu.com/blog/wp-content/uploads/2013/07/geometric-interpretation-of-dirichlet-distribution.png
Kyunghoon Kim Graduate Students Pitching Topic Modeling 28 / 37

41. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Latent Dirichlet Allocation
Kyunghoon Kim Graduate Students Pitching Topic Modeling 29 / 37

42. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Latent Dirichlet Allocation
Kyunghoon Kim Graduate Students Pitching Topic Modeling 30 / 37

43. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Latent Dirichlet Allocation
Generative Process
1 choose a distribution over topics
2 repeatedly draw a word(color) from each distribution
3 lookup what each word topic it belongs to by the color
4 choose the word from that distribution
Kyunghoon Kim Graduate Students Pitching Topic Modeling 31 / 37

44. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Latent Dirichlet Allocation
http://parkcu.com/blog/wp-content/uploads/2013/07/LDA.png
Kyunghoon Kim Graduate Students Pitching Topic Modeling 32 / 37

45. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Inference
Given the parameters α, β, the joint distribution of a topic mixture
θ, a set of N topics z, and a set of N words w is given by :
p(θ, z, w|α, β) = p(θ|α)
N
n=1
p(zn|θ)p(wn|zn, β) (7)
Kyunghoon Kim Graduate Students Pitching Topic Modeling 33 / 37

46. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Inference
Given the parameters α, β, the joint distribution of a topic mixture
θ, a set of N topics z, and a set of N words w is given by :
p(θ, z, w|α, β) = p(θ|α)
N
n=1
p(zn|θ)p(wn|zn, β) (7)
Integrating over θ and summing over z, we obtain the marginal
distribution of a document:
p(w|α, β) = p(θ|α)
N
n=1 zn
p(zn|θ)p(wn|zn, β) dθ (8)
Kyunghoon Kim Graduate Students Pitching Topic Modeling 33 / 37

47. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Inference
p(D|α, β) =
M
d=1
p(w|α, β) (9)
Kyunghoon Kim Graduate Students Pitching Topic Modeling 34 / 37

48. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Inference
• Gibbs Sampling(MCMC)
• Variational Inference
• · · ·
Kyunghoon Kim Graduate Students Pitching Topic Modeling 35 / 37

49. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
Result
[1]
Kyunghoon Kim Graduate Students Pitching Topic Modeling 36 / 37

50. Bayes Law
Bayesian Network
Latent Dirichlet Allocation
References
References I
David M Blei, Andrew Y Ng and Michael I Jordan. “Latent
dirichlet allocation”. In: the Journal of machine Learning
research 3 (2003), pp. 993–1022.
Rachel Schutt and Cathy O’Neil. Doing data science: Straight
talk from the frontline. ” O’Reilly Media, Inc.”, 2013.
Seunghwan Shin. Probablistic programming - basic principle.
” Acorn publish”, 2015.
Kyunghoon Kim Graduate Students Pitching Topic Modeling 37 / 37