Presentation from Moscow Data Fest #1, September 12. Moscow Data Fest is a free one-day event that brings together Data Scientists for sessions on both theory and practice. Link: http://www.meetup.com/Moscow-Data-Fest/

1. Introduction to
word embeddings
Pavel Kalaidin
@facultyofwonder
Moscow Data Fest, September, 12th, 2015

4. distributional hypothesis

5. лойс

6. годно, лойс
лойс за песню
из принципа не поставлю лойс
взаимные лойсы
лойс, если согласен
What is the meaning of лойс?

7. годно, лойс
лойс за песню
из принципа не поставлю лойс
взаимные лойсы
лойс, если согласен
What is the meaning of лойс?

8. кек

9. кек, что ли?
кек)))))))
ну ты кек
What is the meaning of кек?

10. кек, что ли?
кек)))))))
ну ты кек
What is the meaning of кек?

11. vectorial representations
of words

12. simple and flexible
platform for
understanding text and
probably not messing up

13. one-hot encoding?
1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

14. co-occurrence matrix
recall: word-document co-occurrence
matrix for LSA

15. credits: [x]

16. from entire document to
window (length 5-10)

17. still seems suboptimal ->
big, sparse, etc.

18. lower dimensions, we
want dense vectors
(say, 25-1000)

19. How?

20. matrix factorization?

21. SVD of co-occurrence
matrix

22. lots of memory?

23. idea: directly learn low-
dimensional vectors

24. here comes word2vec
Distributed Representations of Words and Phrases and their Compositionality, Mikolov et al: [paper]

25. idea: instead of capturing co-
occurrence counts
predict surrounding words

26. Two models:
C-BOW
predicting the word given its context
skip-gram
predicting the context given a word
Explained in great detail here, so we’ll skip it for now Also see: word2vec Parameter
Learning Explained, Rong, paper

28. CBOW: several times faster than skip-gram,
slightly better accuracy for the frequent words
Skip-Gram: works well with small amount of
data, represents well rare words or phrases

29. Examples?

36. Wwoman
- Wman
= Wqueen
-
Wking
classic example

37. <censored example>

38. word2vec Explained: Deriving Mikolov et al.’s Negative-Sampling
Word-Embedding Method, Goldberg et al, 2014 [arxiv]

39. all done with gensim:
github.com/piskvorky/gensim/

40. ...failing to take advantage of
the vast amount of repetition
in the data

41. so back to co-occurrences

42. GloVe for Global Vectors
Pennington et al, 2014: nlp.stanford.
edu/pubs/glove.pdf

43. Ratios seem to cancel noise

44. The gist: model ratios with
vectors

45. The model

46. Preserving
linearity

47. Preventing mixing
dimensions

48. Restoring
symmetry, part 1

49. recall:

51. Restoring symmetry, part 2

52. Least squares problem it is now

53. SGD->AdaGrad

54. ok, Python code

55. glove-python:
github.com/maciejkula/glove-python

56. two sets of vectors
input and context + bias
average/sum/drop

57. complexity |V|2

58. complexity |C|0.8

59. Evaluation: it works

60. #spb
#gatchina
#msk
#kyiv
#minsk
#helsinki

61. Compared to word2vec

62. #spb
#gatchina
#msk
#kyiv
#minsk
#helsinki

64. t-SNE:
github.com/oreillymedia/t-SNE-tutorial
seaborn:
stanford.edu/~mwaskom/software/seaborn/

65. Abusing models

66. music playlists:
github.com/mattdennewitz/playlist-to-vec

67. deep walk:
DeepWalk: Online Learning of Social
Representations [link]

68. user interests
Paragraph vectors: cs.stanford.
edu/~quocle/paragraph_vector.pdf

69. predicting hashtags
interesting read: #TAGSPACE: Semantic
Embeddings from Hashtags [link]

70. RusVectōrēs: distributional semantic
models for Russian: ling.go.mail.
ru/dsm/en/

72. corpus matters

73. building block for
bigger models
╰(*´︶`*)╯

74. </slides>