Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
A word is worth a
thousand vectors
(word2vec, lda, and introducing lda2vec)
Christopher Moody
@ Stitch Fix
About
@chrisemoody
Caltech Physics
PhD. in astrostats supercomputing
sklearn t-SNE contributor
Data Labs at Stitch Fix
git...
Credit
Large swathes of this talk are from
previous presentations by:
• Tomas Mikolov
• David Blei
• Christopher Olah
• Ra...
word2vec
lda
1
2
lda2vec
1. king - man + woman = queen
2. Huge splash in NLP world
3. Learns from raw text
4. Pretty simple algorithm
5. Comes pret...
word2vec
1. Set up an objective function
2. Randomly initialize vectors
3. Do gradient descent
w
ord2vec
word2vec: learn word vector vin
from it’s surrounding context
vin
w
ord2vec
“The fox jumped over the lazy dog”
Maximize the likelihood of seeing the words given the word over.
P(the|over)
...
w
ord2vec
P(fox|over)
What should this be?
w
ord2vec
P(vfox|vover)
Should depend on the word vectors.
P(fox|over)
w
ord2vec
Twist: we have two vectors for every word.
Should depend on whether it’s the input or the output.
Also a context...
w
ord2vec
“The fox jumped over the lazy dog”
vIN
P(vOUT|vIN)
Twist: we have two vectors for every word.
Should depend on w...
w
ord2vec
“The fox jumped over the lazy dog”
vOUT
P(vOUT|vIN)
vIN
Twist: we have two vectors for every word.
Should depend...
w
ord2vec
“The fox jumped over the lazy dog”
vOUT
P(vOUT|vIN)
vIN
Twist: we have two vectors for every word.
Should depend...
w
ord2vec
“The fox jumped over the lazy dog”
vOUT
P(vOUT|vIN)
vIN
Twist: we have two vectors for every word.
Should depend...
w
ord2vec
“The fox jumped over the lazy dog”
vOUT
P(vOUT|vIN)
vIN
Twist: we have two vectors for every word.
Should depend...
w
ord2vec
“The fox jumped over the lazy dog”
vOUT
P(vOUT|vIN)
vIN
Twist: we have two vectors for every word.
Should depend...
w
ord2vec
“The fox jumped over the lazy dog”
vOUT
P(vOUT|vIN)
vIN
Twist: we have two vectors for every word.
Should depend...
w
ord2vec
P(vOUT|vIN)
“The fox jumped over the lazy dog”
vIN
Twist: we have two vectors for every word.
Should depend on w...
w
ord2vec
“The fox jumped over the lazy dog”
vOUT
P(vOUT|vIN)
vIN
Twist: we have two vectors for every word.
Should depend...
w
ord2vec
“The fox jumped over the lazy dog”
vOUT
P(vOUT|vIN)
vIN
Twist: we have two vectors for every word.
Should depend...
w
ord2vec
“The fox jumped over the lazy dog”
vOUT
P(vOUT|vIN)
vIN
Twist: we have two vectors for every word.
Should depend...
w
ord2vec
“The fox jumped over the lazy dog”
vOUT
P(vOUT|vIN)
vIN
Twist: we have two vectors for every word.
Should depend...
w
ord2vec
“The fox jumped over the lazy dog”
vOUT
P(vOUT|vIN)
vIN
Twist: we have two vectors for every word.
Should depend...
w
ord2vec
“The fox jumped over the lazy dog”
vOUT
P(vOUT|vIN)
vIN
Twist: we have two vectors for every word.
Should depend...
objective
Measure loss between
vIN and vOUT?
vin . vout
How should we define P(vOUT|vIN)?
w
ord2vec
vin . vout ~ 1
objective
vin
vout
w
ord2vec
objective
vin
vout
vin . vout ~ 0
w
ord2vec
objective
vin
vout
vin . vout ~ -1
w
ord2vec
vin . vout ∈ [-1,1]
objective
w
ord2vec
But we’d like to measure a probability.
vin . vout ∈ [-1,1]
objective
w
ord2vec
But we’d like to measure a probability.
softmax(vin . vout ∈ [-1,1])
objective
∈ [0,1]
w
ord2vec
But we’d like to measure a probability.
softmax(vin . vout ∈ [-1,1])
Probability of choosing 1 of N discrete ite...
w
ord2vec
But we’d like to measure a probability.
exp(vin . vout ∈ [0,1])softmax ~
objective
w
ord2vec
But we’d like to measure a probability.
exp(vin . vout ∈ [-1,1])
Σexp(vin . vk)
softmax =
objective
Normalizatio...
w
ord2vec
But we’d like to measure a probability.
exp(vin . vout ∈ [-1,1])
Σexp(vin . vk)
softmax = = P(vout|vin)
objectiv...
w
ord2vec
Learn by gradient descent on the softmax prob.
For every example we see update vin
vin := vin + P(vout|vin)
obje...
word2vec
word2vec
ITEM_3469 + ‘Pregnant’
+ ‘Pregnant’
= ITEM_701333
= ITEM_901004
= ITEM_800456
what about?LDA?
LDA
on Client Item
Descriptions
LDA
on Item
Descriptions
(with Jay)
LDA
on Item
Descriptions
(with Jay)
LDA
on Item
Descriptions
(with Jay)
LDA
on Item
Descriptions
(with Jay)
LDA
on Item
Descriptions
(with Jay)
Latent style vectors from text
Pairwise gamma correlation
from style ratings
Diversity from ratings Diversity from text
lda vs word2vec
word2vec is local:
one word predicts a nearby word
“I love finding new designer brands for jeans”
“I love finding new designer brands for jeans”
But text is usually organized.
“I love finding new designer brands for jeans”
But text is usually organized.
“I love finding new designer brands for jeans”
In LDA, documents globally predict words.
doc 7681
[ -0.75, -1.25, -0.55, -0.12, +2.2] [ 0%, 9%, 78%, 11%]
typical word2vec vector typical LDA document vector
typical word2vec vector
[ 0%, 9%, 78%, 11%]
typical LDA document vector
[ -0.75, -1.25, -0.55, -0.12, +2.2]
All sum to 100...
5D word2vec vector
[ 0%, 9%, 78%, 11%]
5D LDA document vector
[ -0.75, -1.25, -0.55, -0.12, +2.2]
Sparse
All sum to 100%
D...
100D word2vec vector
[ 0%0%0%0%0% … 0%, 9%, 78%, 11%]
100D LDA document vector
[ -0.75, -1.25, -0.55, -0.27, -0.94, 0.44, ...
100D word2vec vector
[ 0%0%0%0%0% … 0%, 9%, 78%, 11%]
100D LDA document vector
[ -0.75, -1.25, -0.55, -0.27, -0.94, 0.44, ...
can we do both? lda2vec
The goal:
Use all of this context to learn
interpretable topics.
P(vOUT |vIN)word2vec
@chrisemoody
word2vec
LDA P(vOUT |vDOC)
The goal:
Use all of this context to learn
interpretable topics.
this document is
80% high fash...
word2vec
LDA
The goal:
Use all of this context to learn
interpretable topics.
this zip code is
80% hot climate
this zip co...
word2vec
LDA
The goal:
Use all of this context to learn
interpretable topics.
this client is
80% sporty
this client is
60%...
lda2vec
word2vec predicts locally:
one word predicts a nearby word
P(vOUT |vIN)
vIN vOUT
“PS! Thank you for such an awesom...
lda2vec
LDA predicts a word from a global context
doc_id=1846
P(vOUT |vDOC)
vOUT
vDOC
“PS! Thank you for such an awesome t...
lda2vec
doc_id=1846
vIN vOUT
vDOC
can we predict a word both locally and globally ?
“PS! Thank you for such an awesome top”
lda2vec
“PS! Thank you for such an awesome top”doc_id=1846
vIN vOUT
vDOC
can we predict a word both locally and globally ?...
lda2vec
doc_id=1846
vIN vOUT
vDOC
*very similar to the Paragraph Vectors / doc2vec
can we predict a word both locally and ...
lda2vec
This works! 😀 But vDOC isn’t as
interpretable as the LDA topic vectors. 😔
lda2vec
This works! 😀 But vDOC isn’t as
interpretable as the LDA topic vectors. 😔
lda2vec
This works! 😀 But vDOC isn’t as
interpretable as the LDA topic vectors. 😔
lda2vec
This works! 😀 But vDOC isn’t as
interpretable as the LDA topic vectors. 😔
We’re missing mixtures & sparsity.
lda2vec
This works! 😀 But vDOC isn’t as
interpretable as the LDA topic vectors. 😔
Let’s make vDOC into a mixture…
lda2vec
Let’s make vDOC into a mixture…
vDOC = a vtopic1 + b vtopic2 +… (up to k topics)
lda2vec
Let’s make vDOC into a mixture…
vDOC = a vtopic1 + b vtopic2 +…
Trinitarian
baptismal
Pentecostals
Bede
schismatic...
lda2vec
Let’s make vDOC into a mixture…
vDOC = a vtopic1 + b vtopic2 +…
topic 1 = “religion”
Trinitarian
baptismal
Penteco...
lda2vec
Let’s make vDOC into a mixture…
vDOC = a vtopic1 + b vtopic2 +…
Milosevic
absentee
Indonesia
Lebanese
Isrealis
Kar...
lda2vec
Let’s make vDOC into a mixture…
vDOC = a vtopic1 + b vtopic2 +…
topic 1 = “religion”
Trinitarian
baptismal
Penteco...
lda2vec
Let’s make vDOC into a mixture…
vDOC = 10% religion + 89% politics +…
topic 2 = “politics”
Milosevic
absentee
Indo...
lda2vec
Let’s make vDOC sparse
[ -0.75, -1.25, …]
vDOC = a vreligion + b vpolitics +…
lda2vec
Let’s make vDOC sparse
vDOC = a vreligion + b vpolitics +…
lda2vec
Let’s make vDOC sparse
vDOC = a vreligion + b vpolitics +…
lda2vec
Let’s make vDOC sparse
{a, b, c…} ~ dirichlet(alpha)
vDOC = a vreligion + b vpolitics +…
lda2vec
Let’s make vDOC sparse
{a, b, c…} ~ dirichlet(alpha)
vDOC = a vreligion + b vpolitics +…
word2vec
LDA
P(vOUT |vIN + vDOC)lda2vec
The goal:
Use all of this context to learn
interpretable topics.
@chrisemoody
this...
word2vec
LDA
P(vOUT |vIN+ vDOC + vZIP)lda2vec
The goal:
Use all of this context to learn
interpretable topics.
@chrisemoody
word2vec
LDA
P(vOUT |vIN+ vDOC + vZIP)lda2vec
The goal:
Use all of this context to learn
interpretable topics.
this zip co...
word2vec
LDA
P(vOUT |vIN+ vDOC + vZIP +vCLIENTS)lda2vec
The goal:
Use all of this context to learn
interpretable topics.
t...
word2vec
LDA
P(vOUT |vIN+ vDOC + vZIP +vCLIENTS)
P(sold | vCLIENTS)
lda2vec
The goal:
Use all of this context to learn
int...
github.com/cemoody/lda2vec
uses pyldavis
API Ref docs (no narrative docs)
GPU
Decent test coverage
@chrisemoody
“PS! Thank you for such an awesome idea”
@chrisemoody
doc_id=1846
Can we model topics to sentences?
lda2lstm
“PS! Thank you for such an awesome idea”
@chrisemoody
doc_id=1846
Can we represent the internal LSTM
states as a dirichlet...
Can we model topics to sentences?
lda2lstm
“PS! Thank you for such an awesome idea”doc_id=1846
@chrisemoody
Can we model t...
?@chrisemoody
Multithreaded
Stitch Fix
Bonus slides
Crazy
Approaches
Paragraph Vectors
(Just extend the context window)
Content dependency
(Change the window grammatically)
S...
CBOW
“The fox jumped over the lazy dog”
Guess the word
given the context
~20x faster.
(this is the alternative.)
vOUT
vIN ...
LDA
Results
context
H
istory
I loved every choice in this fix!! Great job!
Great Stylist Perfect
LDA
Results
context
H
istory
Body Fit
My measurements are 36-28-32. If that helps.
I like wearing some clothing that is fit...
LDA
Results
context
H
istory
Sizing
Really enjoyed the experience and the
pieces, sizing for tops was too big.
Looking for...
LDA
Results
context
H
istory
Almost Bought
It was a great fix. Loved the two items I
kept and the three I sent back were cl...
What I didn’t mention
A lot of text (only if you have a specialized vocabulary)
Cleaning the text
Memory & performance
Tra...
and now for something completely crazy
All of the following ideas will change what
‘words’ and ‘context’ represent.
paragraph
vector
What about summarizing documents?
On the day he took office, President Obama reached out to America’s enem...
On the day he took office, President Obama reached out to America’s enemies,
offering in his first inaugural address to exte...
On the day he took office, President Obama reached out to America’s enemies,
offering in his first inaugural address to exte...
from	gensim.models	import	Doc2Vec		
fn	=	“item_document_vectors”		
model	=	Doc2Vec.load(fn)		
model.most_similar('pregnant...
translation
(using just a rotation
matrix)
M
ikolov
2013
English
Spanish
Matrix
Rotation
context
dependent
Levy
&
G
oldberg
2014
Australian scientist discovers star with telescope
context +/- 2 words
context
dependent
context
Australian scientist discovers star with telescope
Levy
&
G
oldberg
2014
context
dependent
context
Australian scientist discovers star with telescope
context
Levy
&
G
oldberg
2014
context
dependent
context
BoW DEPS
topically-similar vs ‘functionally’ similar
Levy
&
G
oldberg
2014
context
dependent
context
Levy
&
G
oldberg
2014
Also show that SGNS is simply factorizing:
w * c = PMI(w, c) - log k
This ...
deepwalk
Perozzi
etal2014
learn word vectors from
sentences
“The fox jumped over the lazy dog”
vOUT vOUT vOUT vOUT vOUTvOU...
Playlists at
Spotify
context
sequence
learning
‘words’ are songs
‘sentences’ are playlists
Playlists at
Spotify
context
Erik
Bernhardsson
Great performance on ‘related artists’
Fixes at
Stitch Fix
sequence
learning
Let’s try:
‘words’ are styles
‘sentences’ are fixes
Fixes at
Stitch Fix
context
Learn similarity between styles
because they co-occur
Learn ‘coherent’ styles
sequence
learning
Fixes at
Stitch Fix?
context
sequence
learning
Got lots of structure!
Fixes at
Stitch Fix?
context
sequence
learning
Fixes at
Stitch Fix?
context
sequence
learning
Nearby regions are
consistent ‘closets’
A specific lda2vec model
Our text blob is a comment that comes from a region_id and a style_id
lda2vec
Let’s make vDOC into a mixture…
vDOC = 10% religion + 89% politics +…
topic 2 = “politics”
Milosevic
absentee
Indo...
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
word2vec, LDA, and introducing a new hybrid algorithm: lda2vec
Prochain SlideShare
Chargement dans…5
×

word2vec, LDA, and introducing a new hybrid algorithm: lda2vec

70 385 vues

Publié le

Available with notes:
http://www.slideshare.net/ChristopherMoody3/word2vec-lda-and-introducing-a-new-hybrid-algorithm-lda2vec

(Data Day 2016)

Standard natural language processing (NLP) is a messy and difficult affair. It requires teaching a computer about English-specific word ambiguities as well as the hierarchical, sparse nature of words in sentences. At Stitch Fix, word vectors help computers learn from the raw text in customer notes. Our systems need to identify a medical professional when she writes that she 'used to wear scrubs to work', and distill 'taking a trip' into a Fix for vacation clothing. Applied appropriately, word vectors are dramatically more meaningful and more flexible than current techniques and let computers peer into text in a fundamentally new way. I'll try to convince you that word vectors give us a simple and flexible platform for understanding text while speaking about word2vec, LDA, and introduce our hybrid algorithm lda2vec.

Publié dans : Sciences
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD FULL eBOOK INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookeBOOK Crime, eeBOOK Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD FULL eBOOK INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookeBOOK Crime, eeBOOK Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

word2vec, LDA, and introducing a new hybrid algorithm: lda2vec

  1. 1. A word is worth a thousand vectors (word2vec, lda, and introducing lda2vec) Christopher Moody @ Stitch Fix
  2. 2. About @chrisemoody Caltech Physics PhD. in astrostats supercomputing sklearn t-SNE contributor Data Labs at Stitch Fix github.com/cemoody Gaussian Processes t-SNE chainer deep learning Tensor Decomposition
  3. 3. Credit Large swathes of this talk are from previous presentations by: • Tomas Mikolov • David Blei • Christopher Olah • Radim Rehurek • Omer Levy & Yoav Goldberg • Richard Socher • Xin Rong • Tim Hopper
  4. 4. word2vec lda 1 2 lda2vec
  5. 5. 1. king - man + woman = queen 2. Huge splash in NLP world 3. Learns from raw text 4. Pretty simple algorithm 5. Comes pretrained word2vec
  6. 6. word2vec 1. Set up an objective function 2. Randomly initialize vectors 3. Do gradient descent
  7. 7. w ord2vec word2vec: learn word vector vin from it’s surrounding context vin
  8. 8. w ord2vec “The fox jumped over the lazy dog” Maximize the likelihood of seeing the words given the word over. P(the|over) P(fox|over) P(jumped|over) P(the|over) P(lazy|over) P(dog|over) …instead of maximizing the likelihood of co-occurrence counts.
  9. 9. w ord2vec P(fox|over) What should this be?
  10. 10. w ord2vec P(vfox|vover) Should depend on the word vectors. P(fox|over)
  11. 11. w ord2vec Twist: we have two vectors for every word. Should depend on whether it’s the input or the output. Also a context window around every input word. “The fox jumped over the lazy dog” P(vOUT|vIN)
  12. 12. w ord2vec “The fox jumped over the lazy dog” vIN P(vOUT|vIN) Twist: we have two vectors for every word. Should depend on whether it’s the input or the output. Also a context window around every input word.
  13. 13. w ord2vec “The fox jumped over the lazy dog” vOUT P(vOUT|vIN) vIN Twist: we have two vectors for every word. Should depend on whether it’s the input or the output. Also a context window around every input word.
  14. 14. w ord2vec “The fox jumped over the lazy dog” vOUT P(vOUT|vIN) vIN Twist: we have two vectors for every word. Should depend on whether it’s the input or the output. Also a context window around every input word.
  15. 15. w ord2vec “The fox jumped over the lazy dog” vOUT P(vOUT|vIN) vIN Twist: we have two vectors for every word. Should depend on whether it’s the input or the output. Also a context window around every input word.
  16. 16. w ord2vec “The fox jumped over the lazy dog” vOUT P(vOUT|vIN) vIN Twist: we have two vectors for every word. Should depend on whether it’s the input or the output. Also a context window around every input word.
  17. 17. w ord2vec “The fox jumped over the lazy dog” vOUT P(vOUT|vIN) vIN Twist: we have two vectors for every word. Should depend on whether it’s the input or the output. Also a context window around every input word.
  18. 18. w ord2vec “The fox jumped over the lazy dog” vOUT P(vOUT|vIN) vIN Twist: we have two vectors for every word. Should depend on whether it’s the input or the output. Also a context window around every input word.
  19. 19. w ord2vec P(vOUT|vIN) “The fox jumped over the lazy dog” vIN Twist: we have two vectors for every word. Should depend on whether it’s the input or the output. Also a context window around every input word.
  20. 20. w ord2vec “The fox jumped over the lazy dog” vOUT P(vOUT|vIN) vIN Twist: we have two vectors for every word. Should depend on whether it’s the input or the output. Also a context window around every input word.
  21. 21. w ord2vec “The fox jumped over the lazy dog” vOUT P(vOUT|vIN) vIN Twist: we have two vectors for every word. Should depend on whether it’s the input or the output. Also a context window around every input word.
  22. 22. w ord2vec “The fox jumped over the lazy dog” vOUT P(vOUT|vIN) vIN Twist: we have two vectors for every word. Should depend on whether it’s the input or the output. Also a context window around every input word.
  23. 23. w ord2vec “The fox jumped over the lazy dog” vOUT P(vOUT|vIN) vIN Twist: we have two vectors for every word. Should depend on whether it’s the input or the output. Also a context window around every input word.
  24. 24. w ord2vec “The fox jumped over the lazy dog” vOUT P(vOUT|vIN) vIN Twist: we have two vectors for every word. Should depend on whether it’s the input or the output. Also a context window around every input word.
  25. 25. w ord2vec “The fox jumped over the lazy dog” vOUT P(vOUT|vIN) vIN Twist: we have two vectors for every word. Should depend on whether it’s the input or the output. Also a context window around every input word.
  26. 26. objective Measure loss between vIN and vOUT? vin . vout How should we define P(vOUT|vIN)?
  27. 27. w ord2vec vin . vout ~ 1 objective vin vout
  28. 28. w ord2vec objective vin vout vin . vout ~ 0
  29. 29. w ord2vec objective vin vout vin . vout ~ -1
  30. 30. w ord2vec vin . vout ∈ [-1,1] objective
  31. 31. w ord2vec But we’d like to measure a probability. vin . vout ∈ [-1,1] objective
  32. 32. w ord2vec But we’d like to measure a probability. softmax(vin . vout ∈ [-1,1]) objective ∈ [0,1]
  33. 33. w ord2vec But we’d like to measure a probability. softmax(vin . vout ∈ [-1,1]) Probability of choosing 1 of N discrete items. Mapping from vector space to a multinomial over words. objective
  34. 34. w ord2vec But we’d like to measure a probability. exp(vin . vout ∈ [0,1])softmax ~ objective
  35. 35. w ord2vec But we’d like to measure a probability. exp(vin . vout ∈ [-1,1]) Σexp(vin . vk) softmax = objective Normalization term over all words k ∈ V
  36. 36. w ord2vec But we’d like to measure a probability. exp(vin . vout ∈ [-1,1]) Σexp(vin . vk) softmax = = P(vout|vin) objective k ∈ V
  37. 37. w ord2vec Learn by gradient descent on the softmax prob. For every example we see update vin vin := vin + P(vout|vin) objective vout := vout + P(vout|vin)
  38. 38. word2vec
  39. 39. word2vec
  40. 40. ITEM_3469 + ‘Pregnant’
  41. 41. + ‘Pregnant’
  42. 42. = ITEM_701333 = ITEM_901004 = ITEM_800456
  43. 43. what about?LDA?
  44. 44. LDA on Client Item Descriptions
  45. 45. LDA on Item Descriptions (with Jay)
  46. 46. LDA on Item Descriptions (with Jay)
  47. 47. LDA on Item Descriptions (with Jay)
  48. 48. LDA on Item Descriptions (with Jay)
  49. 49. LDA on Item Descriptions (with Jay)
  50. 50. Latent style vectors from text Pairwise gamma correlation from style ratings Diversity from ratings Diversity from text
  51. 51. lda vs word2vec
  52. 52. word2vec is local: one word predicts a nearby word “I love finding new designer brands for jeans”
  53. 53. “I love finding new designer brands for jeans” But text is usually organized.
  54. 54. “I love finding new designer brands for jeans” But text is usually organized.
  55. 55. “I love finding new designer brands for jeans” In LDA, documents globally predict words. doc 7681
  56. 56. [ -0.75, -1.25, -0.55, -0.12, +2.2] [ 0%, 9%, 78%, 11%] typical word2vec vector typical LDA document vector
  57. 57. typical word2vec vector [ 0%, 9%, 78%, 11%] typical LDA document vector [ -0.75, -1.25, -0.55, -0.12, +2.2] All sum to 100%All real values
  58. 58. 5D word2vec vector [ 0%, 9%, 78%, 11%] 5D LDA document vector [ -0.75, -1.25, -0.55, -0.12, +2.2] Sparse All sum to 100% Dimensions are absolute Dense All real values Dimensions relative
  59. 59. 100D word2vec vector [ 0%0%0%0%0% … 0%, 9%, 78%, 11%] 100D LDA document vector [ -0.75, -1.25, -0.55, -0.27, -0.94, 0.44, 0.05, 0.31 … -0.12, +2.2] Sparse All sum to 100% Dimensions are absolute Dense All real values Dimensions relative dense sparse
  60. 60. 100D word2vec vector [ 0%0%0%0%0% … 0%, 9%, 78%, 11%] 100D LDA document vector [ -0.75, -1.25, -0.55, -0.27, -0.94, 0.44, 0.05, 0.31 … -0.12, +2.2] Similar in fewer ways (more interpretable) Similar in 100D ways (very flexible) +mixture +sparse
  61. 61. can we do both? lda2vec
  62. 62. The goal: Use all of this context to learn interpretable topics. P(vOUT |vIN)word2vec @chrisemoody
  63. 63. word2vec LDA P(vOUT |vDOC) The goal: Use all of this context to learn interpretable topics. this document is 80% high fashion this document is 60% style @chrisemoody
  64. 64. word2vec LDA The goal: Use all of this context to learn interpretable topics. this zip code is 80% hot climate this zip code is 60% outdoors wear @chrisemoody
  65. 65. word2vec LDA The goal: Use all of this context to learn interpretable topics. this client is 80% sporty this client is 60% casual wear @chrisemoody
  66. 66. lda2vec word2vec predicts locally: one word predicts a nearby word P(vOUT |vIN) vIN vOUT “PS! Thank you for such an awesome top”
  67. 67. lda2vec LDA predicts a word from a global context doc_id=1846 P(vOUT |vDOC) vOUT vDOC “PS! Thank you for such an awesome top”
  68. 68. lda2vec doc_id=1846 vIN vOUT vDOC can we predict a word both locally and globally ? “PS! Thank you for such an awesome top”
  69. 69. lda2vec “PS! Thank you for such an awesome top”doc_id=1846 vIN vOUT vDOC can we predict a word both locally and globally ? P(vOUT |vIN+ vDOC)
  70. 70. lda2vec doc_id=1846 vIN vOUT vDOC *very similar to the Paragraph Vectors / doc2vec can we predict a word both locally and globally ? “PS! Thank you for such an awesome top” P(vOUT |vIN+ vDOC)
  71. 71. lda2vec This works! 😀 But vDOC isn’t as interpretable as the LDA topic vectors. 😔
  72. 72. lda2vec This works! 😀 But vDOC isn’t as interpretable as the LDA topic vectors. 😔
  73. 73. lda2vec This works! 😀 But vDOC isn’t as interpretable as the LDA topic vectors. 😔
  74. 74. lda2vec This works! 😀 But vDOC isn’t as interpretable as the LDA topic vectors. 😔 We’re missing mixtures & sparsity.
  75. 75. lda2vec This works! 😀 But vDOC isn’t as interpretable as the LDA topic vectors. 😔 Let’s make vDOC into a mixture…
  76. 76. lda2vec Let’s make vDOC into a mixture… vDOC = a vtopic1 + b vtopic2 +… (up to k topics)
  77. 77. lda2vec Let’s make vDOC into a mixture… vDOC = a vtopic1 + b vtopic2 +… Trinitarian baptismal Pentecostals Bede schismatics excommunication
  78. 78. lda2vec Let’s make vDOC into a mixture… vDOC = a vtopic1 + b vtopic2 +… topic 1 = “religion” Trinitarian baptismal Pentecostals Bede schismatics excommunication
  79. 79. lda2vec Let’s make vDOC into a mixture… vDOC = a vtopic1 + b vtopic2 +… Milosevic absentee Indonesia Lebanese Isrealis Karadzic topic 1 = “religion” Trinitarian baptismal Pentecostals Bede schismatics excommunication
  80. 80. lda2vec Let’s make vDOC into a mixture… vDOC = a vtopic1 + b vtopic2 +… topic 1 = “religion” Trinitarian baptismal Pentecostals bede schismatics excommunication topic 2 = “politics” Milosevic absentee Indonesia Lebanese Isrealis Karadzic
  81. 81. lda2vec Let’s make vDOC into a mixture… vDOC = 10% religion + 89% politics +… topic 2 = “politics” Milosevic absentee Indonesia Lebanese Isrealis Karadzic topic 1 = “religion” Trinitarian baptismal Pentecostals bede schismatics excommunication
  82. 82. lda2vec Let’s make vDOC sparse [ -0.75, -1.25, …] vDOC = a vreligion + b vpolitics +…
  83. 83. lda2vec Let’s make vDOC sparse vDOC = a vreligion + b vpolitics +…
  84. 84. lda2vec Let’s make vDOC sparse vDOC = a vreligion + b vpolitics +…
  85. 85. lda2vec Let’s make vDOC sparse {a, b, c…} ~ dirichlet(alpha) vDOC = a vreligion + b vpolitics +…
  86. 86. lda2vec Let’s make vDOC sparse {a, b, c…} ~ dirichlet(alpha) vDOC = a vreligion + b vpolitics +…
  87. 87. word2vec LDA P(vOUT |vIN + vDOC)lda2vec The goal: Use all of this context to learn interpretable topics. @chrisemoody this document is 80% high fashion this document is 60% style
  88. 88. word2vec LDA P(vOUT |vIN+ vDOC + vZIP)lda2vec The goal: Use all of this context to learn interpretable topics. @chrisemoody
  89. 89. word2vec LDA P(vOUT |vIN+ vDOC + vZIP)lda2vec The goal: Use all of this context to learn interpretable topics. this zip code is 80% hot climate this zip code is 60% outdoors wear @chrisemoody
  90. 90. word2vec LDA P(vOUT |vIN+ vDOC + vZIP +vCLIENTS)lda2vec The goal: Use all of this context to learn interpretable topics. this client is 80% sporty this client is 60% casual wear @chrisemoody
  91. 91. word2vec LDA P(vOUT |vIN+ vDOC + vZIP +vCLIENTS) P(sold | vCLIENTS) lda2vec The goal: Use all of this context to learn interpretable topics. @chrisemoody Can also make the topics supervised so that they predict an outcome.
  92. 92. github.com/cemoody/lda2vec uses pyldavis API Ref docs (no narrative docs) GPU Decent test coverage @chrisemoody
  93. 93. “PS! Thank you for such an awesome idea” @chrisemoody doc_id=1846 Can we model topics to sentences? lda2lstm
  94. 94. “PS! Thank you for such an awesome idea” @chrisemoody doc_id=1846 Can we represent the internal LSTM states as a dirichlet mixture?
  95. 95. Can we model topics to sentences? lda2lstm “PS! Thank you for such an awesome idea”doc_id=1846 @chrisemoody Can we model topics to images? lda2ae TJ Torres
  96. 96. ?@chrisemoody Multithreaded Stitch Fix
  97. 97. Bonus slides
  98. 98. Crazy Approaches Paragraph Vectors (Just extend the context window) Content dependency (Change the window grammatically) Social word2vec (deepwalk) (Sentence is a walk on the graph) Spotify (Sentence is a playlist of song_ids) Stitch Fix (Sentence is a shipment of five items)
  99. 99. CBOW “The fox jumped over the lazy dog” Guess the word given the context ~20x faster. (this is the alternative.) vOUT vIN vINvIN vINvIN vIN SkipGram “The fox jumped over the lazy dog” vOUT vOUT vIN vOUT vOUT vOUTvOUT Guess the context given the word Better at syntax. (this is the one we went over)
  100. 100. LDA Results context H istory I loved every choice in this fix!! Great job! Great Stylist Perfect
  101. 101. LDA Results context H istory Body Fit My measurements are 36-28-32. If that helps. I like wearing some clothing that is fitted. Very hard for me to find pants that fit right.
  102. 102. LDA Results context H istory Sizing Really enjoyed the experience and the pieces, sizing for tops was too big. Looking forward to my next box! Excited for next
  103. 103. LDA Results context H istory Almost Bought It was a great fix. Loved the two items I kept and the three I sent back were close! Perfect
  104. 104. What I didn’t mention A lot of text (only if you have a specialized vocabulary) Cleaning the text Memory & performance Traditional databases aren’t well-suited False positives
  105. 105. and now for something completely crazy
  106. 106. All of the following ideas will change what ‘words’ and ‘context’ represent.
  107. 107. paragraph vector What about summarizing documents? On the day he took office, President Obama reached out to America’s enemies, offering in his first inaugural address to extend a hand if you are willing to unclench your fist. More than six years later, he has arrived at a moment of truth in testing that
  108. 108. On the day he took office, President Obama reached out to America’s enemies, offering in his first inaugural address to extend a hand if you are willing to unclench your fist. More than six years later, he has arrived at a moment of truth in testing that The framework nuclear agreement he reached with Iran on Thursday did not provide the definitive answer to whether Mr. Obama’s audacious gamble will pay off. The fist Iran has shaken at the so-called Great Satan since 1979 has not completely relaxed. paragraph vector Normal skipgram extends C words before, and C words after. IN OUT OUT
  109. 109. On the day he took office, President Obama reached out to America’s enemies, offering in his first inaugural address to extend a hand if you are willing to unclench your fist. More than six years later, he has arrived at a moment of truth in testing that The framework nuclear agreement he reached with Iran on Thursday did not provide the definitive answer to whether Mr. Obama’s audacious gamble will pay off. The fist Iran has shaken at the so-called Great Satan since 1979 has not completely relaxed. paragraph vector A document vector simply extends the context to the whole document. IN OUT OUT OUT OUTdoc_1347
  110. 110. from gensim.models import Doc2Vec fn = “item_document_vectors” model = Doc2Vec.load(fn) model.most_similar('pregnant') matches = list(filter(lambda x: 'SENT_' in x[0], matches)) # ['...I am currently 23 weeks pregnant...', # '...I'm now 10 weeks pregnant...', # '...not showing too much yet...', # '...15 weeks now. Baby bump...', # '...6 weeks post partum!...', # '...12 weeks postpartum and am nursing...', # '...I have my baby shower that...', # '...am still breastfeeding...', # '...I would love an outfit for a baby shower...'] sentence search
  111. 111. translation (using just a rotation matrix) M ikolov 2013 English Spanish Matrix Rotation
  112. 112. context dependent Levy & G oldberg 2014 Australian scientist discovers star with telescope context +/- 2 words
  113. 113. context dependent context Australian scientist discovers star with telescope Levy & G oldberg 2014
  114. 114. context dependent context Australian scientist discovers star with telescope context Levy & G oldberg 2014
  115. 115. context dependent context BoW DEPS topically-similar vs ‘functionally’ similar Levy & G oldberg 2014
  116. 116. context dependent context Levy & G oldberg 2014 Also show that SGNS is simply factorizing: w * c = PMI(w, c) - log k This is completely amazing! Intuition: positive associations (canada, snow) stronger in humans than negative associations (what is the opposite of Canada?)
  117. 117. deepwalk Perozzi etal2014 learn word vectors from sentences “The fox jumped over the lazy dog” vOUT vOUT vOUT vOUT vOUTvOUT ‘words’ are graph vertices ‘sentences’ are random walks on the graph word2vec
  118. 118. Playlists at Spotify context sequence learning ‘words’ are songs ‘sentences’ are playlists
  119. 119. Playlists at Spotify context Erik Bernhardsson Great performance on ‘related artists’
  120. 120. Fixes at Stitch Fix sequence learning Let’s try: ‘words’ are styles ‘sentences’ are fixes
  121. 121. Fixes at Stitch Fix context Learn similarity between styles because they co-occur Learn ‘coherent’ styles sequence learning
  122. 122. Fixes at Stitch Fix? context sequence learning Got lots of structure!
  123. 123. Fixes at Stitch Fix? context sequence learning
  124. 124. Fixes at Stitch Fix? context sequence learning Nearby regions are consistent ‘closets’
  125. 125. A specific lda2vec model Our text blob is a comment that comes from a region_id and a style_id
  126. 126. lda2vec Let’s make vDOC into a mixture… vDOC = 10% religion + 89% politics +… topic 2 = “politics” Milosevic absentee Indonesia Lebanese Isrealis Karadzic topic 1 = “religion” Trinitarian baptismal Pentecostals bede schismatics excommunication

×