38. How does LSTM work?
1. LSTM replaces logistic or tanh hidden units with “memory cells” that
can store an analog value.
2. Each memory cell has its own input and output gates that control.
3. There is a forget gate which the analog value stored in the memory ce
decays.
4. For periods when the input and output gates are off and the forget gate is
not causing decay, a memory cell simply holds its value over time.
Le, Jaitly, & Hinton (2015)
39. 別モデル GRU An alternative of the LSTM
h
~h
x
y
r: reset
gate
input
output
uupdate
gate
ut
= s (Wu
+ uu
)
ht
= f(Wh
+ uh
(ut
@ )
rt
= s (Wr
+ ur
ht-1
)
tilde(h) = (1- rt
) ht
+ rt
tilde(ht-1
)
yt
= Wy
tilde(ht
)
40. 別モデル GRU An alternative of the LSTM
h
~h
x
y
r: reset
gate
input
output
uupdate
gate
ut = σ (Wuxt + Uuht−1) .
ht = ϕ (Wxt + Uh (ut ⊙ht−1)) ,
rt = σ (Wr xt + Urht−1) ,
˜ht = (1 − rt) ht + rt
˜ht−1,
yt = Wy
˜ht
44. Pascanu (2014) より
y( t )
h( t )h( t − 1)
x( t )
y( t )
h( t )h( t − 1)
x( t )
y( t )
h( t )h( t − 1)
x( t )
( a ) ( b ) ( c )
y( t )
h( t )h( t − 1)
x( t )
y( t )
h( t − 1)
x( t ) z( t )
z( t )h( t )
( d ) ( e )
図 4.27 パスカヌらの文献 108) の図 2 を改変
45. Pascanu (2014)より
I * xi
m
h
m ′1h ′1
m ′2
h ′2
m 1h1
m 2
h2
m ′
h ′
h ′
2 次元格子状 LSTM
ブロック
標準の LSTM
ブロック
1 次元格子状 LSTM
ブロック
3 次元格子状 LSTM
ブロック