【DL輪読会】Mastering Diverse Domains through World Models
•
0 j'aime•755 vues
2023/1/13 Deep Learning JP http://deeplearning.jp/seminar-2/
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Plus de Deep Learning JP
Plus de Deep Learning JP (20)
Dernier
Dernier (20)
【DL輪読会】Mastering Diverse Domains through World Models
- 1. Mastering Diverse Domains through World Models Shohei Taniguchi, Matsuo Lab
- 2. ॻࢽใ Mastering Diverse Domains through World Models • ஶऀ • Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, Timothy Lillicrap • ֓ཁ • ੈքϞσϧΛͬͨڧԽֶशख๏Dreamerͷվળ൛ (ver. 3) • εΫϥονͷڧԽֶशͰॳΊͯMinecraftͰμΠϠϞϯυΛͱΔ͜ͱʹޭ https://arxiv.org/abs/2301.04104 2
- 3. Minecraft ObtainDiamond • MinecraftͰμΠϠϞϯυΛͱΔλεΫ • ใुɼதؒΞΠςϜ͔μΠϠΛͱͬͨͱ͖ͷΈಘΒΕΔ • NeurIPSͰ2019͔Βίϯϖ͕ߦΘΕ͓ͯΓɼRLڀݚͷ1ͭϚΠϧετʔϯ • ͜Ε·ͰεΫϥονͷRLͰμΠϠ֫ಘ·Ͱ ޭͨ͠ྫͳ͠ • ਓؒͷσϞΛ͏ख๏Ͱͷޭྫ͋Γ
- 4. ൃද֓ཁ • લఏࣝ • ੈքϞσϧ x ڧԽֶश • PlaNet, Dreamer, DreamerV2 • DreamerV3 • ·ͱΊ εϥΠυͷҰ෦ΛҎԼ͔Βྲྀ༻͍ͯ͠·͢ https://www.slideshare.net/ShoheiTaniguchi2/ss-238325780 4
- 7. ੈքϞσϧ x ڧԽֶश ֶशͷྲྀΕ 1. ํࡦ Ͱ͔ڥΒσʔλ ΛूΊΔ 2. Λ༻͍ͯੈքϞσϧ Λֶश 3. ੈքϞσϧΛ༻͍ͯํࡦ Λߋ৽ • 1 ~ 3Λ܁Γฦ͢ π D D = {x1, a1, r1, …, xT, aT, rT} D pψ pψ (x1:T, r1:T ∣ a1:T) π https://arxiv.org/abs/1903.00374 7
- 8. World Models [Ha and Schmidhuber,2018] • ੈքϞσϧܥͷڀݚͷΓͱ͍͑Δจ • ੈքϞσϧͷֶशɿVAE + MDN-RNN • ํࡦͷֶशɿCMA-ES • ࠓճৄ͍͠༰ׂѪ͠·͢ ʢҎԼͷεϥΠυͳͲΛࢀরʣ https://www.slideshare.net/masa_s/ss-97848402 https://worldmodels.github.io/ https://arxiv.org/abs/1803.10122 8
- 9. PlaNet [Hafner,et al.,2019] • ੈքϞσϧͷֶशɿ • Recurrent State Space Model • ํࡦͷֶशɿCEM • ϞσϧϑϦʔͱ΄΅ಉͷੑೳ ্ɿ࣮ͰڥͷϩʔϧΞτ ԼɿੈքϞσϧʹΑΔγϛϡϨʔγϣϯ DM Control SuiteͰͷ࣮݁ݧՌ https://arxiv.org/abs/1811.04551 https://planetrl.github.io/ 9
- 10. Ψεܕঢ়ଶۭؒϞσϧ Gaussian State Space Model • ঢ়ଶભҠ֬ʹਖ਼نΛ͏Ϟσϧ • • ؔ ʹDNNͳͲΛ༻͍Δ • ͜Εͩͱ࣮ݧతʹ͏·͍͔͘ͳ͍ʢޯফࣦͳͲʣ pψ (st+1 ∣ st, at) = Normal (μψ (st, at), diag (σ2 ψ (st, at))) μψ, σ2 ψ ot at rt st ot+1 at+1 rt+1 st+1 10
- 11. ࠶ؼతঢ়ଶۭؒϞσϧ Recurrent State Space Model (RSSM) • ঢ়ଶ ΛܾఆతʹભҠ͢Δ ͱ ֬తʹભҠ͢Δ ʹ͚ͯϞσϧԽ͢Δ • LSTMͳͲͷRNNܕͷؔ s h z ht+1 = fψ (ht, st, at) pψ (st ∣ ht) = Normal (μψ (ht), diag (σ2 ψ (ht))) fψ xt at rt st xt+1 at+1 rt+1 st+1 ht ht+1 11
- 12. RSSMΛ͏ͱ͔ͳΓੑೳ্͕͕Δ ࠶ؼతঢ়ଶۭؒϞσϧ Recurrent State Space Model (RSSM) 12
- 13. Dreamer [Hafner,et al.,2019] • PlaNetΛϕʔεʹͯ͠ɺ ํࡦͷֶशΛActor-Criticʹܕมߋ • Ձؔʹ ऩӹΛ༻͍Δ • PlaNet͔Βੑೳ͕େ෯ʹվળ λ https://arxiv.org/abs/1912.01603 https://ai.googleblog.com/2020/03/introducing-dreamer-scalable.html 13
- 14. Ձؔͷਪఆ ϕϧϚϯํఔࣜ εςοϓʹ֦ு͢Δͱ Vπ (st) = 𝔼 π [r (st, at)] + Vπ (st+1) n Vπ n (st) = 𝔼 π [ n−1 ∑ k=1 r (st+k, at+k) ] + Vπ (st+n) 14
- 15. Ձؔͷਪఆ ͰࢦฏۉΛͱΔͱ ͜ΕΛ ऩӹͱͿݺ Vπ n (st) = 𝔼 π [ n−1 ∑ k=1 r (st+k, at+k) ] + Vπ (st+n) n = 1,…, ∞ V̄π (st, λ) = (1 − λ) ∞ ∑ n=1 λn−1 Vπ n (st) λ 15
- 16. Ձؔͷਪఆ DreamerͰɺ ऩӹΛՁؔͷλʔήοτͱ͢Δ ͨͩ͠ɺࢦฏۉͷదͳେ͖͞ʢ ͱ͢ΔʣͰଧͪΔ λ θ ← θ − ηθ ∇θ 𝔼 pψ,πϕ [ V πϕ θ (st) − V̄π (st, λ) 2] H V̄π (st, λ) ≈ (1 − λ) H−1 ∑ n=1 λn−1 Vπ n (st) + λH−1 Vπ H (st) 16
- 18. DreamerV2 [Hafner,et al.,2020] Dreamerͷվྑ൛ 1. જࡏมʹࢄͳΧςΰϦΧϧΛ͏ 2. Τϯίʔμ͕աʹਖ਼ଇԽ͞Εͳ͍Α͏ʹ KL߲ͷֶशΛௐ͢Δ • AtariͰਓؒϨϕϧͷੑೳΛୡ 18
- 20. ࢄજࡏม • ࢄʹͨ͜͠ͱͰɼޯͷਪఆʹreparameterization trick͑ͳ͘ͳΔ • ΘΓʹstraight-through estimatorͰਪఆ • ਪఆྔʹόΠΞε͕Δ͕ɼ࣮͕؆୯ 20
- 21. KL Balancing • ੈքϞσϧͷϩεʹ͓͍ͯɼKL߲encoderͱભҠϞσϧͷpriorΛ͚ۙͮΔ ਖ਼ଇԽͷׂΛ͢Δ • ͔͠͠ɼಛʹֶशॳʹظભҠϞσϧ͕ेʹֶशͰ͖͍ͯͳ͍ঢ়ଶͩͱ ͜ͷKLਖ਼ଇԽ͕ͳ͘ڧΓֶ͗ͯ͢शͷ͛ʹͳΔ 21
- 22. KL Balancing • EncoderͱભҠϞσϧͷKL߲ʹ͍ͭͯͷֶशΛௐ͢Δ͜ͱͰܰݮ • 0.8ʹઃఆ α 22
- 23. ࣮ݧ • AtariͰਓؒ͑ • ϞσϧϑϦʔͷDQN, RainbowͳͲΑΓ͍ڧ 23
- 25. DreamerV3 25
- 26. DreamerV3 • DreamerV2ΛΑΓ൚༻తʹ͑Δख๏ʹ͢ΔͨΊʹ͍͔ͭ͘ΛՃ • υϝΠϯ͕มΘͬͯৗʹಉ͡ϋΠύϥͰֶशͰ͖ΔΑ͏ʹ 1. ؍ଌใुͷΛsymlogؔͰม͢Δ 2. ActorͷతؔͰ ऩӹͷΛਖ਼نԽ͢Δ λ 26
- 27. Symlog Prediction • υϝΠϯ͕มΘΔͱɼ؍ଌใुͷͷεέʔϧ͕มΘΔͷͰɼ ஞҰϋΠύϥΛௐ͢Δඞཁ͕͋Δ • ͦΕΛ͠ͳ͍͍ͯ͘Α͏ʹɼsymlogؔΛ͔͚Δ͜ͱͰΛ͋Δఔἧ͑Δ • ՄͳؔͳٯͷͰɼؔٯΛ͔͚ΕݩͷʹͤΔ 27