EMNLP 2021 proScript (summary slides)

1. proScript:
Partially Ordered Scripts Generation
Keisuke Sakaguchi, Chandra Bhagavatula, Ronan Le Bras,
Niket Tandon, Peter Clark, Yejin Choi
https://proscript.allenai.org/

2. What is Script? Why is it important?
“a script is a stereotyped sequence of actions that defines
a well-known situation and has associated with it”
Roger Schank and Robert Abelson (1977)
2
What is Script? Scenario:
Travel to Hawaii

3. What is Script? Why is it important?
Images from: https://images.app.goo.gl/B24EbZbZ8Xzw7STUA https://images.app.goo.gl/iWRLkmm3TTWGD4jJ7 https://images.app.goo.gl/WDjZtLHW3Gw36eS9A
“a script is a stereotyped sequence of actions that defines
a well-known situation and has associated with it”
Roger Schank and Robert Abelson (1977)
3
✦Script is an essential part commonsense knowledge.
✦Script helps to represent and understand causal structure of events.
✦Script allows inference about implicit cause and effect relationship.
What is Script?
Why is it important?
Scenario:
Travel to Hawaii

4. Research Problem: trade-off between quality and scale 4
Quality Scalability
Induce from texts
(e.g., Chambers and Jurafsky, 2008)
- +
Event sequence alignment
(e.g., Regneri et al., 2010)
+ -
proScript + +

5. Our contributions 5
We collect 6.4k partially ordered scripts, proScript,
which is substantially larger than prior datasets.
With proScript, we introduced two complementary tasks
and models. (edge prediction and script generation)
We show the first time that pre-trained neural LM can be
adapted to generate partial-order scripts.

6. 1. proScript: Crowdsourced 6.4k partial-order scripts 6
Suppose a scenario where someone wants to “travel to Hawaii”.
Q1: Describe 5 to 7 essential steps and each time duration. (Note: the order does not matter.)
decide schedule 1 hour
book a flight
go to airport
Q2. Create a flowchart of the steps
(possibly in partial order, where temporal
ordering is required only when it is necessary.)
30 minutes
1 hour
Collect “scenarios” (e.g., travel to Hawaii, bake a cake) from existing corpora and datasets.
1. DeScript (Wanzare et al., 2016) 2. VirtualHome (Puig et al., 2018) 3. ROCStories (Mostafazadeh et al., 2016)

7. 2. Two complementary tasks and models
1. proScript Edge Prediction
7
find the cake recipe
gather the ingredients
turn on the oven
mix the ingredients
put the cake batter in the oven
bake for the right amount of time
take the cake out of the oven
Scenario: bake a cake
Given: Scenario and randomly shuffled events
2. proScript Generation

8. 2. Two complementary tasks and models
1. proScript Edge Prediction
8
2. proScript Generation
find the cake recipe
gather the ingredients
turn on the oven
mix the ingredients
put the cake batter in the oven
bake for the right amount of time
take the cake out of the oven
Scenario: bake a cake
Given: Scenario and randomly shuffled events Given: Scenario and the number of events (to generate)
Scenario: bake a cake
Number of events: 7

9. 2. Two complementary tasks and models
1. proScript Edge Prediction
9
2. proScript Generation
find the cake recipe
gather the ingredients
turn on the oven
mix the ingredients
put the cake batter in the oven
bake for the right amount of time
take the cake out of the oven
Scenario: bake a cake
Given: Scenario and randomly shuffled events Given: Scenario and the number of events (to generate)
Scenario: bake a cake
Number of events: 7
DAGs are flattened by DOT language

10. 3. Generate partial-order Scripts with neural LM 10
Scenario: play the organ
Scenario: drink a glass of milk
walk to the kitchen
open the refrigerator
remove milk from refrigerator
close the refrigerator
pour milk into pot to warm a bit
pour milk into glass to drink
raise glass to lips
find sheet music to play
sit down at the organ bench set up the sheet
warm up on the organ
play the music on the organ

11. Evaluation: Graph Edit Distance (lower GED, the better) 11
random baseline
proScript generator
Human
0 3 6 9 12
2.78
4.73
11.3
← smaller the better
Human (2.7) < proScript generator (4.7) << Random (11.3)
proScript generator

12. Check out the paper for more details! 12
Pairwise comparison between Human vs. Model,
Qualitative analysis of graph edits,
Results on edge prediction task,
and a lot of other results and analysis…

13. Summary 13
We collect 6.4k partially ordered scripts, proScript,
which is substantially larger than prior datasets.
With proScript, we introduced two complementary tasks
and models. (edge prediction and script generation)
We show the first time that pre-trained neural LM can be
adapted to generate partial-order scripts.
Data is available:
https://proscript.allenai.org/