Synchronous Online Experiments with NodeGame

Synchronous Online
Experiments
nodeGame.org
Stefano Balietti
Microsoft Research
Computational Social
Science Symposium
Cologne
5th November 2018
@balietti
@nodegameorg
stbaliet@microsoft.com

Postdoc Researcher
Microsoft Research - Computational Social Science New York City
Previously
Postdoc Network Science Institute, Northeastern University
Fellow IQSS, Harvard University
PhD Computational Social Science, ETH Zurich
http://stefanobalietti.com
Your Instructor: Stefano Balietti

Main Research Interests:
 Incentives schemes for peer review systems
 Consensus formation and social influence
 Inequality, redistribution, and preferences thereof
 Optimal experimental design
 Philosophy of science, in particular Paul Feyerabend
http://stefanobalietti.com
Your Instructor: Stefano Balietti

Some Stats:
• 5 Undergrad / 3 PhD / 3 Postdoc / 4 Professor / 1 Other
• 8 Female / 8 Male
• 5 Computer Science / 4 Media&Comm. / 3 Sociology / 1 Econ / 1
Political Science / 1 Other
Your Fellow Participants

1. Review fundamental concepts of experimental design
2. Understand challenges of opportunities for online
synchronous experiments
3. Acquire enough knowledge of JavaScript programming
4. Learn the how to design, implement, and run a basic
experiment with nodeGame (and contribute to it?)
5. Learn more advanced nodeGame features OR
implement your own game OR Amazon Mechanical Turk
Goals of the Workshop

1. Review fundamental concepts of experimental design
2. Understand challenges of opportunities for online
synchronous experiments
3. Acquire enough knowledge of JavaScript programming
4. Learn the how to design, implement, and run a basic
experiment with nodeGame (and contribute to it?)
5. EXTRA: advanced nodeGame features, implement own
game, Amazon Mechanical Turk, JavaScript, …
Goals of the Workshop

1. Git Latest https://git-scm.com (Windows users install and use Git Bash)
2. Node.JS 10.x https://nodejs.org/en
3. NodeGame v4.x https://nodegame.org
Additionally,
• Make sure to have a Text Editor with syntax highlighting associated with .js files
• Simple editor: Notepad++ (Windows), TextMate, TextWrangler (Mac)
• More advanced: Atom, Sublime, Emacs, Vim
• Might want to setup a free account on Heroku https://www.heroku.com
• Might want to signup to the nodeGame’s forum
For Hands On: Install

• An experiment is a methodological procedure carried out with the
goal of verifying, falsifying, or establishing the validity of a
hypothesis.
• A test under controlled conditions that is made to demonstrate a
known truth, examine the validity of a hypothesis, or determine
the efficacy of something previously untried.
• An experiment is an empirical method that arbitrates between
competing hypotheses.
What Is an Experiment?

• It is an experiment where participants
• (i) interact in a common environment,
• (ii) at the same time,
• (iii) and their actions have an immediate effect on the decision
process, and experimental outcome (e.g., monetary payoff)
• Synchronous experiments can be turn-based or real-time
What Is a Synchronous Experiment?

The extent to which a causal
inference based on a study is
warranted by an experiment.
Internal Validity

3 Pillars of Experimental Design
Control RandomizationIndependence

Control
Deliberate choice of the variables that will be kept constants
and those that will be varied: treatment variables

Independence
Treatment variables must not be correlated

• Participants are randomly assigned to treatments
• Treatments are executed in controlled order
• Stratification might be required
Randomization

Short History of Experimental Research
“There is a property common to almost all
the moral sciences, and by which they are
distinguished from many of the physical;
that is, that it is seldom in our power to
make experiments with them”
John Stuart Mill, 1836

Friedman, Cassar (2004) Economists go to the laboratory. Who, what, when and why.

“Economics unfortunately cannot perform the controlled
experiments of chemists or biologists because [it] cannot
easily control other important factors. Like astronomers or
meteorologists, [it] generally must be content largely to
observe.”
Samuelson and Nordhaus 1985, Economics Textbook (italics added)
“There is no laboratory in which
economists can test their hypotheses”
1993

Nobel prize for Vernon Smith
“for having established laboratory
experiments as a tool in empirical
economic analysis, especially in the
study of alternative market mechanisms”
2002

Induced Value Theory
“The key idea of induced value theory is that proper use of a reward
medium allows an experimenter to induce pre-specified characteristics,
and the subjects innate characteristics become largely irrelevant. ”
(Vernon Smith, 1976)
Monotonicity: Participants always prefer more of the reward medium (m)
Salience: Changes in m are directly related to a participant’s actions
Dominance: changes in a subject’s utility from participating in the
experiment are primarily due to changes in m.

Induced Value Theory
“The key idea of induced value theory is that proper use of a reward
medium allows an experimenter to induce pre-specified characteristics,
and the subjects innate characteristics become largely irrelevant. ”
(Vernon Smith, 1976)
Monotonicity: Participants always prefer more of the reward medium (m)
Salience: Changes in m are directly related to a participant’s actions
Dominance: changes in a subject’s utility from participating in the
experiment are primarily due to changes in m.
Can you think of examples where failing to induce values can affect experimental results?

Some Limitations of Traditional Experimental Research

Some Limitations of Traditional Experimental Research
• Small groups of individuals, generally 4-12 (bounded by size uni. Lab)
• Behavior of individuals in social networks under-researched
• Limited in the number of hypotheses that can be tested
• Observations are limited in time
• Generally high costs for an academic budget
• Limited external validity:
• Sample WEIRD student
• Artificial conditions

Goals for Present and Future Behavioral Research

• Survey Experiments
• Game-Based Asynchronous Experiments
• Game-Based Synchronous Experiments
Examples of Online Experiments

Survey Experiments
Kuziemko et al. (2015) How Elastic Are Preferences for Redistribution
Insert
a new block of questions, or
provide additional info
Measure preferences for outcome
variable at the individual level
Relatively simple to implement

Game-Based Asynchronous
Experiments
Create a common environment
where a single variable is changed (e.g
network structure)
Observe variation on outcome variable at the
group level
Generally complex design
Can extend for long periods of time Centola (2010) The Spread of Behavior in Online Social Networks

Game-Based Synchronous
Experiments
Create a common environment
where a single variable is changed
(e.g., group size or payoff)
Observe variation on outcome
variable at the group level
Can be really complex or stylized
Mao et al. (2016) An Experimental Study of Team Size and Performance on a Complex Task

COMPLEXITY OF EXECUTION
Survey
Experiments
Game-Based
Asynchronous
Experiments
Game-Based
Synchronous
Experiments

Challenges in Online Experiments
• Recruitment
• Payment
• Anonymity
Dropouts
• Data Quality
• Farming
• Cheating
• Technical Barriers for
implementation
• Cross-Device Compatibility
• Accessibility (impaired users)
•Fairness
• Interference

Recruiting Platforms
• Amazon Mechanical Turk
https://www.mturk.com
• TurkPrime
https://www.turkprime.com
• Figure Eight (ex Crowd Flower, ML)
https://www.figure-eight.com
• Prolific.ac
https://www.prolific.ac
• YouGov (country-specific)
https://yougov.co.uk
• PollFish (also built-in surveys)
https://www.pollfish.com
• Psychological Research on the Net
http://psych.hanover.edu/research/exponnet.html
• The Web Experiment List
http://www.wexlist.net
• Online Social Psychology Studies
http://www.socialpsychology.org/expts.ht
m
• Reddit (for bursty access)
https://www.reddit.com/r/WebGames

Citizen Science Initiatives
• Volunteer Science
https://volunteerscience.com
• Science@Home
https://www.scienceathome.org
• Zoouniverse
https://www.zooniverse.org
• Lab in the Wild
https://labinthewild.org
• CitizenLab
https://www.citizenlab.co

Ming et al. (2016) “The Communication Network Within the Crowd”

Challenges in Online Experiments
• The same person uses the same computer to participate repeatedly.
• The same person uses different computers to participate repeatedly.
• Different persons use the same computer to participate.
• Anonymity might increase deviant behavior
• (but get access to specialized populations)

Improve Data Quality and Attention
Content responsive faking:
(a) responses influenced by the item content but (b) are not completely
accurate.
purposeful faking (e.g., malingering in medical surveys)
socially desirable response of both intentional and nonintentional varieties
Content nonresponsivity:
responding without regard to item content.
Large variance estimation (2.5 – 50%; 10.6 for students)
Inattentive only on some questions
Meade, A. W. and S. B. Craig (2012). Identifying Careless Responses in Survey Data. Psychological methods 17(3), 437.

Improve Data Quality and Attention
• Engagement is key
ui design, interactive elements, colors, length, feedback…
• Nonsensical or “bogus” items
“30th February”
• Instructed response items
“To monitor quality, please respond with a two for this item”
• Unlikely sets
“Select all the good qualities you possess”
• Consistency indexes
Evaluating differences among responses to items highly similar in content
• Too quick response times and other obvious patterns in responses
Meade, A. W. and S. B. Craig (2012). Identifying Careless Responses in Survey Data. Psychological methods 17(3), 437.

Commitment Device
• Ask participants if they paid attention to the previous set of
questions
• It prompts participants to pay attention to the subsequent set
of questions
• Its purpose is fulﬁlled regardless of whether the respondents
answer honestly

Commitment Device Example
Before proceeding to the next set of questions, we want to ask for your feedback
about the responses you provided so far. It is vital to our study that we only include
responses from people who devoted their full attention to this study. This will not
aﬀect in any way the payment you will receive for taking this survey. In your honest
opinion, should we use your responses, or should we discard your responses
since you did not devote your full attention to the questions so far?
- Yes, I have devoted full attention to the questions so far and I think you should
use my responses for your study
- No, I have not devoted full attention to the questions so far and I think you should
not use my responses for your study
Alesina, Miano, and Stancheva (2018) “Immigration and Redistribution” NBER 24733

AMT Bot Scare
TurkPrime Blog
https://blog.turkprime.com/after-the-bot-scare-understanding-whats-been-happening-with-data-collection-on-mturk-and-how-to-stop-it
https://blog.turkprime.com/moving-beyond-bots-mturk-as-a-source-of-high-quality-data
https://github.com/MAHDLab/rIP
https://itaysisso.shinyapps.io/Bots
Tools to filter Farmers’ IP addresses
Dennis et al. (2018)
“MTurk Workers' Use of Low-Cost 'Virtual Private Servers' to Circumvent Screening Methods: A Research Note”
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3233954
Ahler et al. (2018)
The Micro-Task Market for “Lemons”: Collecting Data on Amazon’s Mechanical Turk
http://gsood.com/research/papers/turk.pdf

Fairness for Crowd Workers
• IRB, IRB, IRB (Institutional Review Board)
• Crowd worker have flexibility, but almost no
rights
• Low pay, menial tasks, no accrual of sick days,
or contributions to pension, no unions, etc.
http://www.ghostwork.info
http://www.inthecrowd.org Mary Gray & Sid Suri

Dropouts (Attrition)
Arechar et al. (2018) “Conducting interactive experiments online”
“The most severe problem for online interactive studies, and the largest
discrepancy with laboratory experiments, is attrition (participant dropout).”
Major challenge to internal validity, if dropout rates vary with treatment, selection bias
may arise

Dropouts (Attrition)
Arechar et al. (2018) “Conducting interactive experiments online”
Just a little bit
more than
50% of the
groups that
started
finished
without
dropouts
(18%) overall dropout rate
As dropouts are spread across experimental groups, the fraction of
the data set affected by these dropouts is considerably larger.

What Keeps Participants In?
Intrinsic Motivation
Volunteer Science, Galaxy Zoo, Science@Home (Deci, 2005)
Monetary incentives
about 75% of US Turkers reported that MTurk was their primary or secondary
source of income (Paolacci et al, 2009)
Curiosity
Solving and unrelated, but intriguing puzzle (Law et al., 2016)
Personalized Feedback
Lab-in-the-Wild (Ye et al., 2017)

Two Simple Techniques To Limit Dropouts
• Implement warm-up phase (nodeGame levels)
• Implement seriousness checks
Reips (2002) Standards for Internet-Based Experimenting

What to Do When Dropouts Happen?

• Terminate experiment immediately
• Continue with less participants (notify? Does behavior change?)
• Replace with bots (notify? Behavior is different)
• Replace with other humans (difficult to implement technically and humanly)

• Terminate experiment immediately
• Continue with less participants (notify? Does behavior change?)
• Replace with bots (notify? Behavior is different)
• Replace with other humans (difficult to implement technically and humanly)
 Try to design games that are dropout-robust, Make Leaving Harder, Inform
about total time for experiment, Inform about waiting times during the experiment,
Wait for reconnections, Notify

List of Platforms for Online experiments
Group-Behavior
• nodeGame 
https://nodeGame.org
• Wextor (pioneer)
https://www.wextor.eu
• Otree (large base)
https://www.otree.org
• Lioness (new)
https://lioness-lab.org
• Breadboard (networks)
http://breadboard.yale.edu
Individual
• JSPsych (many plugins)
https://www.jspsych.org
• PsiTurk (groups possible)
https://psiturk.org
Other Resources
• Browser Stack
https://www.browserstack.com
• CSS Frameworks
https://getbootstrap.com
• Empirica (very new)
https://empirica.ly
• TurkServer (discontinued)
https://turkserver.readthedocs.io

nodeGame: the Good Parts
• Powerful API to customize experiments
• Access to low-level details
• Integrated JS database
• Fast and highly scalable
• Game Levels
• Modular design (games, widgets, window)
• Well-documented (and active Forum)
• Integration with Amazon Mechanical Turk

Who Develops NodeGame?
Me! (shakty) B. Vartok, L. Miserez, J. Dorrie,
P.Kueng, Ewen Wang, and others.
(thanks!)

It comes from Node.JS, in which it is implemented, particularly
suited for real-time interactive games.
Why Is It Called NodeGame?
Why Node.JS ?
Built on Chrome's V8 JavaScript engine, it uses an event-driven, non-
blocking I/O model that makes it lightweight and efficient.
Node.js' package ecosystem, npm, is the largest ecosystem of open
source libraries in the world.

 JavaScript was developed in May 1995 by
Brendan Eich for Netscape Communications Corp
 Was created in 10 days in order to
accommodate the Navigator 2.0 Beta release
 Initially called Mocha, later renamed LiveScript in
September, and later JavaScript in the same month
JavaScript

 Microsoft introduced JScript as reverse-engineered implementation of
Netscape's JavaScript in 1996 in Internet Explorer 3
 In 1996 Netscape submitted JavaScript to European Computer Manufacturers
Association (ECMA) to create and industry standard
 In 1997 ECMAScript was released
 Between 1997 and 2009 5 standard have been released.
 July 2015 ECMASCRIPT V6 released.
JavaScript

 Node.JS was invented in 2009 by Ryan Dahl and
other developers working at Joyent
 Combination of Google's V8 JavaScript engine, an
event loop, and a low-level I/O API
 npm, the node package manager, in 2011
 Versions: 0.10, 0.12, 4.0 … 10.0!
Node.JS

JavaScript is #1 Language on Github http://githut.info

Hands On 1
Go to Google.com and open the JavaScript console of your browser:
(ctrl+shift+I or Right Click/Inspect Element)

Some methods are immediately available to manipulate the page elements:
Hands On 1
// Fetches the HTML element with given id.
var a = document.getElementById("hplogo");
// Changes one of its attributes.
a.src = "https://upload.wikimedia.org/wikipedia/commons/e/e9/Bing_logo.svg";
// Defines an onclick event-handler.
a.onclick = function() {
// Redirect to a new page.
window.location.href = "http://bing.com";
};

var a = 1; // numeric
var b = 'Hello world!’; // string
var c = { key: 'value’ }; // object
var d = [ 'value1’, “value2”, 3 ]; // array
var e = function(p) { return p+1; }; // function
 Variables are loosely typed
 Variables are scoped within the function in which they are declared
 Functions and array are objects
 All the variables declared above are global
Variable Declaration in JS
• ES6 introduced let const

How to make variables private ?
Declare them inside an own function!
This technique is called: closure
function foo(value) {
var a = value;
}
foo(10);
console.log(a); // undefined

Self-executing anonymous function
(function(value) {
var a = value;
})(10);

(function(value) {
var a = value;
})(10);
What is an advantage of closures?

(function(value) {
var a = value;
})(10);
What is an advantage of closures?
• They do not pollute the
global space
• Needed when you create
functions in a loop

function foo() {
var a = 1;
}
foo();
What happens is we do
not use the var keyword?

function foo() {
var a = 1;
}
foo();
JS will try to access the global variable a

function foo() {
var a = 1;
}
foo();
What if there is no global variable a?

function foo() {
var a = 1;
}
foo();
console.log(a); // undefined 1
What if there is no global variable a?
Variable leaking into the global scope

Avoid Programming Mistakes
There are ways to avoid these types of mistakes in JS
 Use a linting tool: JSHint/JSLint/ESLint/JSCS
https://www.sitepoint.com/comparison-javascript-linting-tools
 JS Strict Mode
“use strict”;

 Open the Node.JS terminal and launch nodeGame
Hands On 2: nodeGame

cd = change directory (cd .. = dir up)
ls = list
node = type to launch Node.JS programs
Hint! Use TAB to autocomplete the name of files

 Take a look at the different components of nodeGame
 Home Page, Game Interface, Requirement Room, Waiting Room, Game,
EndScreen Widget, The Monitor Interface, Get Saved Data

Configuration and game files are separated in different folder
Game’s Anatomy

Configuration and game files are separated in different folder
Game’s Anatomy
• Define treatments
• Define the sequence of stages and steps of the experiment
• Implement the rules of interactions among participants
• General setup
• Define the graphical contents of the game
• Define the rules about how to assign treatments to
game rooms, and participants to treatments

Hands On 3: Folder waitroom/
// Display buttons to play with bots and select treatment
ALLOW_PLAY_WITH_BOTS: true
ALLOW_SELECT_TREATMENT: true
// How many clients must connect before groups are formed
POOL_SIZE: 2
// The size of each group
GROUP_SIZE: 2
Waiting-Room-v4
 Change waiting room options
 Start 2 groups simultaneously with random treatment assignment (hint: POOL_SIZE),
 Remove Select Treatments button

Folder game/
Define the sequence of stages
and steps of the experiment
Define game variables and
group them into treatments

Stages Definition
stager
.next('id_of_stage’)
stager
.step(‘id_of_first_step_within_stage’)
.step(‘id_of_second_step_within_stage’)
stager
.repeat('stage_to_repeat’, 3)
We use the stager API to define the sequence
A sequence contains stages, and stages contain steps
Stager-API-v4

Stages Definition
Game Sequence Code Snippet
stager.stage("instructions")
.step("instructions_1")
.step("instructions_2")
.step("instructions_3");
stager.stage("quiz");
stager.repeat("game", 3)
stager.stage("questionnaire");
stager.extendStage("game", {
steps: [ "offer",
"respond",
"display_results" ]
});
Stager-API-v4

Hands On 4
// Skip stages from the sequence.
stager.skip('precache');
stager.skip('selectLanguage');
stager.skip('quiz');
stager.skip('instructions');
stager.skip('mood');
 Skip some stages of the game sequence (very useful for debugging)

Settings and Treatments Settings-and-Treatments-v4

Multiline Comments
Game variables (these two are game-specific)
Node.JS makes the content of this object available outside of the file

• Variable TIMER is read by nodeGame’s engine
• It defines the max duration (in milliseconds) of each step
• The names of the properties are the names of the steps in
the sequence

• Variable treatments is read by nodeGame’s engine
• It is an object indexed by the names of the treatments
• Each treatment contains all the variables defined outside of
the treatments object, plus what it is defined inside

Hands On 5
// Change the number of repetition to 1.
REPEAT: 1,
• Change the number of repetition of the ultimatum stage

Hands On 5
// Change the number of repetition to 1.
REPEAT: 1,
module.exports = function(stager, settings) {
stager
.next('precache')
.next('selectLanguage')
.next('instructions')
.next('quiz')
.next('mood')
.repeat('ultimatum', settings.REPEAT)
.next('questionnaire')
.next('endgame')
.gameover();

Hands On 6: Create a New Game
cd bin
node nodegame create-game gamename
 Open a terminal and start the utility to create new games
 Choose a name and follow instructions on screen

cd bin
node nodegame create-game gamename
• Personalize the name of the game as you wish

Folder game/client_types/
• Implement the sequence
• The same sequence can be implemented differently,
depending by who is playing or where the code is
going to be executed
Human, web browser
Computer (orchestrator), server

Client Type player.js
 Contains code that will be executed on the client (web browser)
 Has access to all standard JS objects AND nodeGame client API
 However, it is “compiled” on the server, so it also has access to some server
objects

Information Flow for Client Type
• Decide a treatment and
• Load a client type with correct settings
Read
3
1 2
Read
1 23

How To Implement a Sequence
 Use Stager API stager.[method]
 To initialize game
stager.setOnInit(func);
 To add properties to steps and stages
stager.extendStep("stepId", { property: value });
stager.extendStage(…)
 frame  What to show (interface)
 cb  How to handle user clicks
 … and many others

 Implement Game Sequence: change contents of header
Hands On 6

Header
• It is created at initialization
• It stays throughout the whole game

Header Widget
Widget Widget
• It stays throughout the whole game
• Widgets such as timers, stage counters, and done button are generally added here

Frame
• Its content is updated at every step

Hands On 7
module.exports = function(treatmentName, settings, stager, setup, gameRoom) {
stager.setOnInit(function() {
// Initialize the client.
var header, frame;
header = W.generateHeader();
frame = W.generateFrame();
// …
• Anonymous initialization function passed as
parameter: it will be executed on the client
• Variable W (stands for Window) contains
methods to manipulate the interface of the game
• Function setOnInit is executed on the server
• It adds a property to the stager
• At the end, the stager is sent to the client

Hands On 7
• Variable node contains the entire nodeGame client API
• Variable node.game contains all game-related methods and
objects, including sequence, database, player list, etc.
• You can add your own variables to node.game
• Variable node.widgets contains methods to create and manipulate widgets
• Widgets are reusable code portions, usually something visible on the interface
node.game.visualTimer = node.widgets.append('VisualTimer', header);

 Implement Game Sequence: Change contents of frame
 Go inside folder public/
 Copy public/instructions.html into public/instructions_mod.html
 Make a change to the HTML content
Hands On 8 A

 Implement Game Sequence: Change contents of frame
 Rename to instructions_mod.htm
 Extend instruction step to load new page
Hands On 8 B
stager.extendStep('instructions', {
frame: 'instructions_mod.htm'
});
• Method extendStep adds/overwrites a property into a
specific step
• Property frame controls which page is loaded from public/
• Property frame can also be a function or an object
• Property frame could also load a template from views/

 Executed immediately after the frame has fully loaded
 The scope is the main body (not the frame, which must be accessed via W)
1. Bind page element to actions (W.gid, W.setInnerHTML, onclick)
2. Send messages to server or other players (node.say)
3. Handle incoming messages from server or other players (node.on.data)
4. Decide when the step is finished (node.done)
Step Callbacks Step-Callback-Functions-v4
stager.extendStep('instructions', {
cb: function() {
// Do something.
}
});

node.on.data("offer", function(msg) {
// Display offer using W.
// msg.data = 1
W.setInnerHTML("decision",
"The dictator offered: " + msg.data);
// End step.
node.done();
});
var submit = W.getElementById("submitOffer");
var v = W.gid("offer").value; // 1
submit.onclick = function() {
// Send offer to server, it will store it
// and forward it to observer.
node.done({ offer: v });
};
I say 99 for me
and 1 for you.
Dictator Observer

// Send offer to observer.
node.say("offer", "partner_id", v);
};
I say 99 for me
and 1 for you.
Dictator Observer
• This is what server
does, we will see it
later in details
// msg.data = 1
// End step.
node.done();
});

};
I say 99 for me
and 1 for you.
Dictator Observer
• W.gid = W.getElementById
• Returns HTML element from frame
// msg.data = 1
// End step.
node.done();
});

};
I say 99 for me
and 1 for you.
Dictator Observer
• W.gid = W.getElementById
• Returns HTML element from frame
When both players
are DONE the
game advances to
next step
// msg.data = 1
// End step.
node.done();
});

 Modify game step
 Observer replies a message to the split to approve or disapprove (need to change HTML also)
Hands On 9

 Modify game step
 Observer replies a message to the split to approve or disapprove (need to change HTML also)
Hands On 9
<div id="observer" style="display: none">
<h4 id="waitingFor">
Waiting for the decision of the dictator<span id="dots"></span>
</h4>
<span id="decision"></span>
<p style="display: none" id="reply_p">
Tell the dictator what you think
<input type="text" id="observer_reply" />
<button id="submit_reply">Submit</button>
</p>
<br/>
</div>

Hands On 9
Observer
W.setInnerHTML("decision", "The dictator offered: " + msg.data);
// Show hidden part.
W.show("reply_p");
// Add listener that sends the text back to dictator.
W.gid("submit_reply").onclick = function() {
// Say message directly to dictator.
node.say("reply", "dictator_id", W.gid("observer_reply");
// End step.
node.done();
});
});

Hands On 9
Dictator
// We send offer directly to observer.
// End step,
// store offer in server.
node.on.data("reply", function(msg) {
W.writeln("Your partner thinks: " + msg.data || "nothing");
node.timer.randomDone({ offer: v });
});
};

Hands On 9
Dictator
// End step,
});
};
Same labels as in Observer: “offer” and “reply”

Hands On 9
Dictator
// End step,
});
};
Randomly ends step

Hands On 9
Dictator
// End step,
});
};
Randomly ends step
Logical OR, shortcut to assign a default
value, if the first term is falsy

https://dorey.github.io/JavaScript-Equality-Table/
=== (strict equality)== (involves type conversion)

Client Type logic.js
 Contains code that will be executed on the server
 Has access to the nodeGame client API AND to all server objects
 What does it do?
 It waits for all players to finish a step, if there is nothing better to do
 Make sure the timers are respected
 Can push/kick out non-responsive players (Matcher API)
 It receives incoming messages from players and stores them (node.game.db)
 It matches partners, and assigns roles

 Contains code that will be executed on the server
 Has access to the nodeGame client API AND to all server objects
 What does it do?
 It waits for all players to finish a step, if there is nothing better to do
 Make sure the timers are respected
 Can push/kick out non-responsive players (Matcher API)
 It receives incoming messages from players and stores them (node.game.memory)
 It matches partners, and assigns roles

stager.extendStep('bidder', {
matcher: {
// Available roles.
roles: [ 'BIDDER', 'RESPONDENT' ],
// Each player is matched with a
// partner and is assigned a role.
match: 'roundrobin',
// How to continue matching after
// all combinations are exhausted
cycle: 'repeat_invert'
}
});
Logic Client Type
Matching-Roles-Partners-v4

roles: {
BIDDER: {
frame: 'bidder.htm',
cb: function() {
// Bidder stuff.
}
},
RESPONDENT: {
frame: 'resp.htm',
cb: function() {
// Respondent stuff.
}}}
});
Player Client Type
matcher: {
// Available roles.
}
});
Logic Client Type

roles: {
BIDDER: {
frame: 'bidder.htm',
cb: function() {
// Bidder stuff.
}
},
RESPONDENT: {
frame: 'resp.htm',
cb: function() {
// Respondent stuff.
}}}
});
Player Client Type
matcher: {
// Available roles.
}
});
Logic Client Type
Logic lists roles
Player implements

Memory Database
node.game.memory
Memory-Database-v4

Memory Database
node.game.memory
Memory-Database-v4
node.done({ offer: 1 });
{
stage: { stage: 2,
step: 1,
round: 1 },
player: "id12345678",
timestamp: 1480336721939
offer: 1,
timeup: false,
time: 4321, //From step
done: true
};

Memory Database
node.game.memory
Memory-Database-v4
// Set an object in memory explicitly.
node.set({ foo: "bar" });
{
stage: { stage: 2,
step: 2,
round: 1 },
player: "id12345678",
timestamp: 148033845673
foo: "bar" ,
};

Memory Database Memory-Database-v4
var m = node.game.memory;
// Lazy evaluation: add a bunch of statements
// and evaluates all of them at the end.
m.select("player", "=", "id12345678");
m.and("stage.step", "=", 1)
.and("offer") // Must have property “offer”
.fetch()[0]; // Returns array, get first element
// Use indexes.
// By sender.
m.player["id12345678"].last();
// By stage.
m.stage["2.1.1"].select("offer").fetch()[0];

 Explore this RunKit on selecting and fetching items from node.game.memory
 https://runkit.com/shakty/gamedb
Hands On 10

What’s Left Out
 Levels
 Widget Steps
 Authorization
 Get Help / Contribute
 Dropout & Reconnections
 Upload Your Game to the Cloud
 Amazon Mechanical Turk

What More?
 Technical Topics
 Levels
 Widget Steps
 Authorization
 Requirements
 Get Help / Contribute
 Dropout & Reconnections
 Go Live
 Upload Your Game to the Cloud
 Amazon Mechanical Turk

Extend
Sent to client as part nodegame-full.js

Extend
Core components for the client
Sent to client as part nodegame-full.js
After editing a core component, rebuild
nodegame-server/
public/javascripts/nodegame-full.js
node launcher -b widgets,window
(nodegame-client automatically rebuilt with –b)

Extend
SASS CSS framework
https://sass-lang.com/
Do changes to:
public/css/stylesheets/nodegame.scss
node launcher -b css
Builds nodegame.css

AMT
1. Upload/install nodeGame + game to the cloud
e.g., Digital Ocean or Heroku cloud: https://github.com/nodeGame/nodegame-
heroku
2. Create a new HIT on Amazon Mechanical Turk (AMT), e.g. using the requester
interface or other service API.
3. Use the nodegame-mturk package
https://github.com/nodeGame/nodegame-mturk/
to handle:
- Connection and authentication from AMT HIT page to nodeGame server.
- Opening/expiring/extending HIT while running it.
- Payments, bonuses and qualifications after HIT is done.

AMT HIT Page
Check the boilerplate in the public folder of the nodegame-mturk
package

AMT HIT Page
• The important things:
• Inform about the task (in general terms)
• Make it really clear it is a SYNCHRONOUS task
• Explain expected duration and payoff structure
• Explain technical requirements, if any
• Optionally, have a Trouble Ticket
• Exit code input

AMT HIT Page
• Display link to game server only after acceptance
• Get turker information, upon acceptance, from the page and send
it to the server along the query string or via a separate AJAX request
HITID = turkGetParam('hitId’);
WORKER_ID = turkGetParam('workerId’);
ASSIGNMENT_ID = turkGetParam('assignmentId');
• These functions are not
officially supported by
AMT, so they might
disappear in the future
• These info is stored in
the page as hidden input

AMT Issues for Synchronous Games
1. Not a survey or classification task, need to have DIFFERENT
people at the same time on your platform
2. Btw, not too few, not too many
3. Waiting time in the waiting room should be limited and
homogenous across participants
4. Demographics should be stratified as much as possible
5. Participants should not re-take the same experiment again
6. Dropouts rate (non-completing the HIT) significantly affects the
whole experiment, increasing costs $$ and time to complete it

Pay minimum wage (at least). Make bonus dependent on performance

Rule of thumb: 2.8 times desired number of participants
Expire HIT as soon as desired number of participant entered the
experimental pipeline (+small buffer)
3x

Rule of thumb: 2.8 times desired number of participants
Expire HIT as soon as desired number of participant entered the
experimental pipeline (+small buffer)
Beware of “HIT Grabbers”: use scripts to accept HITs very quickly, and
then take it when they have time;
Often, they only see the title, so make sure to write in title something
like: “MUST TAKE IMMEDIATELY”

Dispose participants who waited too long (compensated)

Waiting room can group players based on features
Collect some demographics features in a pre-experimental part (level 1)
Start 2 (or more) groups at the same time

Create one HIT (no group) with 3x rule
Keep track of turker ids (nodegame-mturk)
Assign qualification via AMT API (nodegame-mturk)

6. Dropouts rate (non-completing the HIT) signigicantly affects the
whole experiment, increasing costs $$ and time to complete it
• Hey, we talked about this already! (see previous slides)

AMT Dropouts and Relative Cumulative Payoff
• Tested combinations of
demographics and other
game characteristics
• The only significant difference
is between dropouts and
participants who stay is their
relative payoff.

Keep a Good Requester Score
https://turkerview.com
https://turkopticon.ucsd.edu (will be discontinued)

External Validity
External validity:
The extent to which the results of a study can be generalized
or extended to others.

External Validity
Galileo's hand-made drawing of the moon surface. Sidereus Nuncius (1610)

External Validity
Telescope:
From Land to Sky

External Validity
Telescope:
From Land to Sky
Experiments:
From Lab to Real Life

 To access variables between files must be explicitly exported and required
// File A.
var a = 1;
module.exports = a;
// File B.
var a = require('./A.js')
How to export
multiple variables?
// File A.
module.exports = { a: 1, b: 2 };
// File B.
var a = require('./A.js').a;
Node.JS Export Variables

 JS functions are objects
 Can be passed as parameters to other functions
 Treat differently different input parameters
 Can have properties
 Can be executed with different contexts
 Two types exists: declaration and expressions
 Always remember the scope of creation
JS Functions

JS Functions
Create an array of 10 functions returning the index in which they are
inserted in the array.
Is the code below correct?
var i, len, obj;
len = 10, obj = [];
for (i = 0 ; i < len ; i++) {
obj[i] = function() { return i; }
}

JS Functions
Is the code below correct?
var i, len, obj;
len = 10, obj = [];
for (i = 0 ; i < len ; i++) {
obj[i] = function() { return i; }
}
• Creates 10 functions all
returning the value 10,
because they are all
referencing variable i, which
has value 10 when the loop
ends

JS Functions
You need a closure! for (i = 0 ; i < len ; i++) {
obj[i] = (function(i) {
return function() {
return i;
}
})(i);
}

JS Function Input Parameters
// Internally modifies input.
function a(obj, num, str) {
obj.a = 10;
num = 1;
str = 'a';
}
var obj = {}, num = 0, str = '';
a(obj, num, str);
console.log(obj);
console.log(num);
console.log(str); ?

// Internally modifies input.
function a(obj, num, str) {
obj.a = 10;
num = 1;
str = 'a';
}
var obj = {}, num = 0, str = '';
a(obj, num, str);
console.log(obj); // { a: 10 }
console.log(num); // 0
console.log(str); // ''

console.log(obj); // { a: 10 }
console.log(num); // 0
console.log(str); // ''
 Objects are passed as reference, while numbers and strings as
copy
 Modifying a copy does not affect the value outside the function,
modifying the reference does.

JS this
function a() { return this.a; }
a(); // undefined
// Create a context.
var foo = { a: 1};
// call and apply set the this value
a.call(foo, 1, 2, 3); // 1;
a.apply(foo, [1, 2, 3]); // 1;
 The value of this is dynamic in JavaScript
 It is determined when function is called, not when it is declared.

JS this
What will be printed to console?
function A() {
this.a = 1;
(function() {
console.log(this.a);
})();
}
// Create a new object.
var a = new A();

JS this
function A() {
this.a = 1;
(function() {
})();
}
// Create a new object.
var a = new A();
• It will print undefined
• How to adapt to print 1?

JS this
function A() {
this.a = 1;
var that = this;
(function() {
console.log(that.a);
})();
}
• It will print 1
• The reference to this is
stored in another variable

JS this
function A() {
this.a = 1;
var that = this;
(function() {
})();
}
• It will print 1
Why is “this” solution better than call or apply?

JS this
function A() {
this.a = 1;
var that = this;
(function() {
})();
}
• It will print 1
because you can reuse that multiple times!
Why is “this” solution better than call or apply?

 In JS, each object inherits methods and properties from a parent object
called prototype
 In turn, also the prototype object can have an own prototype, and all the
properties are are inherited through the prototype chain
 It is possible to extend an object by extending its prototype or the
prototype of its prototype...
 This pattern is called prototypical inheritance, and it is extremely
powerful – if well understood
JS Inheritance

function A() {
this.a = 1;
}
A.prototype.printA = function() {
}
var a = new A();
a.printA(); // 1;
JS Prototypical Inheritance

function A() {
this.a = 1;
}
A.prototype.printA = function() {
}
var a = new A();
a.printA(); // 1;
 What is the difference with defining
define the method printA inside the
constructor? (this.printA = function … )

// Create a second object.
var a2 = new A();
// Assign property to method printA.
a2.printA.foo = 1;
// Property is also on object a
// because it is the prototype
// to be modified.
console.log(a.print.foo); // 1

 Extending the prototype of the function leads to faster
object creations because all the methods are already
existing and only need to be referenced instead of being
created
 However sometimes you need to have a clear separation
between methods of objects of the same class

Looping in Objects (For In)
 Javascript does not guarantee clear separation between
variables of the prototype and of the object itself
 Therefore, when looping through the properties of an
object it is necessary to invoke the method
.hasOwnProperty

Looping in Objects (For In)
var triangle = { a: 1, b: 2, c: 3 };
function ColoredTriangle() {
this.color = "red";
}
ColoredTriangle.prototype = triangle;
var obj = new ColoredTriangle();
for (var prop in obj) {
if (obj.hasOwnProperty(prop)) {
console.log(obj[prop]);
}
}

Linting Tool: JSHint
Try it on : http://jshint.com

 Really easy to install and use
 Install with the node package manager
 What is the option -g doing?
Linting Tool: JSHint
 You might be interested in one of the plugins for editors (vim, emacs, sublime...)
jshint your_file.js --show-non-errors
npm install -g jshint

"use strict";
(function(value) {
a = value;
})(10)
a = value;
^
ReferenceError: a is not defined
at /home/stefano/Documents/myworkshops/nodegame_CESS/scripts/A.js:4:7
at Object.<anonymous> (/home/stefano/Documents/myworkshops/nodegame_CESS/scripts/A.js:5:3)
at Module._compile (module.js:460:26)
at Object.Module._extensions..js (module.js:478:10)
at Module.load (module.js:355:32)
at Function.Module._load (module.js:310:12)
at Function.Module.runMain (module.js:501:10)
at startup (node.js:129:16)
at node.js:814:3
Strict Mode on

Synchronous Online Experiments with NodeGame

Recommandé

Recommandé

Contenu connexe

Similaire à Synchronous Online Experiments with NodeGame

Similaire à Synchronous Online Experiments with NodeGame (20)

Dernier

Dernier (20)

Synchronous Online Experiments with NodeGame