Invited talk by Antti Oulasvirta / Aalto University, given for the Interacting Minds Center at Aarhus University. October 2017

1. Inverse Modeling for
Cognitive Science
”In the Wild”
October 10, 2017
Antti Oulasvirta
Aalto University
work with Antti Kangasrääsiö, Andrew Howes,
Jukka Corander, Samuel Kaski, Byungjoo
Lee, Kashyap Todi, Jussi Jokinen
Invited talk given for the Interacting Minds Center
at Aarhus University, Denmark

2. It has become easy to collect
(lots of) data about people.
But it is hard to explain the
what, how, and why

3. Sherlock Holmes

4. Sherlock’s problem: Inverse modeling
How to estimate (theoretically plausible) model
parameters without intervention and from limited,
noisy, naturalistic observational data?
• Complicated by the strategic flexibility and
idiosyncratic properties of the human. Any behavior
can be produced by numerous cognitions
Absence of rigorous methodology hampers theory-
formation
12.3.2018
4

5. Behavioral & cognitive sciences
meet
machine learning…

6. In this talk
Theorizing in Cognitive Science and HCI: A Crisis?
Inverse Modeling
Examples
ABC: Rigorous Inverse Modeling Methodology

7. Crisis?

8. Scientific theory-formation and modeling
http://www.jfsowa.com/figs/mthworld.gif
“A scientific model seeks to represent empirical objects, phenomena, and physical
processes in a logical and objective way. All models are in simulacra, that is, simplified
reflections of reality that, despite being approximations, can be extremely useful.”

9. A wealth of cognitive models
Symbolic models
E.g., ACT-R, EPIC, GOMS
Neural models
E.g., perceptron, HNNs
Bounded rationality
E.g., Information foraging theory
Chronometric models
E.g., drift diffusion models
HVS models
E.g., saliency models

10. “Psychology as the science of design”
CogTool
12.3.2018
10

11. Ulric Neisser 1978: Ecological validity
crisis
“If X is an interesting or
socially important aspect of
memory, then psychologists
have hardly ever studied X.”

12. John Carroll: Artefact as theory-nexus
“Other theorists hold that pursuing the
goal of developing cognitive science
theories of HCI may impair progress
toward usefully understanding HCI
phenomena and effectively
contributing to design. This approach
stresses the distortion and
oversimplification inherent in
laboratory-bound psychology and in
conventional views of theory-based
design."
12.3.2018
12

14. Yvonne Rogers: “In the Wild Theories”
“Likewise, it has proven
difficult to say with any
confidence the extent to which
a system or particular interface
function can be mapped back
to a theory. Typically, theories
end up as high-level design
implications, guidelines, or
principles in interaction
design"

15. We need rethink how we
construct models that are
both theoretically
plausible and fit data well
12.3.2018
15

16. Inverse modeling

17. General steps in modeling
1. Model design
• How to formulate the model?
2. Study design
• How to obtain data to evaluate models?
3. Parameter inference
• Which parameter values are probable for the observations?
4. Model evaluation
• How well is the model able to reproduce observations?
5. Model selection
• Which model is the most probable explanation for the data?

18. Forward vs. inverse modeling
From model to data (forward) -- from data to model (inverse)
12.3.2018
18

19. Almost nobody talks about
inverse modeling….

20. Inverse modeling: the problem
• Model fit is determined jointly by the adequacy of the model
structure, and by the values of the model variables.
• Some of the variable values have been decided prior to empirical
observations, and have their place as theoretical postulates.
• Others, called free parameters, are determined from new
empirical data, and these parameters can take almost any value,
ranging from theoretically justified case-specific values to values
that cannot be justified theoretically.
• How to determine these values?

21. Model parameters: Example
Layout learning
[Jokinen et al. CHI2017]
12.3.2018
21
evisual search model predictsvisual search times for new and changed layouts. For a noviceuser without any prior exposureto thelayout,
edicts that of the three elements chosen for this comparison, the salient green element is the fastest to find. After learning the locations of
the expert model finds all fairly quickly. At this point, one blue element and the green element change place. Search times for the moved
arelonger than for thegreen element, becausethe model remembers thedistinctivefeaturesof thelatter.
Figure 2. On the basis of expected utility, the controller requests atten-
tion deployment to a new visual element from theeye-movement system.
This directs attention to the most salient unattended visible object and
results in its encoding. If locational or feature information is accessi-
ble in the LTM, the controller, learning the utilities of its actions, can
optionally also request these features to be considered in the attention
deployment. Encoded objects are stored in VSTM, which inhibits revis-
its. Location and visual features of the elements are stored in LTM for
Encoding an object allows t
the target or adistractor. Bef
jects, it needs to attend one.
holds a visual representatio
controller’srequest it resolv
to oneof theobjectsin it. Th
the properties of thevisual o
in the visual representation
feature isvisually represente
ae
where eistheeccentricity o
the size) and a and b are fre
visual featurein question. Th
a = 0.104 and b = 0.85 for
and 0.142 and 0.96 for size
On thebasis of therepresen
given a total activation as a
top-down activations. Botto
an object, calculated as the d
other objects of the environm
of the linear distance d betw
BAi =
objects
Â
j
f eatu
Â
k
Two objects are dissimilar f
shared exactly between them
tion. Hence, bottom-up activ

22. Parameters
12.3.2018
22

23. Example: Text entry model
[Jokinen et al. submitted]
12.3.2018
23

24. Inverse modeling 101
Finding best parameters for a model
12.3.2018
24

25. Some methods
1. Adopt values from literature
2. OLS (Ordinary least squares) for regression models
3. Manual tuning
4. Grid search
12.3.2018
25
Laborious
Unsystematic
Prone to error
Prone to bias
Prone to cheating
Almost never
done

26. We lack appropriate inverse
modeling methods for generative
models
12.3.2018
26

27. Long-term goal for inverse modeling in
cognitive sciences and HCI
Given behavioral data, infer:
• capacities and traits like visual acuity, working memory capacity,
personality types…
• motivations, or goals, interests and preferences
• beliefs: mental representations
• behavioral strategies: individual and task-specific ways of acting
• …
12.3.2018
27

28. Forward modeling problem
“Define model design MA that explains interesting behaviors.”
Data spaceModel space
MA

29. Inverse modeling problem
“Find parametrization θ for MA that maximizes model fit.”
Data spaceModel space
MA(●)
BA
θ

30. The essence of theorizing
“Define a model for which parametrizations exist that yield valid
predictions for a large set of interesting observations.”
Data spaceModel space
MA(●)
BA
∃θ ⊂ Θ

31. In many areas of engineering and
natural sciences, inverse
modeling is integral to theory-
formation
12.3.2018
31

32. Example: Climate models
12.3.2018
32
http://ies-webarchive-ext.jrc.it/ies/uploads/images/our%20activities/inverse%20modelling_1.jpg

33. Example: Reservoir modeling
12.3.2018
33
https://media.licdn.com/mpr/mpr/AAEAAQAAAAAAAANFAAAAJDU2Z
Dc3Mzg4LWQ4ODctNDE3MS04YjY3LTgyZmMzZjJmMmVmNw.jpg

34. Examples: Inverse
modeling in HCI

35. Inverse modeling in HCI
From user data to a simulator model
12.3.2018
35

36. Inferring neuromuscular noise to
optimize CD gain
12.3.2018
36
Kalman filter
&
submovement
efficiency model
Lee et al. arXiv 2017

37. Biomechanical simulation
Motion data
 Inverse
kinematics
 Inverse
dynamics
 Static
optimization
 Muscle
activations
 Fatigue
12.3.2018
37Bachynskyi et al. CHI 2106

38. “Remulation”
Inverse modeling may be avoided if a person’s history is fully
observable to a cognitive model that can “remulate” it
12.3.2018
38Todi et al. IUI 2018

39. “Familiarisation”1.Most-Encountered 2.Serial Position Curve 3.Visual Statistical Learning
History
earning 4.GenerativeModel of Positional Learning
Original
Todi et al. IUI 2018

40. In most cases we do not have
access to full histories of
people

41. Field data make inverse
modeling even worse…

42. Inverse modeling
Field data insist on a larger number of free parameters
As the number of free parameters increases, model
identifiability decreases
Number of
free parameters
Parameter independence
Unidentifiable
Identifiable

43. For ”in the wild” cogsci,
we need more powerful
inverse modeling methods

44. Approximate
Bayesian
Computation (ABC)

45.  A handy review

47. ABC is a principled way to find optimal
model parameters
Figure 1. This paper studies methodology for inference of parameter
values of cognitive models from observational data in HCI. At the bot-
tom of the figure, we have behavioral data (orange histograms), such as
task solution, only the objecti
straints of thesituation, weca
theoptimal behavior policy. H
that isinferring theconstraints
optimal, isexceedingly difficu
quality and granularity of pre
this inversereinforcement lear
to beunreasonable when often
data exists, such as isoften the
Our application case is a rece
[13]. The model studied here
tation of search behavior, and
completion times, in varioussi
parametric assumptions about
visual system (e.g., fixation dur

48. Approximate Bayesian Inference
1. Inverse modeling with black-box models
2. Works with likelihood-free models (simulators)
3. Noise-tolerant, sample-efficient
4. A global method (cf. local method)
5. Computes a posterior distribution for parameter space

49. How ABC works
1. Choose parameter values for the model
2. Simulate predictions
3. Evaluate discrepancy between predictions and observations
4. Use a probabilistic model to estimate discrepancy in
different regions of parameter space
5. (Repeat until converged)
12.3.2018
49

50. How ABC works
12.3.2018
50
Approximate Bayesian Computation (ABC)

51. How ABC works
12.3.2018
51
Approximate Bayesian Computation (ABC)

52. How ABC works
12.3.2018
52
Approximate Bayesian Computation (ABC)

53. How ABC works
12.3.2018
53
Approximate Bayesian Computation (ABC)

54. How ABC works
12.3.2018
54
Approximate Bayesian Computation (ABC)

55. How ABC works
12.3.2018
55
Approximate Bayesian Computation (ABC)
Indicates most likely value and uncertainty

56. Uses of ABC
Optimal selection and calibration of model for data
1. Model selection (trying out different models)
2. Parameter inference (choosing best parameters)
3. Posterior inference (understanding the space of plausible
explanations)
12.3.2018
56

57. Results in
“Inverse Computational
Rationality”
Towards “Cognitive science in the wild”

58. 12.3.2018
58

59. Inverse modeling of human data is hard
Multiple explanations to any observation
• Different observations can be produced by same mechanism
Stochasticity
Sparse data
Large individual and contextual variability
12.3.2018
59Kangasrääsiö et al. CHI 2107

60. “Bounded agents”
Behavior manifests optimal adaptation to bounds

61. People as agents:
Markov Decision Process
12.3.2018
61

62. Computational rationality: assumptions
• Assume that users behave (approximately) to maximize utility
given limits on their own capacity
• People are “Bounded agents”
• Optimality determined by (1) the environment; (2) goals; and (3)
the user’s cognitive and perceptual capabilities
• Optimal behavioral strategies can be estimated using
reinforcement learning
• No need for hard-wiring task procedures (cf. “old cognitive
models”)
12.3.2018
62

63. Model of menu search
[Chen et al. CHI’15]
Finds optimal gaze
pattern given menu
design and
parameters of the
visual and cognitive
system
12.3.2018
63

64. Case: Menu interaction
12.3.2018
64
Given click times only, predict parameters of HVS
Kangasrääsiö et al. CHI 2107
Click times

65. ABC improves fit over manual tuning
12.3.2018
65Kangasrääsiö et al. CHI 2107
Mean TCT 0.92s
Mean TCT 1.49 s
Mean TCT 0.93 s

66. ABC allows comparison and
exploration of model variants
12.3.2018
66Kangasrääsiö et al. CHI 2107

67. Posterior estimation
ABC yields a posterior distribution for the parameters
12.3.2018
67

68. ABC increases model fit to individuals
12.3.2018
68Kangasrääsiö et al. CHI 2107

69. Explaining individual differences
12.3.2018
69

70. Conclusion
Machine learning for theorizing in cognitive science

71. “Algorithmic Sherlock Holmes”

72. Lots of potential for cognitive and
behavioral sciences to infer…
1. Cognitive capabilities like working memory capacity
2. Belief system (e.g., associative memory structures)
3. Interests, goals, preferences
4. Personality and other more stable traits
5. Cultural differences
6. Situational differences (tasks, context, …)
7. …
12.3.2018
72

73. Issues pointed out to me
Computational costs; convergence; large number of parameters
 Truly uncontrolled behavior remains out of reach?
The psychologist’s fallacy
Limits of mathematical description of human mind

74. ELFI package
Needed
1. a model with tunable
parameters
2. Prior knowledge of
reasonable parameter
ranges
3. Observation dataset
4. Discrepancy measure
http://elfi.readthedocs.io/en/latest/

75. From observed behavior
to data-generating models
October 10, 2017
Antti Oulasvirta
Aalto University
work with Antti Kangasrääsiö, Andrew Howes,
Jukka Corander, Samuel Kaski, Byungjoo
Lee, Kashyap Todi, Jussi Jokinen