To press a button, a finger must push down and pull up with the right force and timing. How the motor system succeeds in button-pressing, in spite of neural noise and lacking direct access to the mechanism of the button, is poorly understood. This paper investigates a unifying account based on neurome- chanics. Mechanics is used to model muscles controlling the finger that contacts the button. Neurocognitive principles are used to model how the motor system learns appropriate muscle activations over repeated strokes though relying on degraded sensory feedback. Neuromechanical simulations yield a rich set of predictions for kinematics, dynamics, and user performance and may aid in understanding and improving input devices. We present a computational implementation and evaluate predictions for common button types.
Forensic Biology & Its biological significance.pdf
Neuromechanics of a Button Press: A talk at CHI 2018, April 2018
1. Neuromechanics
Antti Oulasvirta, Sunjun Kim, and Byungjoo Lee bit.do/neuromechanics
of a Button Press
Related papers at CHI 2018:
1. Control-theoretic Models of Pointing Tue 9-11.30 517C
2. Impact Activation Improves Rapid Key Pressing Mon 16.30-17.50 514AB
3. Moving Target Selection: A Cue Integration Model This session
2. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics10x slowdown
Long-term goal:
A theory of input
Design
Feedback
Goals
Anatomy
Skill
Movement
Efficiency
Efficacy
Effort
3. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Existing theories
Information theory
Human performance
Control theory
Cognitive models
Characterize
Intrapolate
Poor transfer
Change in device or task
insists on data collection
or manual task modeling
4. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
The brain’s point-of-view
5. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Design
Anatomy
Goals
Feedback
Kinematics
Dynamics
Precision
Effort
Human-like
responses
Adapt
A generative
approach
6. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
The
physical
The
neural
What is neuromechanics?
Neural principles of motor control in biomechanical systems
The
physiological
[Enoka 2009]
7. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Input is control of mediated sensations
The
brain
Peripheral
nervous system
Sensory
nervous system
Action
potential
Physical
stimulation
Feedforward
Feedback
Limbs
Sensory organs
Device
8. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Probabilistic
internal model
Sensory
signals
Activation
signal
Cue
integration
Integrated
percept
Perceptual
control task
Prediction
error
Prediction
Muscle, bone,
tissue, device,
sensory organs
Overview of the theory
9. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Theory: Elements
Biomechanical simulation
Cue integration
Probabilistic internal model
Perceptual control
[Enoka 2009]
[Sung-Hee Lee 2009]
[Ernst 2004, 2006]
[Powers 1973, 2009]
[Clark 2013]
[Hohwy 2016]
10. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Too brief for closed-loop control
Button mechanism not perceived
Ephemeral sensations
Noisy neuromuscular system
10x slowdown
Precise time and force
Effective skill transfer
Ability to adapt and recover
Button-
pressing:
A miracle
11. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Random
movements are
unsuccessful
12. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Biomechanics
simulation
Noisy muscle
activation signal
Mechanoreceptive
sensor
Proprioceptive
sensor
13. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Silicone
finger tip
Pressure
sensor
Hill type
muscle
Joint angle
sensor
Real button
Robotic
implementation
+ Noise
+ Noise+ Noise
14. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
No motor noise
à super-human performance
15. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Sensory signals
Cue integration
Perceptual center
eeds a threshold value. This
ls: visual and auditory (beep).
o-beep delays are assumed to
utation of p-Centers
m is connected to four extero-
ion, proprioception, audition,
ity i produces a p-center pci.
sfer of a neural signal evoked
hanoreceptors. We are espe-
noreceptors on the finger pad
ming of a button press. Slowly
ensitive to coarse spatial struc-
flat top surface of the button),
ers respond to motion. Kim
in signals from the fingertip
ion, and jerk from the finger
t force and indentation have
d force correlates highly with
use maximum likelihood estimation (MLE) to obtain
estimate of pco. For another implementation of cue inte
see [35]. In MLE, assuming that a single-cue estim
unbiased but corrupted by Gaussian noise, the optimal s
for estimating pco is a weighted average [16, 17]:
pco = Â
i
wi pci where wi =
1/s2
i
Âi 1/s2
i
with wi being the weight given to the ith single-cue es
and s2
i being that estimate’s variance. Figure 6 sho
emplary p-center calculations: signal-specific (pci) an
grated p-centers (pco) from 100 simulated runs of NEU
CHANIC pressing a tactile button. Note that absolute
ences among pci do not affect pco, only signal varian
The integrated timing estimate is robust to long delays
auditory or visual feedback. This assumption is base
study showing that physiological events that take place q
within a few hundred milliseconds, do not tend to be
over- nor underestimations of event durations [14].
Maximum likelihood estimator
“When was the
button activated?”
Modality-specific
noise variances
Proprio-
ceptive
Tactile
Integration is
sensitive to how
reliable the cues are
16. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
No mechanoreceptive
feedback signal
17. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Perceptual control task
Choose (1) a motor command and (2) expected perception that
minimizes this objective:
The
oned
ntin-
prio-
ion),
Gaus-
form
error
tiva-
cuss
esult
ptual
enter
com-
f cue
(GP)
and
e the
with signal offset µ, signal amplitude t, and duration s of the
agonist (A+) muscle. We have set physiologically plausible
extrema (min and max) for the activation parameters. Note
that this formulation assumes that the antagonist muscle re-
sists motion passively. More determinate pull-up motion can
be achieved by adding similar parameters for the antagonist
muscle (A-).
The objective is to determine motor command (q) and as-
sociated estimate of perceived button activation (pce) that
minimize error:
min
q,pce
EP(q, pce)+EA(q)+EC(q)+wFM(q) (2)
where EP is perceptual error, EA is error in activating the
button, and EC is error in making contact (button cap not
touched). FM is muscle force expenditure computed from the
Hill muscle model (see below), and w is a tuning factor. We
assume that activation and contact errors are trivial to perceive.
Therefore, EA and EC are binary: 1 in the case of error and 0
otherwise. Perceptual error EP is defined as distance (in time)
between expected p-center pce and observed p-center pco:
Objective function
Perceptual
error
Activation
failure
(binary)
Contact
failure
(binary)
Muscle
force
User goal:
Lightness
vs.
precision
18. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Perceptual error
Minimize error between expected and perceived activation time
19. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
No button-
activation term
20. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
No muscle
effort term
21. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
A probabilistic internal model
learns motor outputs that minimize the perceptual objective
A Bayesian optimizer with a Gaussian Process prior
Approximate Bayesian Computation (ABC)Approximate Bayesian Computation (ABC) Approximate Bayesian Computat
...
Finds a good button
press after 10-20 trials
22. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Model
parameters
Table 1. Model parameters. Button parameters here given for physical
buttons. Task parameters (e.g., finger starting height) are given in text.
f denotes function
Variable Description Value, Unit Ref.
fr Radius of finger cone 7.0 mm
fw Length of finger 60 mm
rf Density of finger 985 kg/m3
cf Damping of finger pulp 1.5 N·s/m [64]
kf Stiffness of finger pulp f, N/m [65]
wb Width of key cap 14 mm
db Depth of key cap 10 mm
rb Density of key cap 700 kg/m3
cb Damping of button 0.1 N·s/m
ks Elasticity of muscle 0.8·PCSA [38]
kd Elasticity of muscle 0.1·ks [38]
kc Damping of muscle 6 N·s/m [38]
PCSA Phys. cross-sectional area 4 cm2
L0ag, L0an Initial muscle length 300 mm
sn Neuromuscular noise 5·10 2
sm Mechanoreception noise 1·10 8
sp Proprioception noise 8·10 7
sa Sound and audition noise 5·10 4
sv Display and vision noise 2·10 2
system. We have set these parameters manually in order to
reproduce certain basic effects: Neuromuscular noise, which
reflects the joint additive contribution of neural and muscular
Physically
measurable
Tuned based on
literature
23. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Simulation workbench (MATLAB)
bit.do/neuromechanics
But are the
outputs
realistic?
24. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Precision & success: Predictions
Figure 7. Data collection on press kinematics: A single-subject study.
High-fidelity optical motion tracking was used to track a marker on
the finger nail. A custom-made single-button setup was created using
switches and key caps from commercial keyboards.
SIMULATIONS: COMPARING BUTTON DESIGNS
Most precise Less precise Least precise
High success High success Low success
Push-
button
Touch Mid-air
Order supported by literature
25. Data collection: A single-subject study
“Press rhythmically in a
manner natural for you”
26. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Mid-air button: Kinematics
[Torre & Balasubramaniam 2009]
Human Model
Similar to our data and literature
27. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Dynamics
Force-displacement curves
Predicted
peak forces
1.5-2.9N
Force ranges similar
to literature
cle force–displacement behavior for a tactile
with an effort-minimizing term in the objective
task performance (performance in button activation). We co
clude, that although much work remains to be done, the resul
support the ’optimal black box’ assumption. And many mor
analyses could done, such as looking at the effect of unreliab
feedback, oscillation of the finger tip, such as when walkin
or the effects that impairments like essential tremor have.
FUTURE WORK
Modeling latent neural and cognitive constructs, such as nois
poses a scientific challenge for future research. Change i
noise parameters has a large and poorly understood effect o
dynamics downstream. However, without noise, a button ca
be activated with arbitrary precision. For example, cuttin
sensory noise parameters to 10 9 reduces perceptual error t
the order of 1.5·10 6 s. Our noise model was tuned manuall
to reproduce some standard findings on sensory modalities. T
“Light touch”
Force(N)
28. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Touch buttons: Kinematics
Displacement-velocity curves
e 8. Displacement–velocity curves for four button types from single-subject recordings (top) and simulations (bottom
Shape similar except in release
Human Model
Similar result for push-buttons
29. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Signal-dependent
noise
Muscle and
joint models
Force
perception
Noise parameter
identification
Limitations
30. Neuromechanics of a Button Press
Oulasvirta, Kim, & Lee Proc. CHI 2018 bit.do/neuromechanics
Neuro-
science PhysicsPhysiology
Machine
learning
Biomech.
simulation
EE and signal
processing
A unifying account
A generative simulation
Can it be made to work beyond buttons?
31. bit.do/neuromechanics
Antti Oulasvirta, Sunjun Kim, Byungjoo Lee
Acknowledgements: Jong-In Lee, Aleksi
Pesonen, Yunfei Xiu, and Crista Kaukinen
Related papers at CHI 2018:
1. Control-theoretic models of Pointing Tue 9.00 517C
2. Impact Activation Improves Rapid Key Pressing Mon 16.30 514AB
3. Moving Target Selection: A Cue Integration Model This session
Figure 8. Displacement–velocity curves for four button types from single-subject recordings (top) and simulations (bottom).
Data Matlab codeRobot