Wayne gray presentation

Asymptotes, Plateaus, and Limits to Human Performance
Wayne D. Gray
Cognitive Science Department, Rensselaer Polytechnic Institute
Presentation to the IBM Cognitive Systems Institute – 2015.Feb.05
Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 1 / 30

THANKS TO THESE SUPPORTERS!

The Cognitive Science of The Little Engine that Could
A view of human systems (aka people) as attempting to optimize
performance under constraints. Where these constraints come from:
Bounds on our innate cognitive capacities.
Limits to our acquired skill and knowledge.
The structure of the external task environment in which we
operate.
The goals we are trying to achieve.
If this human system is chugging along, like the Little Engine that
Could, at some given performance ceiling – we can then ask whether
this limit this due:
to external factors that can be altered,
to internal factors that can be altered, or
to factors that cannot be altered.

PERFORMANCE GENERALLY IMPROVES WITH
PRACTICE. . . BUT WHY?
W. L. Bryan
With practice, performance whether it is with
telegraphy, typing, software programs,
arithmetic, programming, mnemonics, or
video games generally improves.
But experts are not simply faster than
novices; rather, they develop a hierarchy of
habits than enable them to “step leagues”
while novices are “bustling over furlongs or
inches”.
CITATION
Bryan, W. L. & Harter, N. (1897). Studies in the physiology and psychology of
the telegraphic language. Psychological Review, 4(1), 27–53
Bryan, W. L. & Harter, N. (1899). Studies on the telegraphic language: the
acquisition of a hierarchy of habits. Psychological Review, 6(4), 345–375

PERFORMANCE GENERALLY IMPROVES WITH
PRACTICE . . . EXCEPT WHEN IT DOESN’T!
Hypothesized that telegraphic expertise consisted of a hierarchy
of habits.
Plateaus were periods in which elements at one level of the
hierarchy were being combined so as to be used at a higher level.
dots and dashes → letters → words → phrases
But . . .

THESE PLATEAUS COULD LAST A LONG LONG TIME
E. L. Thorndike
“I venture to prophesy that the thousand
bookkeepers in, say, the grocery stores of New
York who have each had a thousand hours of
practice at addition, are still, on the average,
adding less than two-thirds as rapidly as they
could, and making twice as many errors as they
would at their limit.”
“It appears likely that the majority of teachers make no gain in
efficiency after their third year of service, but I am confident that
the majority of such teachers could teach very much better than
they do.”
“It seems to me therefore that mental training in schools, in
industry and in morals is characterized, over and over and over
again, by spurious limits – by levels or plateaus of efficiency
which could be surpassed.”
CITATION
Thorndike, E. L. (1913). Educational Psychology Vol II: The Psychology of
Learning. NYC: Teachers College, Columbia University

PLAN FOR THIS TALK
Skip the remainder of the ﬁrst 90 years (1897 – 1987) of scientiﬁc
research on expert performance.
Jump to 1987 to Carroll & Rosson’s Paradox of the Active User and
Ericsson’s (1993) Deliberate Practice and the view shared by both
that expert is not good enough.
Three types of performance asymptotes and one type of plateau.
Summarize everything in time for questions!

Outline
1 Mere Expertise is Not Good Enough
2 Plateaus and Asymptotes
3 Outside the Lab: Good → Better → Best!

OUTSIDE THE LAB – THE PARADOX OF THE ACTIVE
USER
Jack Carroll Mary Beth Rosson
In 1987, Carroll and Rosson coined the term,
Paradox of the Active User, to refer to the
“suboptimal use of oﬃce productivity software”
by people who use the systems daily across the
course of weeks, months, and years.
Carroll and Rosson, who at that time worked for
the IBM Watson Research Center, shared the
HCI community’s concern that the expected
productivity gains of the computer revolution
were not occurring.

MERE EXPERTISE IS NOT GOOD ENOUGH
Anders Ericsson
A few years later, based on his studies of human
expertise, Ericsson (1993) concluded that, “the
belief that a suﬃcient amount of experience or
practice leads to maximal performance appears
incorrect”.
CITATION
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of
deliberate practice in the acquisition of expert performance. Psychological
Review, 100(3), 363–406

LIMITS TO EXPERTISE
After years of lurking in the background, the
plateau had returned to front and center.

RESOLVING THE PARADOX OF THE ACTIVE USER
Suboptimal performance can be amazingly stable! Fu and Gray (2004).
What is optimal?
Is it optimal to learn 100 diﬀerent commands and procedures that
each do one thing very well (fast, precisely, . . . )?
Is it optimal to learn one command and procedure that can be
tweaked into doing 100 diﬀerent things? (slowly, approximately, . . . )
Breaking a habit – the case of hunt & peck versus touchtyping
CITATION
Fu, W.-T. & Gray, W. D. (2004). Resolving the paradox of the active user:
Stable suboptimal performance in interactive tasks. Cognitive Science,
28(6), 901–935

Outline

PLATEAUS VERSUS ASYMPTOTES: POLE VAULTING
A history of technological innovations as the composition of the
pole changed from ash (wood), to bamboo (wood), to
ﬁberglass/carbon.
Each technology, enabled pole vaulters to break new records
Followed by invention of new methods that resulted in new
rounds of record breaking as those methods were adopted and
adapted by athletes.
Asymptote → New technology → New methods → Asymptote

PLATEAUS VERSUS ASYMPTOTES: HIGH JUMPING
A history of innovations in methods.
Method – Fosbury Flop could have been invented earlier.
Plateau → New Method → New Plateau
The Scissors and Straddle technique for high jumping.
The Fosbury Flop technique for high jumping.

PLATEAUS AND ASYMPTOTES
How do plateaus and asymptotes come about??

ASYMPTOTE DUE TO ARTIFACT DESIGN
Crossman’s (1959) study of cigar rollers in Cuba. Plot shows a continued
increase in performance over a two year period (estimated as 3 million
cigars) and then a ﬂattening of the curve.
Newell and Rosenbloom (1981, p. 7) attribute this ﬂattening to a “known
lower bound for the performance time” in this task; namely, the “cycle time
of the machine.”

ASYMPTOTE DUE TO SYSTEM DESIGN
A ﬁeld trial of two workstations for Telephone Operators
(Gray, John, & Atwood, 1993).
Expected call times to decrease across the 4-mon trial.
But after 2-mon worktimes stabilized with times per call
slower than for the old workstation
Slow enough to increase annual operating costs by $6.2
million (in 2014 dollars).
Diagnosis:
Designers believed call time driven by the # of keys-per-call.
Predicted savings of 4.1 s in mean item per call for annual savings of $24m.
BUT cognitive modeling showed that old workstation enabled Operators to
interleave keypresses, chats with customer, and wait time for external
databases.
Conclusion:
Based on the models, Operators were becoming more expert at the new
workstation but asymptotes due to systems design prevented these gains in
expertise from yielding performance increments.

ASYMPTOTE DUE TO MEASUREMENT METHOD
Example from Space Fortress in which a measurement method was introduced
about 1994 that has aﬀected many of the conclusions reached by researchers ever
since.
To simplify the story . . . changed scoring so that it included 4 component scores
and one overall score. These measures can be shown to be (1) not independent of
each other and (2) two of these measures asymptote even as skilled performance
increases.

ASYMPTOTE OR PLATEAU? THE CASE OF DIGIT SPAN
The Digit Span Task – An important part of the Wechsler Adult
Intelligence Scale (WAIS) IQ test (and others).
Digits (0-9) are read at the rate of 1 per sec.
Followed immediately by ordered recall.
If all digits were recalled correctly, the length of the next run of digits
was increased by 1.
If all are not correct, the next run is decreased by 1.
The population norm is 7 ± 2.
Is this an asymptote due to limitations built into the human brain? or
Is this a plateau due to massive stable suboptimal performance on the
part of the entire human population?

ASYMPTOTE OR PLATEAU? THE CASE OF DIGIT SPAN
Well? . . . This is an IQ test item!! Therefore it MUST be measuring an
individual diﬀerence variable that diﬀers between humans but is stable, or
asymptotic, for any given individual.
Right . . . ?

IS HUMAN DIGIT SPAN A POPULATION ASYMPTOTE
OR PLATEAU? – PLATEAU!!
The diﬀerence between a plateau and asymptote is made clear by the
existence of extreme experts with a known history of transcending the
plateau.
DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD
SFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSF
10
20
30
40
50
60
70
80
0 10 20 30 40 50
Practice (in 5 day blocks)
DigitSpan
Perhaps the human asymptote is 80 ± 2 instead. . . ????

Strategy Plateaus and the Paradox of the Active User.
Issue: The difference between a plateau and an asymptote may be
hard to determine.
Asymptotes may reflect a problem that can be fixed by design
(whether artifact design or system design); however, plateaus due to
strategy-induced suboptimality may arise when the strategies deployed
do not enable utility maximization in the task environment.
Overcoming such strategy-induced suboptimality is usually very
difficult.
Example of transfer from visually-guided to touch typing.

Outline

Real World Tasks?

Outside the Lab – Resolving the Paradox of the Active User
If “most people and professionals reach a stable performance
asymptote within a limited time period” (Ericsson, 2004), then
practice does not make perfect.
Unless, as a society, we can be content with “stable suboptimal
performance plateaus” then something more is needed. We
suggest that this “something more” is research into the
acquisition of expertise in mundane (i.e., everyday) task
environments.

Thank You!!

REFERENCES I
Bryan, W. L. & Harter, N. (1897). Studies in the physiology and psychology
of the telegraphic language. Psychological Review, 4(1), 27–53.
Bryan, W. L. & Harter, N. (1899). Studies on the telegraphic language: the
acquisition of a hierarchy of habits. Psychological Review, 6(4),
345–375.
Crossman, E. R. F. W. (1959). A theory of the acquisition of speed-skill.
Ergonomics, 2(2), 153–166.
Ericsson, K. A. (2004). Deliberate practice and the acquisition and
maintenance of expert performance in medicine and related domains.
Academic Medicine, 79(10, S), S70–S81.
Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role of
deliberate practice in the acquisition of expert performance.
Psychological Review, 100(3), 363–406.

REFERENCES II
Fu, W.-T. & Gray, W. D. (2004). Resolving the paradox of the active user:
Stable suboptimal performance in interactive tasks. Cognitive Science,
28(6), 901–935.
Gopher, D., Weil, M., & Bareket, T. (1994). Transfer of skill from a
computer game trainer to ﬂight. Human Factors, 36(3), 387–405.
Gray, W. D., John, B. E., & Atwood, M. E. (1993). Project Ernestine:
Validating a GOMS analysis for predicting and explaining real-world
performance. Human-Computer Interaction, 8(3), 237–309.
Newell, A. & Rosenbloom, P. S. (1981). Mechanisms of skill acquisition and
the law of practice. In J. R. Anderson (Ed.), Cognitive skills and their
acquisition (pp. 1–55). Hillsdale, NJ: Lawrence Erlbaum Associates.
Thorndike, E. L. (1913). Educational Psychology Vol II: The Psychology of
Learning. NYC: Teachers College, Columbia University.

Wayne gray presentation

Recommandé

Recommandé

Contenu connexe

En vedette

En vedette (20)

Similaire à Wayne gray presentation

Similaire à Wayne gray presentation (20)

Plus de diannepatricia

Plus de diannepatricia (20)

Dernier

Dernier (20)

Wayne gray presentation