wayne gray presentation

Asymptotes, Plateaus, and Limits to Human Performance

Wayne D. Gray

Cognitive Science Department, Rensselaer Polytechnic Institute

Presentation to the IBM Cognitive Systems Institute – 2015.Feb.05

Gray, et al. (RPI) Expert is Not Good Enough! 2015.02.05 1 / 30

THANKS TO THESE SUPPORTERS!


The Cognitive Science of The Little Engine that Could


A view of human systems (aka people) as attempting to optimizeperformance under constraints. Where these constraints come from:

Bounds on our innate cognitive capacities.Limits to our acquired skill and knowledge.The structure of the external task environment in which weoperate.The goals we are trying to achieve.

If this human system is chugging along, like the Little Engine thatCould, at some given performance ceiling – we can then ask whetherthis limit this due:

to external factors that can be altered,to internal factors that can be altered, orto factors that cannot be altered.

PERFORMANCE GENERALLY IMPROVES WITHPRACTICE. . . BUT WHY?


W. L. Bryan

With practice, performance whether it is withtelegraphy, typing, software programs,arithmetic, programming, mnemonics, orvideo games generally improves.But experts are not simply faster thannovices; rather, they develop a hierarchy ofhabits than enable them to “step leagues”while novices are “bustling over furlongs orinches”.

CITATION

Bryan, W. L. & Harter, N. (1897). Studies in the physiology and psychology ofthe telegraphic language. Psychological Review, 4(1), 27–53

Bryan, W. L. & Harter, N. (1899). Studies on the telegraphic language: theacquisition of a hierarchy of habits. Psychological Review, 6(4), 345–375

PERFORMANCE GENERALLY IMPROVES WITHPRACTICE . . . EXCEPT WHEN IT DOESN’T!


Hypothesized that telegraphic expertise consisted of a hierarchyof habits.Plateaus were periods in which elements at one level of thehierarchy were being combined so as to be used at a higher level.dots and dashes → letters → words → phrasesBut . . .

TlIIC PsYt HOLOGICW- RuVIIiW VOL. I V . , P.LATE

Article of BRYAN \ND NOBLE.

^ -

THESE PLATEAUS COULD LAST A LONG LONG TIME


E. L. Thorndike

“I venture to prophesy that the thousandbookkeepers in, say, the grocery stores of NewYork who have each had a thousand hours ofpractice at addition, are still, on the average,adding less than two-thirds as rapidly as theycould, and making twice as many errors as theywould at their limit.”

“It appears likely that the majority of teachers make no gain inefficiency after their third year of service, but I am confident thatthe majority of such teachers could teach very much better thanthey do.”

“It seems to me therefore that mental training in schools, inindustry and in morals is characterized, over and over and overagain, by spurious limits – by levels or plateaus of efficiencywhich could be surpassed.”

CITATION

Thorndike, E. L. (1913). Educational Psychology Vol II: The Psychology ofLearning. NYC: Teachers College, Columbia University

PLAN FOR THIS TALK

Skip the remainder of the first 90 years (1897 – 1987) of scientificresearch on expert performance.Jump to 1987 to Carroll & Rosson’s Paradox of the Active User andEricsson’s (1993) Deliberate Practice and the view shared by boththat expert is not good enough.Three types of performance asymptotes and one type of plateau.Summarize everything in time for questions!


Outline

1 Mere Expertise is Not Good Enough

2 Plateaus and Asymptotes

3 Outside the Lab: Good → Better → Best!


OUTSIDE THE LAB – THE PARADOX OF THE ACTIVEUSER


Jack Carroll Mary Beth Rosson

In 1987, Carroll and Rosson coined the term,Paradox of the Active User, to refer to the“suboptimal use of office productivity software”by people who use the systems daily across thecourse of weeks, months, and years.

Carroll and Rosson, who at that time worked forthe IBM Watson Research Center, shared theHCI community’s concern that the expectedproductivity gains of the computer revolutionwere not occurring.

MERE EXPERTISE IS NOT GOOD ENOUGH


Anders Ericsson

A few years later, based on his studies of humanexpertise, Ericsson (1993) concluded that, “thebelief that a sufficient amount of experience orpractice leads to maximal performance appearsincorrect”.

CITATION

Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role ofdeliberate practice in the acquisition of expert performance. PsychologicalReview, 100(3), 363–406

LIMITS TO EXPERTISE


TlIIC PsYt HOLOGICW- RuVIIiW VOL. I V . , P.LATE

Article of BRYAN \ND NOBLE.

^ -

After years of lurking in the background, theplateau had returned to front and center.

RESOLVING THE PARADOX OF THE ACTIVE USER

Suboptimal performance can be amazingly stable! Fu and Gray (2004).What is optimal?

Is it optimal to learn 100 different commands and procedures thateach do one thing very well (fast, precisely, . . . )?Is it optimal to learn one command and procedure that can betweaked into doing 100 different things? (slowly, approximately, . . . )

Breaking a habit – the case of hunt & peck versus touchtyping


CITATION

Fu, W.-T. & Gray, W. D. (2004). Resolving the paradox of the active user:Stable suboptimal performance in interactive tasks. Cognitive Science,28(6), 901–935

Outline





PLATEAUS VERSUS ASYMPTOTES: POLE VAULTING


A history of technological innovations as the composition of thepole changed from ash (wood), to bamboo (wood), tofiberglass/carbon.Each technology, enabled pole vaulters to break new recordsFollowed by invention of new methods that resulted in newrounds of record breaking as those methods were adopted andadapted by athletes.Asymptote → New technology → New methods → Asymptote

PLATEAUS VERSUS ASYMPTOTES: HIGH JUMPING


A history of innovations in methods.Method – Fosbury Flop could have been invented earlier.Plateau → New Method → New Plateau

The Scissors and Straddle technique for high jumping.

The Fosbury Flop technique for high jumping.

PLATEAUS AND ASYMPTOTES

How do plateaus and asymptotes come about??


ASYMPTOTE DUE TO ARTIFACT DESIGN


Crossman’s (1959) study of cigar rollers in Cuba. Plot shows a continuedincrease in performance over a two year period (estimated as 3 millioncigars) and then a flattening of the curve.

Newell and Rosenbloom (1981, p. 7) attribute this flattening to a “knownlower bound for the performance time” in this task; namely, the “cycle timeof the machine.”

ASYMPTOTE DUE TO SYSTEM DESIGN


A field trial of two workstations for Telephone Operators(Gray, John, & Atwood, 1993).

Expected call times to decrease across the 4-mon trial.

But after 2-mon worktimes stabilized with times per callslower than for the old workstation

Slow enough to increase annual operating costs by $6.2million (in 2014 dollars).

Diagnosis:

Designers believed call time driven by the # of keys-per-call.

Predicted savings of 4.1 s in mean item per call for annual savings of $24m.

BUT cognitive modeling showed that old workstation enabled Operators tointerleave keypresses, chats with customer, and wait time for externaldatabases.

Conclusion:

Based on the models, Operators were becoming more expert at the newworkstation but asymptotes due to systems design prevented these gains inexpertise from yielding performance increments.

ASYMPTOTE DUE TO MEASUREMENT METHOD


Example from Space Fortress in which a measurement method was introducedabout 1994 that has affected many of the conclusions reached by researchers eversince.To simplify the story . . . changed scoring so that it included 4 component scoresand one overall score. These measures can be shown to be (1) not independent ofeach other and (2) two of these measures asymptote even as skilled performanceincreases.

ASYMPTOTE OR PLATEAU? THE CASE OF DIGIT SPAN

The Digit Span Task – An important part of the Wechsler AdultIntelligence Scale (WAIS) IQ test (and others).

Digits (0-9) are read at the rate of 1 per sec.Followed immediately by ordered recall.If all digits were recalled correctly, the length of the next run of digitswas increased by 1.If all are not correct, the next run is decreased by 1.

The population norm is 7± 2.Is this an asymptote due to limitations built into the human brain? orIs this a plateau due to massive stable suboptimal performance on thepart of the entire human population?


ASYMPTOTE OR PLATEAU? THE CASE OF DIGIT SPAN

Well? . . . This is an IQ test item!! Therefore it MUST be measuring anindividual difference variable that differs between humans but is stable, orasymptotic, for any given individual.

Right . . . ?


IS HUMAN DIGIT SPAN A POPULATION ASYMPTOTEOR PLATEAU? – PLATEAU!!


The difference between a plateau and asymptote is made clear by theexistence of extreme experts with a known history of transcending theplateau.

DDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDDD

SFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSFSF

10

20

30

40

50

60

70

80

0 10 20 30 40 50Practice (in 5 day blocks)

Dig

it S

pa

n

Perhaps the human asymptote is 80± 2 instead. . . ????

Strategy Plateaus and the Paradox of the Active User.

Issue: The difference between a plateau and an asymptote may behard to determine.Asymptotes may reflect a problem that can be fixed by design(whether artifact design or system design); however, plateaus due tostrategy-induced suboptimality may arise when the strategies deployeddo not enable utility maximization in the task environment.

Overcoming such strategy-induced suboptimality is usually verydifficult.Example of transfer from visually-guided to touch typing.


Outline





Real World Tasks?


Outside the Lab – Resolving the Paradox of the Active User


If “most people and professionals reach a stable performanceasymptote within a limited time period” (Ericsson, 2004), thenpractice does not make perfect.Unless, as a society, we can be content with “stable suboptimalperformance plateaus” then something more is needed. Wesuggest that this “something more” is research into theacquisition of expertise in mundane (i.e., everyday) taskenvironments.


Thank You!!

REFERENCES I

Bryan, W. L. & Harter, N. (1897). Studies in the physiology and psychologyof the telegraphic language. Psychological Review, 4(1), 27–53.

Bryan, W. L. & Harter, N. (1899). Studies on the telegraphic language: theacquisition of a hierarchy of habits. Psychological Review, 6(4),345–375.

Crossman, E. R. F. W. (1959). A theory of the acquisition of speed-skill.Ergonomics, 2(2), 153–166.

Ericsson, K. A. (2004). Deliberate practice and the acquisition andmaintenance of expert performance in medicine and related domains.Academic Medicine, 79(10, S), S70–S81.

Ericsson, K. A., Krampe, R. T., & Tesch-Römer, C. (1993). The role ofdeliberate practice in the acquisition of expert performance.Psychological Review, 100(3), 363–406.


REFERENCES II

Fu, W.-T. & Gray, W. D. (2004). Resolving the paradox of the active user:Stable suboptimal performance in interactive tasks. Cognitive Science,28(6), 901–935.

Gopher, D., Weil, M., & Bareket, T. (1994). Transfer of skill from acomputer game trainer to flight. Human Factors, 36(3), 387–405.

Gray, W. D., John, B. E., & Atwood, M. E. (1993). Project Ernestine:Validating a GOMS analysis for predicting and explaining real-worldperformance. Human-Computer Interaction, 8(3), 237–309.

Newell, A. & Rosenbloom, P. S. (1981). Mechanisms of skill acquisition andthe law of practice. In J. R. Anderson (Ed.), Cognitive skills and theiracquisition (pp. 1–55). Hillsdale, NJ: Lawrence Erlbaum Associates.

Thorndike, E. L. (1913). Educational Psychology Vol II: The Psychology ofLearning. NYC: Teachers College, Columbia University.


wayne gray presentation

Technology

rpi expert

hierarchy of habits

human performance wayne

long long time gray

plateaus of eciency

given performance ceiling

majority of teachers

telegraphic language