![Page 1: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/1.jpg)
Graphics and Words
When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make
sense? How to integrate images and words The Narrative thread
![Page 2: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/2.jpg)
Consider that hieroglyphs gave way to more abstract symbols
Why turn back the clock?
x 1
i
-17K years
![Page 3: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/3.jpg)
Imagens
Pavio’s dual coding theory
Visual System
AuditorySystem
Verbal
VisualImage Information
InformationFrom speech
Text Information
Associate structure
Logogens Verbalresponses
Non-verbalresponses
![Page 4: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/4.jpg)
Capacity of visual working memory (Vogal, Woodman, Luck, 2001)
Task – change detection Can see 3.3 objects Each object can have several attributes
OrientationShape size, texture
1 second
![Page 5: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/5.jpg)
Capacity of verbal working memory
Used to be though of a 7 +/- 2 It is now thought of as more a duration of
~ 2sec and a store of proto-verbal codes. Words have an enormously rich semantic
structure Sometimes we have a choice
Spatial prepositions: above, across, alongside, amidst, apart, around, back, behind, below, beneath, beside, between, beyond, by (out of approx 80: From Pinker)
![Page 6: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/6.jpg)
Reasoning with words and graphics
1 2 3 4 5 6 7 8 9 Get a set of numbers, such that 3 of them
add to 15.
![Page 7: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/7.jpg)
![Page 8: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/8.jpg)
Can we have a visual language?
![Page 9: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/9.jpg)
The nature of language Words are arbitrary Chomsky, grammar with innate deep structures. Common to computer languages Austin - Illocutionary force (speech acts) Critical period for language development But, being verbal is not essential to language
development Sign languages for the deaf are the most perfect
examples of visual language
![Page 10: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/10.jpg)
Sign languages
ASL Danish SL Chinese SL
Arose spontaneouslyAre not related to verbal languagesHave grammarBecome more abstract over time
NSLGoldin-Meadow
![Page 11: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/11.jpg)
What can we do with language?
Description The ability to communicate procedures and
sequences of operations – including logic – if, but, causes, do a then b then c
A vehicle for thought when internalized Communicate goals, intentions (illucutionary acts). Make things happen Nit picking (mutual grooming)
![Page 12: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/12.jpg)
“If there is no fresh trout at Whole Foods then go to the fishmarketOn 19th and get salmon.”
![Page 13: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/13.jpg)
There are special brain areas for language
Production
Wernicke
BrocaUnderstanding
Production
VisualCortex
AuditoryCortex
![Page 14: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/14.jpg)
Can we have a visual language?
Yes – but …
![Page 15: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/15.jpg)
If we don’t learn it early somesome capabilities are best handled verbally
![Page 16: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/16.jpg)
The visual system gives us
Visual processing andvisual recognition
Rapid recognition and pattern finding
![Page 17: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/17.jpg)
Abstraction Pattern
Jane is Jim’s boss Jim is Joe’s boss Anne works for Jane Mark works for Jim Anne is Mary’s boss Anne is Mike’s boss
Joe Mary
Jane
Mike
Jim
Mark
Anne
Org chart
![Page 18: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/18.jpg)
Visual and verbal pseudo-code
While letters in stack Take a letter Put a stamp on it Put it in the ‘out tray’
Visual programming languages have a history of failure
get line of textfrom input file
change charactersto upper case
write line to outputfile
more input?yes
no
Data flow diagrams are defunct
![Page 19: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/19.jpg)
Examples of visual languages
Sanscrit Petri-nets Khoros
![Page 20: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/20.jpg)
SanscritSum integers from 1 to 3
for i = 1 to 3 dosum = sum + i;
![Page 21: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/21.jpg)
Petri Net languagePetri nets are stochastic – timed attributed (tokens on nodes, transitions)
![Page 22: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/22.jpg)
Khoros
![Page 23: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/23.jpg)
Attaching images and words
Pointing, Diexis and the diectic gesture
Can be a glance or a nod An elementary speech
act Shown to disambiguate
verbal communications Why the mouse is so
powerful The basis of shared
environments
![Page 24: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/24.jpg)
Attaching words to images- use gestalt laws
A square
helplet me out!
Some simple shapes
hexagon
ab
c
d
![Page 25: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/25.jpg)
Integrated pictures and words moreEffective: Chandler and Sweller 1991
![Page 26: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/26.jpg)
Narrative: Visualizations can be used to help tell a story
Can something visual carry a narrative.
In psychological terms the important thing is a leading of the thread of ideas (=attention). In data analysis the analyst has the narrative
thread. In a presentation the audience is following the
presenter’s thread. (actually a hybrid is best).
![Page 27: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/27.jpg)
Narrative
Can be accomplished with either images or words
A leading of attention
![Page 28: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/28.jpg)
Degrees of narrative
Strong leading Free exploration
![Page 29: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/29.jpg)
XPlane
![Page 30: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/30.jpg)
![Page 31: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/31.jpg)
![Page 32: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/32.jpg)
Recap
Words for procedure, logic and abstract meanings/ images for pattern and structure
Both words and images can sustain narratives (leading attention)
![Page 33: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/33.jpg)
Emotional and motivational effects (Images can be more efficient) and automatic.
![Page 34: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/34.jpg)
Gillnet Set Surface Buoy
Lobster Trap Trawl
Surface Buoy
Landry 2002
![Page 35: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/35.jpg)
![Page 36: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/36.jpg)
Theory: Graphics and Words
Words for procedural logic: conditionals, qualifiers, if-then else, while. + causality.
Not the formal logic of math but a kind of concrete logic. (A vowel must have an odd number on the back).
![Page 37: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/37.jpg)
Other kinds of gestures
Beat gestures for emphasis Verb gestures showing how to do
something
McNeil Hand and mind
![Page 38: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/38.jpg)
Issues in shared environments
Speech + Pointer + Visuals – most important components
Subtle ways of directing attention also important in meeting dynamics.
![Page 39: Graphics and Words When should we use a visual display? When should be use words? Dual coding theory What is a visual language? – does it make sense? How](https://reader036.vdocument.in/reader036/viewer/2022081603/56649e2f5503460f94b1f321/html5/thumbnails/39.jpg)
Peter