Understanding Visualization through Spatial Ability Differences

Maria C. Velez, Deborah Silver and Marilyn Tremaine

Rutgers University2005

2

What this Talk is About

Different people have been shown to have a lot of trouble with 3D visualizations

To investigate this issue further, we ran an experiment comparing measured spatial skills to abilities to understand visualizations

The results suggested key problems individuals were having and ways in which we can make the visualizations understandable by a larger audience

3

Motivation

Issues in Visualization Understanding

Classic 3D visualizations (2D projection and slices) have been found to be suboptimal for tasks like understanding shape and 3D space layout.

Examples of conventional visualization displays used in medical and weather imaging

4

Previous Solutions to Visualization Difficulties

Improvements have been proposed:

Exovis Cube Corner

Training methods have been developed:

“Pool of water” metaphor used in cutting planes training

Improvements are ad hoc rather than theory-based

Mental Rotation Test used to study spatial comprehension

5

Focus – What Makes a Visualization Difficult?

We use differences in human spatial abilities to understand the problems that affect people’s understanding of a visualization• Controlling for human variability makes effects detectable• Looking at extremes helps us understand normal behavior

Our questions• Does everyone solve problems similarly?• Do they make the same error?• How does diversity in the population affect performance in the

visualization?• Do the solutions proposed help everyone equally?

6

Research Approach

I. Select a set of cognitive skills that are likely to play a role in visualization understanding

II. Measure these cognitive skills with standardized tests using a group of subjects selected for their variability

III. Measure the level of visualization understanding of the subjects (via one type of “prototypical” visualization test)

IV. Match the visualization performance results to the standardized test results

V. Examine the properties of the visualization for both successful and unsuccessful comprehensions for each spatial ability subgroup

VI. Examine the error distributions of the wrong answers for each spatial ability subgroup

7

Human Spatial Abilities

What are spatial abilities?

• Skills involving the retrieval, retention and transformation of visual information in a spatial context.

Are there other relevant cognitive factors ?

Spatial Orientation

Spatial Location Memory

Targeting

Spatial Visualization

Disembedding

Spatial Perception

Visual Memory

Perceptual Speed

IDetermine which cognitive skills that might play a role in visualization understanding

8

Standardized Tests

Measuring spatial abilities• We measured spatial abilities through the Kit of • Factor-Referenced Cognitive Tests available at ETS.

IIMeasure a subset of those cognitive skills with standardized tests using a group of subjects selected for their variability

• Spatial Orientation: Cube Comparison Test

• Spatial Visualization: Paper Folding Test

• Disembedding: Hidden Patterns Test

• Visual Memory: Shape Memory Test

• Perceptual Speed: Identical Figures Test

9

The Visualization Test IIIMeasure the level of visualization understanding of the subjects (via a “prototypical” visualization test)

Goal: Examine the comprehension of a “prototypical” visualization: orthogonal projection• Basic visualization without bells and whistles• Easy to learn by untrained experiment participants• Use geometrical (geon-like and compounded) and

common realistic objects

Geon-like Compounded Realistic

10

The Visualization Test – Screen 1

Mentally form an

image of the object and its

alignment

11

The Visualization Test – Screen 2

Select the object that represents the object

creating the projections

This should be your answer

12

Experiment Design

Measures of performance• Accuracy: percentage of correct answers• Analysis Time : Time spent analyzing the object’s

projections (seconds)• Selection Time : Time spent selecting the answer

(seconds)Analysis Time and Selection Time are measured

independently

Experiment Participants (selected for variability)• 56 paid participants, 50% percent female• Average age: 21 years (range: 18 to 31 years) • 84% undergraduate, 16% graduate students

13

Experimental Method

Experiment Procedure

• Five paper-based cognitive factor tests• General instructions and five practice questions• Computer-based visualization test : 60 minutes

to complete 38 questions • Debriefing explaining the purpose of the

experiment

14

Road Map to Analysis

Spatial test resultsStandardized cognitive test

Performance

Analysis of subject’s errors(Case by case analysis)

Correlation

Visualization properties results

Spatial ability groups results

Correlation

Differences

15

Expected Relation Between Spatial Ability and Visualization Performance

Shape Memory

Cube Comparison

Paper Folding

Pattern Matching

Identical Figures

Accuracy Low Positive /

No correlation

+ High + High + Low No correlation

Time of Analysis

Low Positive /

No correlation

+ High + High + Low - High

Time of Selection

Low Positive /

No correlation

+ High + High + Low - High

+ Positive correlation- Negative correlation

16

Spatial Test Results

Analysis• Pearson correlation analysis between performance and scores in

standardized tests

IVMatch the visualization performance results to the standardized test results

Results

Shape Memory

Cube Comparison

Paper Folding

Pattern Matching

Identical Figures

Accuracy + Low + High + High No Corr. - Low

Time of Analysis No Corr. No Corr. No Corr. No Corr. - Low

Time of Selection No Corr. No Corr. No Corr. - Medium - Low

Implications• Visualization comprehension on diverse populations affected by

spatial ability diversity• Paper tests were time constrained which may have affected the

time correlations

Not ExpectedExpected

17

Visualization Properties Results

For the object’s properties (i.e. surfaces, edges and vertices) we calculated:• Total count in the original 3D object• Distinct properties that would be visible in a wireframe

rendering of the projection.• Visible properties in a uniformly shaded object

e1

e2

e3

e4

e5

e6e7 e1

e2

e3

e4

e5

e6e7e1

e24

e35

e6

Total count of edges: 12

Distinct edges: 7 Visible edges: 4

18

Visualization Properties Results

Analysis• Pearson correlation analysis between performance and property

counts and ratios

Implications• The hidden geometric properties make visualization understanding

cognitively harder and thus, more time consuming• Rotation of objects and animation will help users’ comprehension• Complex objects require slower animations to give viewer time to extract

information

CountsRatios Visualized / Distinct

Edges Vertices Surfaces

Accuracy No Corr. + High + Medium + Medium

Time of Analysis + Medium - High - High - High

Time of Selection No Corr. No Corr. No Corr. No Corr.

Results

19

Other Visualization Properties Results

A learning curve was not detected (see figure)

HGFEDCBA

35.00

30.00

25.00

20.00

15.00

10.00

Tim

e (s

eco

nd

s)

Time of answerselection

Time of projectionanalysis

Analysis time vs. Selection time

Repeated object in order of appearance

73% 52% 46% 73% 63% 70% 70% 46% Accuracy

Accuracy was affected by choices that differed from the correct answer by small differences in orientation

No significant performance differences were found between geometric and realistic objects

20

Spatial Ability Groups Results

No significant property differences between:• All questions with high percentage of correct answers• All questions with high percentage of incorrect answers

VExamine the properties of the visualization for both successful and unsuccessful comprehensions for each spatial ability subgroup

BUMMER!

Our Next Step is to Look at the Data in More Detail

Divide-up participants 3 groups and selected the High Spatial (HS) and Low Spatial (LS) ability participants (based on Paper Folding Test).• Knowing a source of variability and looking at the extremes

helps to make the effect visible

21

Spatial Ability Groups Results

Analysis• Are there properties that only high spatial people use?• Compare Properties of Questions Answered Correctly by high

spatial participants to Properties of all Questions

Results • Total Number of Edges and Total Number of Vertices were found

significantly higher in questions which the high spatial participants answered correctly

• The Ratio of Distinct to Visualized Surfaces was found significantly higher in questions answered correctly by high spatial participants

Implications• High spatial participants understand more complex objects and can

process a higher number of hidden properties

22

Analysis of Subject’s Errors VI Examine the error distributions of the wrong

answers for each spatial ability subgroup

Analysis• Create a bar graph showing distribution of answers for each

question• Analyze the questions where distributions clearly not evenly

distributed

Interesting results for further analysis (possible strategies)

Frequency of answers

High spatial ability 6

Low spatial ability 8

High spatial ability

Low spatial ability

23

Issues

Experiment trials organized according to what was believed to be trial difficulty. This organization was wrong.

There was ambiguity in the answers that participants had to choose between, in particular because participants were allowed to rotate the answers, they were not able to see the differences in orientation between two possible answers

Only projection visualization was studied and thus, the results cannot be readily extrapolated to many other 3D visualizations

The object properties manipulated in the questions were horizontal and vertical alignment. Future studies will include properties such as size, shape (sides), aspect ratio.

24

Summary of Results

Spatial abilities are related to 3D visualization comprehension

Problem solution time was not found to be related to visualization accuracy

Counts of geometric properties affected visualization accuracy for low spatial subjects, and time of analysis for everyone

The “hidden” geometric properties in the visualization affect visualization accuracy for low spatial subjects

Small rotation differences are difficult to detect in a visualization

A case by case analysis suggests that high spatial and low spatial ability participants use different strategies

25

Larger Implications of Research

Visualization designers can use measures of cognitive ability to help understand what makes visualizations hard/easy to comprehend

Using interactive rotation and animations is likely to help users better understand visualizations

Visualization difficulty may be highly variable for a diverse population

There exist educated people who cannot understand simple 3D visualizations

26

Acknowledgments

Thanks to our reviewers for their comments

This research is supported by the National Science Foundation through the SGER grant # 0503680

27