October 21, 2019 Sam Siewert

CS415Human Computer Interaction

Lecture 9, Part 1 – HCI Evaluation

Exam #1Returned on Canvas

Ave=80, High=92

Discuss Solutions in Class (posted on Canvas)

Commonly Missed Questions

Sam Siewert 2

Shifting Emphasis after Exam #1

Exam #2 - On Canvas (Outside of Class)

One Last Practice AssignmentARUniversal Design

More time for Design Work

Proposal

Work on Teams to Design & Prototype

Design Evaluation

Present for Final Exam

Review of Experimental Design

Evaluation Design and Significance

Sam Siewert 3

Design of Evaluation ExperimentsIndependent and Dependent Variables – Clearly Identify– Independent – changed to test effects upon Dependent – Dependent – measured during evaluation experiment

Evaluations– Participants (User groups)– Hypothesis – what do you think will happen? Ok to be right or

wrong!– IV, DV – Identify, Plan method to change IV, methods to measure

DV (ideally more than one measurement)– Experimental Design

Between groups (p. 331), Single-Blind or Double Blind with more UsersWithin groups – repeated measures, Users repeat with different conditions (IV modification)

– Task – What is “done” in the experiment (Don’t Confuse with IV)– Analysis – typically, a T-test for significance (avoid null hypothesis)

Sam Siewert 4

Example Evaluation ExperimentDriving Assistant – Lane Departure Warning and Control

– Participants – Licensed Drivers (16-25, 25-45, 45-65, 65+), well-rested, not driving under influence, undistracted

– Hypothesis – Reduces dangerous lane changes, accidents and unintended fog-line/center-line crossings

– IV – Warning Enable/Disable (Experiment #1)– IV - Control Enable/Disable (Experiment #2)– DV – Lane changes at unsafe separation, Number of accidents

(dangerous), Line crossings per mile– Design – Within Groups (Same Drivers for Enable/Disable)– Task – Drive between Prescott and San Diego Airport, or Drive

in a simulator (accidents)– Analysis – T-test

Sam Siewert 5

Intelligent Transportation - SafetyFord Lane Departure Warning and Correction

YouTube Video

Intrusive & Distracting?

Helpful?

How to Evaluate?

Sam Siewert 6

Using Excel - Makes it Easier!Must have Analysis ToolPak Added - Example

Sam Siewert 7

Step #1 Step #2

Step #3

Using Excel for T-test (Tutorial)Use Data, Data Analysis Function

Sam Siewert 8

# of Lane Departures per Hour Driving

Sample Group 1 with Warning Group 2 without Warning1 3 52 2 73 4 44 1 65 0 26 2 87 3 78 5 59 9 7

10 2 8

Mean 3.1 5.9Variance 6.322222222 3.655555556STD 2.514402955 1.91195072

F-Test Two-Sample for Variances

Variable 1 Variable 2Mean 3.1 5.9Variance 6.322222222 3.655555556Observations 10 10df 9 9F 1.729483283P(F<=f) one-tail 0.213453449F Critical one-tail 3.178893104

t-Test: Two-Sample Assuming Unequal Variances

Variable 1 Variable 2Mean 3.1 5.9Variance 6.322222222 3.655555556Observations 10 10Hypothesized Mean Difference 0df 17t Stat -2.803116306P(T<=t) one-tail 0.006112476t Critical one-tail 1.739606726P(T<=t) two-tail 0.012224953t Critical two-tail 2.109815578

(tStat < -t Critical two-tail) OR (tStat > t Critical two-tail), thus reject the null hypothesis

T-test

F-test for variance

User Evaluations and Alternatives

1. Personal inspiration and HCI ideas - design and prototype– Create an interactive shell or programming language (CS332) [1D HCI]– Use a GUI builder (QtCreator, PyTk, SWING, Xamarin) [2D HCI]– Render (Unity, Unreal, Blender, OpenGL, WebGL, x3dom), AR & MV

(OpenCV, OpenNI), VR (OpenXR, X3D) [3D HCI]– AI assistants (ROS, TensorFlow, PyBrain, cuDNN) [AI HCI]

2. Gurus - Principles, Guidelines, Rules (emulate success)3. Standards - Just follow, if you don’t want to innovate with your HCI4. Design of Experiments and Evaluations

– innovative HCI, that is lacking standards, guidelines, so no history– does not fit existing principles of usability

Sam Siewert 9

https://www.york.ac.uk/depts/maths/histstat/lies.htm

Minute Paper - Driver’s AssistantWas the Ford Lane Departure Warning effective?

What approach was taken to evaluate?

Was the evaluation effective?

Sam Siewert 10