The Computer

HCI

Interacting with computers

To understand Human-Computer Interaction… need to understand computers!

what goes in and out of computerdevices, paper, sensors, etc

what can the computer do?memory, processing, etc

Von Neumann Architecture

Timeline

Vacuum Tube Triode (1908)

Transistor1947(silicon, germanium)

Very Large ScaleIntegrated (VLSI)Circuits; 1970s--(> 1,000 transistorsper chip)

Intel Itanium (Tukwila) 2008: 2 billion transistors

The Computer

A computer system is made up of various elements

Each of these elements affects the interaction input devices – text entry and pointing output devices – screen (small&large), digital paper virtual reality – special interaction and display

devices physical interaction – e.g. sound, haptic, bio-sensing paper – as output (print) and input (scan) memory and processing– memory, access, speed of

processing, networks

How many computers …

in your house?

PC DVD, HiFi, TV microwave, cooker,

washing machine, fridge

Air Condition security system can you think of

more?

in your pockets?

PDA smartphone digital camera smart card electronic car key USB memory try your pockets

and bags

Richer interaction

sensorsand deviceseverywhere

TEXT ENTRY DEVICES

keyboards (QWERTY et al.)

chord keyboards, phone pads

handwriting, speech

Input (1)

Batch Large amounts

Pre-prepared information

Standardised data loaded quickly using automated routines

Used a lot in the past (punch cards)

Keyboards

Most common text input device Allows rapid entry of text by

experienced users Keypress closes connection,

causing a character code to be sent

Usually connected by cable, but can be wireless

Keyboards

QWERTY Keyboard (1878) the official standard of

computer keyboards [ISO9995]

was designed for two finger typing

prevents stuck keys (on a typewriter)

the position of the keys the hands are held close together helping to cause Carpal Tunnel Syndrome

can be used without training

is cheap and easy to find

Any other keyboard?

Keyboards (ctd) - layout

Alphabetic layout keys arranged in alphabetic order not faster for trained typists not faster for beginners either!

Dvorak layout common letters under dominant fingers biased towards right hand common combinations of letters alternate between hands 10-15% improvement in speed and reduction in fatigue But no significant adoption, when compared to QWERTY

Keyboards

Over the years, attempts were made to physically improve keyboards … Tilting Curving Splitting

Keyboards (ctd) - special keyboards

Maltron left-handed keyboard

Virtual keyboard

Projection keyboard

Chord keyboards

only a few keys - four or 5 letters typed as combination of keypresses compact size – ideal for portable applications short learning time – keypresses reflect letter shape fast – once you have trained

BUT - social resistance, plus fatigue after extended use

NEW – niche market for some wearables

Keyboards

Chord Keyboards Don’t use an alphabet Use 4/5 keys Rely on combination of keys Ideal for one hand or mobile devices Can suffer fatigue after use New application in mobile devices?

Phone keypad and T9 entry

use numeric keys withmultiple presses2 – a b c 6 - m n o3 - d e f 7 - p q r s4 - g h i 8 - t u v5 - j k l 9 - w x y z

hello = 4433555[pause]555666 surprisingly fast!

T9 predictive entry type as if single key for each letter use dictionary to guess the right word hello = 43556 … ‘hello’ not ‘gdjjm’ note 26 could mean ‘am’ or ‘an’; phone gives those options

Handwriting recognition

Text can be input into the computer, using a pen and a digesting tablet natural interaction

Technical problems: capturing all useful information (path, pressure, etc) in a

natural manner segmenting joined up writing into individual letters coping with different styles of handwriting etc

Used in PDAs, tablet computers, smart white boards …… leave the keyboard on the desk!

Speech recognition

Most successful when: single user – initial training and learns

peculiarities limited vocabulary systems

Improving, but challenges such as: external noise interfering imprecision of pronunciation/articulation

e.g. vagueness, pauses, clearing throat, repetition, etc

large vocabularies different speakers

POSITIONING, POINTING AND DRAWING

mouse, touchpadtrackballs, joysticks etc.touch screens, tabletseyegaze, cursors

Mouse

Movement of mouse moves screen cursor screen cursor oriented in (x, y) plane mouse movement in (x, z) plane …

Since the mouse sits on a desk moving it about is easy users suffer little arm fatigue cursor is small and can thus be manipulated

without obscuring the display

Disadvantage hand-eye coordination problems for novice users

Mouse (ctd)

Two methods for detecting motion Mechanical

ball on underside of mouse turns/rotates as mouse is moved

can be used on almost any flat surface Optical

light emitting diode on underside of mouse may use special grid-like pad or just on an

appropriate surface detects fluctuating alterations in reflected light

intensity to calculate relative motion less susceptible to dust and dirt

Mouse (ctd)

Footmouse Motivation

foot pedals are used heavily in musical instruments (pianos, organs, drums) and in mechanical equipment (cars, cranes, sewing machines)

Relevant hands-occupied situations users with disabilities or with high-back or neck problems

Challenges feet tend to be less precise than hands practicality of having foot controls in the work

environment (e.g. pedals under a desk may be operated accidentally, etc)

Touchpad

a pointing device featuring a specialized surface that can translate the motion and position of a user's fingersto a relative position on screen

operated by stroking a finger over the surface used mainly in laptop computers good ‘acceleration’ settings important

fast stroke lots of pixels per inch moved initial movement to the target

slow stroke less pixels per inch for accurate positioning

Trackball and thumbwheels

Trackball ball is rotated inside static housing

like an upside down mouse! separate buttons for picking/selection very fast for gaming used in some portable and notebook computers

Thumbwheels two dials for X-Y cursor position for fast scrolling – single dial on mouse

Joystick and keyboard nipple

Joystick pressure of stick translates to velocity of movement buttons for selection e.g. on top or on front like a trigger often used for computer games, aircraft controls and 3D

navigation

Keyboard nipple or pointing stick a pointing device typically mounted in a keyboard on

laptops pointing stick can be considered to be a miniature joystick pointing stick operates by sensing applied force the velocity of the pointer depends on the applied force force applied on the pointing stick is echoed on the screen

by movements of the pointer

Touch-sensitive screen

Detect the presence of finger or stylus on the screen. Increasingly being accepted in the mass market especially in

mobile devices e.g. smartphones and tablets direct pointing device more natural gestures through multitouch (i.e. ability to recognize the

presence of two or more points of contact with the screen surface, and therefore supporting more natural gestures such as pinching and reverse pinching)

Advantages: fast, and requires no specialised pointer good for menu selection suitable for use in hostile environment: clean and safe from damage.

Disadvantages: finger can mark screen imprecise (finger is a fairly blunt instrument!) if screen is vertical, lifting arm can be tiring unless supported

Stylus

small pen-like pointer to draw directly on screen

may use touch sensitive surface or magnetic detection

used in PDA, tablets PCs and drawing tables pros

very direct obvious to use

cons can obscure screen

Skinput: Body as an Input Surface

CHI’2010 Use a tiny projector on body to show menus Microphones to listen to taps on hand/arm

Signal processing and machine learning todifferentiate positions

30

Pointing Devices

Integrated gaze and face tracking system. A camera and IR light source tracks gaze by computing the angle between the corneal reflection and the centroid of the pupil.

Eye gaze or eye tracking

a laser light reflects off the retina provides a control interface by tracking

eye/visual movement on the screen e.g. look at a menu item to select it

high accuracy requires headset cheaper and lower accuracy devices

available mainly used for evaluation of interactive

systems e.g. in order to provide information on clicking and scrolling patterns of web users

it is also relevant for disabled users

Discrete positioning controls

in phones, TV controls etc mini-joysticks (mobile phones) central cursor pads & discrete left-

right, up-down buttons (TV remote controls) e.g. +/- for controlling volume &

navigating channels

mainly for menu selection

Pointing Devices (6)

Smart Systems

Minority Report … reality or fiction?

DISPLAY DEVICES

bitmap screens (CRT & LCD)

large & situated displaysdigital paper

Output Devices

Permanent output Paper Microfiche Film

Impermanent output Screen Sound Tactile

Bitmap displays

bitmap display or raster scan the whole screen is scanned sequentially and

horizontally screen is vast number of coloured dots

Visual Output

Cathode Ray Tube (CRT) Text based to high resolution Produce minor health

hazards Since they are radiation emitting

devices!

Liquid Crystal Display (LCD)

Alternative systems might not use a screen … Lights Gauges Dials Can you give examples?

Large and situated displays

Large displays Used for meetings, lectures, etc Some use gas plasma technology to create large flat

bitmap displays Where very large screen areas are required, several

smaller screens, either LCD or CRT, can be placed together in a video wall where each screen displays a portion of the whole (or separate sub-image) and the result is an enormous image

Situated displays As well as for lectures and meetings, display screens

can be used in various public places to display information; they could even be interactive

These are often called situated displays as they take their meaning from the location in which they are situated i.e. the meaning of information or interaction is related to the location

PAPER: PRINTING AND SCANNING

print technology

fonts, page description, WYSIWYG

scanning, OCR

Permanent Output

On paper using printers Impact Printers (Noisy)

Dot Matrix (Slow + Low Quality) Daisy Wheel (Very Slow + 1 Font) Chain and Band printers (Fast + Expensive)

Non-Impact Printers (Silent, High Quality, Fast) Inkjet Thermal (Expensive) Laser (Cost effective + Colour expensive)

Permanent Output

Printers Speed measured in Pages Per Minutes Resolution measured in Dots Per Inch (dpi)

80 dpi (dot matrix) = Near Letter Quality 300 dpi – 1200 dpi (non-impact) = Near Book Quality

Store information in firmware such as Typeface (Fonts) Character set Size

More complex printers use Page Description Language (PDL) Communicates complex commands directly to printer Used in pdf and postscript

Fonts

Font – the particular style of text

Courier font Helvetica font

Palatino font Times Roman font

(special symbol)

Size of a font This is ten point Helvetica

This is twelve point This is fourteen point

This is eighteen point and this is twenty-four point

Fonts (ctd)

Pitch fixed-pitch – every character has the same

width e.g. Courier variable-pitched – some characters wider

e.g. compare the “i” and the “m” in Times New Roman

Serif or Sans-serif sans-serif – square-ended strokes

e.g. Helvetica serif – with splayed ends (such as)

e.g. Times Roman

Readability of text

lowercase All lowercase text considered to be easier to read than all uppercase

UPPERCASE better for individual letters and non-words

e.g. flight numbers: BA793 vs. ba793

Scanners

Take paper and convert it into a digital document

Useful in desktop publishing for

incorporating photographs and other images

document storage and retrieval systems, instead of physical paper storage

Optical character recognition OCR converts bitmap back into text different fonts

create problems for simple “template matching” algorithms

more complex systems segment text, decompose it into lines and arcs, and decipher characters that way

page format columns, pictures, headers and footers

VIRTUAL REALITY AND 3D INTERACTION

positioning in 3D spacemoving and grasping

seeing 3D (helmets and caves)

Positioning and viewing in 3D

Positioning in 3D space 3D mouse: six-degrees of movement: x, y, z + roll, pitch, yaw data glove: to detect finger position VR helmets: to detect head motion and possibly eye gaze whole body tracking: accelerometers, image/video processing,

etc

Viewing in 3D often use stereoscopic vision ( two 2D images are presented

separately to the left and right eye of the viewer, which are then combined in the brain to give the perception of depth), but also possible to use real 3D display (displaying an image in three full dimensions)

Simulators and VR caves

scenes projected on walls realistic environment real controls other people

PHYSICAL CONTROLS, SENSORS ETC.

special displays and gauges

sound, touch, feel, smell

physical controls

environmental and bio-sensing

Sounds

can take many forms beeps bongs (deep ringing sounds) clonks (loud thudding sounds) whistles whirrs (like the sound of rapidly vibrating

wings) used for

error indication (e.g. incorrect command/input) or risky situations (e.g. when deleting files)

confirmation of actions e.g. keyclick notification of events/status e.g. a new email

has arrived, download has been completed

Haptic devices - example

BMW iDrive for selecting and controlling menus one feels little bumps along the way as one

maneuvers through options and menus makes it easier to select options by feel uses haptic technology from Immersion Corp

Physical controls

specialist controls needed … industrial controls, consumer products,

etc.

large buttonsclear dials

tiny buttons

multi-functioncontrol

easy-cleansmooth buttons

Environment and bio-sensing

sensors all around us car courtesy light

small switch on door ultrasound detectors

security, washbasins RFID security tags in shops temperature, weight, location

MediaCup

cup has sensors heat, movement, pressure

broadcasts state (IR) used for awareness

user is moving, drinking, …

incidentally colleagues are aware

Han's purpose to drink coffee

OTHER ASPECTS

Compression

reduce amount of storage required lossless

recover exact text or image – e.g. GIF, ZIP look for commonalities:

text: AAAAAAAAAABBBBBCCCCCCCC 10A5B8C

lossy recover something like original – e.g. JPEG,

MP3 exploit perception

JPEG: lose rapid changes and some colour MP3: reduce accuracy of drowned out notes

Storage formats - media

Images: many storage formats :

(PostScript, GIFF, JPEG, TIFF, PICT, etc.)

plus different compression techniques(to reduce their storage

requirements)

Audio/Video again lots of formats :

(QuickTime, MPEG, WAV, etc.) compression even more important also ‘streaming’ formats for network delivery

Finite processing speed

Designers tend to assume fast processors, and make interfaces more and more complicated

But problems occur, because processing cannot keep up with all the tasks it needs to do cursor overshooting because system has buffered

keypresses icon wars - user clicks on icon, nothing happens,

clicks on another, then system responds and windows fly everywhere

Also problems if system is too fast - e.g. help screens may scroll through text much too rapidly to be read

Computer Memory

Short Term RAM

Long Term Secondary Backing Storage

Hard disk Disk drive CD’s Memory Sticks

Permanent Storage Cheaper Slower Permanent (Non-volatile) Several types

Magnetic Optical

Different capacities Floppies 1.4 Mb Memory Stick 64 Mb + CD 640 Mb DVD 4 Gb – 18 Gb Blue Ray 15 Gb – 30 Gb Harddisk 120 Gb + … Tera Bytes

Two writing types Rewritable (Hard disk) Write Once Read Many WORM (CD, punch cards)

Recap: Boolean Logic

Boolean Expression E = S AND D

Truth table:Value of E for every possible D, S. TRUE=1; FALSE= 0. 001

011

110

000

ESD

Boolean Circuit ES

D

Truth table has rows ifthe number ofvariables is k

2k

Combinational circuit for binary addition?

Want to design a circuit to add any two N-bit integers.

2511001

+ 2911101

54 110110

Is the truth table method useful for N=64?

A Full Adder (from handout)

R-S Flip-Flop S

R

M

A more convenient form of memory

Fact: “Data Flip-Flop” or “D flip flop”;can be implemented using R-S flip flops.

No“undefined”outputs ever!

“Register” with 4 bits of memory

RAM: Implementing “Write”

RAMDecoder(Demux)

DataThe decoder selects which cell in the RAM gets its “Write” input toggled

K-bit address(in binary)

Clock

(simple combinationalcircuit; see logic handout)

Greatly simplified view of modern CPUs.Program (in binary)

stored in memory

Memory Registers Arithmetic and Logic Unit

(ALU)

Control FSM

Lots of Custom HardwareInstruction Pointer

RAM

The “symphony” inside a computer

Clock

Combinational circuit

Memory

ClockedSequentialCircuit(akaSynchronousCircuits)

Limitations on interactive performance

Computation bound Computation takes too long, causing frustration for the

user

Storage channel bound Bottleneck in transfer of data from disk to memory

Graphics bound Common bottleneck: updating displays requires a lot of

effort - sometimes helped by adding a graphics co-processor optimised to take on the burden

Network capacity Many computers networked - shared resources and

files, multiple access to printers, etc - but interactive performance can be reduced by slow network speed

Viruses and Worms

Automated ways of breaking in;Use self-replicating programs

(Recall self-replicating programs: Print the following line twice, the second time in quotes. “Print the following line twice, the second time in quotes.” )

Email Viruses

Infected program, screen saver, or Word document launches virus when opened

Use social engineering to entice you to open the virus attachment

Self-spreading: after you open it, automatically emails copies to everyone in your address book

The Melissa Virus (1999)

Social engineering: Email says attachment contains porn site passwords

Self-spreading: Random 50 people from address book

Traffic forced shutdown of many email servers

$80 million damage 20 months and $5000 fine David L. Smith

Aberdeen, NJ

Combating Viruses

Constant battle between attackers and defenders

Example: Anti-virus software finds “signature” of known

virus Attacker response: Polymorphic virus –

to thwart detection, change code when reproduced

Anti-virus software adapts to find some kinds of polymorphism

But an infinite number of ways to permute viruses available to attackers

The Morris Worm (1988)

First Internet worm Created by student at

Cornell Exploited holes in email

servers, other programs Infected ~10% of the net Spawned multiple copies,

crippling infected servers Sentenced to 3 years

probation, $10,000 fine, 400 hours community service

Robert Tappan Morris

The Slammer Worm (2003)

Fastest spreading worm to date Only 376 bytes—Exploited buffer overflow in

Microsoft database server products Spread by sending infection packets to random

servers as fast as possible, hundreds per second Infected 90% of vulnerable systems within 10

minutes! 200,000 servers No destructive payload, but packet volume shut

down large portions of the Internet for hours 911 systems, airlines, ATMs — $1 billion damage! Patch already available months previously,

but not widely installed

Bot

Zombies

Bot program runs silently in the background, awaiting instructions from the attacker

Attacker’s Program

“Distributed Denial of Service”

Objective: Overwhelm target site with traffic

DDOS Attacks

“Attack www.store.com”

Messages are hard to filter because there are thousands of senders

Sending Spam

“Forward this message:Subject: Viagra!…”

Spyware/Adware

Hidden but not self-replicating

Tracks web activity for marketing, shows popup ads, etc.

Usually written by businesses: Legal gray area

Conclusion

HCI depends on computer operating as an intermediary, achieving

appropriate interaction through a variety of I/O devices

WHAT COMPUTERS JUST CANNOT DO.

M.C. Escher

Print Gallery

High-level description of program that self-reproduces

Print 0

Print 1.

.

.

Print 0

. . . . . .

. . . . . .

. . . . . .

. . . . . .

} Prints binary code of B

}Takes binary string on tape, and in its place prints (in English) the sequence of statements that produce it, followed by the translation of the binary string into English.

A

B

Score:

Audio samples

Spectrogram

How do you represent music?

Creating music: Synthesis

Review Questions

What are the most common interaction devices? What are their limitations? How can some of them be overcome by novel

devices?

Suggest ideas for an interface which uses speech and sound effectively.

How can sound be used to convey information?

Review Questions

What are some of the factors you might consider when designing systems like a public access interactive kiosk?

Identify the strengths and weaknesses of multimedia and hypermedia systems.

Discuss how the speed of access to memory can affect the speed of an interactive computer system. Which other factors can affect such speed?

Revision