implementation

chapter 8(modified)

implementation support

Introduction

how does HCI affect the programmer?

levels of abstractionhardware specific

→ interaction-technique specific

layers of development tools– windowing systems– interaction toolkits– architecture styles and frameworks

windowing systemsdevice independenceresource sharingapplication management

windowing systemsdevice independence

different devices:

mouse, trackpad, joystick

keyboard layouts

screen resolution

device independencelayers of abstraction

Operating System: hardware interface, low-level device drivers

Windowing System: abstract pointer (mouse)abstract display: pixels, Postscript, vector graphics

Toolkit: (e.g. Java AWT) more abstraction!

Application: what the user really wants to do

windowing systems resource sharing

manyapplications

(usually)one screen

one keyboardone mouse

one pair of eyesone set of fingers

resource sharing

window systemvirtual terminalseach app its own PC!

OS:shares disk, CPU, etc.

resource manager

devicedrivers

abstractterminal 1

abstractterm. 2

abstractterm. 3

abstractterm. 4

types of window layout

• tiled (space shared)– web sidebars, expanding widgets

• single app (time shared)– phone, early Windows

• overlapping (time and space)– most common for PCs

resource issuesnot just the screen!

users care about

power management

network utilisation... especially when it costs!

windowing systemsapplication management

many applications ... so

– consistent look and feel

– inter-application sharingcut & paste, drag & dropscripting and automation

– UI to manage it alle.g. Mac Finder, Windows Explorer

app swopping, preferences+extras: dashboard

Architectures of windowing systems

three possible software architectures– all assume device driver is separate– differ where management is implemented

1. each application manages everything– everyone worries about synchronization– poor portability and robustness– can be efficient for low-end devices

2. window mgt. tightly coupled with operating system– applications tied to operating system

3. management role as separate application– maximum portability

early Mac & MS Windowsassumed ‘cooperative’ apps

may be used fordedicated devices

Mac OS and MS Windows

X Windows

X Windows architecture

X Windows architecture (ctd)

pixel imaging model with some pointing mechanism

X protocol defines server-client communication

separate window manager client enforces policies for input/output:– how to change input focus– tiled vs. overlapping windows– inter-client data transfer

not just the PC

phone– similar issues – sharing screen, keyboard, etc.– ‘full screen’ apps, not overlapping windows

but Vodafone 360 had semi-open apps

web– browser takes role of window manager

web micro-apps (e.g. Facebook apps)

– access shared data, screen space

dedicated devices (e.g. microwave control panel)

– mostly code direct to hardwarebut appliance-oriented Java, Windows, ...

programming the applicationtoolkitspaint modelsevent models

toolkitswhat they provide

interaction objects (widgets)e.g. menus, buttons, simple text areas, file selection

N.B. input and output intrinsically linked

layout supportfor resizable windows

abstractions for windowing & OS servicese.g. copy/paste, sound

toolkitshigher level of abstraction than raw WM

promotes consistency– look and feel, widget interactions

easier to code– amenable to object-oriented programming

easier to port– new versions of same platform– cross-platform

• e.g. Java for desktop apps, jQuery for web

interfaces in Java

Java toolkit – AWT (abstract windowing toolkit)

Java classes for buttons, menus, etc.

Notification based– AWT 1.0 – need to subclass basic widgets– AWT 1.1 and beyond – callback objects

Swing toolkit– built on top of AWT – higher level features– uses MVC architecture (see later)

paint modelswhat appears on the screenand when it happens

screen painting layers

> cat fred.txtThis is frd.txt> lsfred.txt tom.txtharry.c dick.java

screen

void main() {Frame f = new Frame();f.show()}

your code

your application

toolkit layer(s)

AWTSwing

other application(s)> cat fred.txtThis is frd.txt> lsfred.txt tom.txtharry.c dick.java

paint modelswhat do you draw

direct to the screendirect to screen via viewport

– clipping, coordinate transformatione.g. raw Java AWT

separate buffer– double buffer to reduce flicker, retained bitmap

option in Java Swing

display list / model– list of operations line, image etc.

toolkit worries about showing these the screene.g. OpenGL, ? MacOS display PDF, JS-DOM

bufferingin toolkit and/or window manager

e.g. Mac OS X

three levels:• nonretained

– no buffering

• retained– wm buffers

hidden parts

• buffered– app/toolkit

writes to buffer

paint modelswhen do you need to draw

internal events– data has changed– application ‘knows’

external events– user / network does things– window manager ‘knows’

… and then tells toolkit

paint modelswhen do you actually draw

internal control– code does things when it wants– but what about external events?

works OK with display list or retained bitmap

external control– called by the toolkit / window manager– but what about internal events?

… purpose of repaint() in Java

draw once per frame (game engines)– variant of external control … but code rather like internal

combined with polling model for input (check button state)

event modelswhen things happen

event models – when things happenread-evaluation loop

repeatread-event(myevent)case myevent.type

type_1:do type_1 processing

type_2:do type_2 processing

...type_n:

do type_n processingend case

end repeat

event models notification-based

void main(String[] args) {Menu menu = new Menu();menu.setOption(“Save”);menu.setOption(“Quit”);menu.setAction(“Save”,mySave)menu.setAction(“Quit”,myQuit)

...}

int mySave(Event e) {// save the current file

}

int myQuit(Event e) {// close down

}

going with the grain

notification style affects the interface– modal dialogue box

• easy with event-loop (just have extra read-event loop)

• hard with notification (need lots of mode flags)

– non-modal dialogue box• hard with event-loop (very complicated main

loop)• easy with notification (just add extra handler)

beware!if you don’t explicitly design, it will just happenimplementation should not drive design

screen your code java api

userclicks

repaint sets‘dirty’ flag

noticesdirty flag

waits formore input

…

paint(Graphics g) { // draw things}

component asked to

paint itselfg.fillRect(…)

myListener(…) { // update state repaint();}

AWT invokes Listener

asynchrony ...things happen in any order

hard to test code– usually one test user at a time– low network load– fast networks and fast machines

race conditions– unexpected event orders– sometimes may seem unlikely, but ...

Birthday surprise problem

people connections

6 15

n n(n-1)/2… …

5 10

4 6

3 3

2 1

1 0

architecture and frameworksseparation and presentation abstraction: Seeheimcomponent models: MVC and PAC

origins of separationUser Interface Management Systems (UIMS)

a level above toolkits (e.g. for non-programmers)

separation: application semantics and presentation– colours & wording different from deep functionality

improves:– portability – runs on different systems– reusability – components reused cutting costs– multiple interfaces – accessing same functionality– customizability – by designer and user

virtually extinct (!) – but the concepts live on– the Seeheim workshop ...

Seeheim model

PresentationDialogueControl

Functionality(applicationinterface)

USERUSER APPLICATION

switch

lexical syntactic semantic

conceptual vs. implementation

Seeheim– arose out of implementation experience– but principal contribution is conceptual– concepts part of ‘normal’ UI language

… because of Seeheim …… we think differently!

e.g. the lower box, the switch• needed for implementation• but not conceptual presentation dialogue application

semantic feedback

different kinds of feedback:– lexical – movement of mouse– syntactic – menu highlights– semantic – sum of numbers changes

semantic feedback often slower– use rapid lexical/syntactic feedback

but may need rapid semantic feedback– freehand drawing– highlight trash can or folder when file dragged




switch


what’s this?




switch


the bypass/switch

rapid semantic feedback

direct communicationbetween application

and presentation

but regulated bydialogue control

monolithic vs. components

Seeheim has big components

often easier to use smaller ones– esp. if using object-oriented toolkits

Smalltalk used MVC – model–view–controller– model – internal logical state of component– view – how it is rendered on screen– controller – processes user input

MVCmodel - view - controller

model

view

controller

MVC issues

• MVC is largely pipeline model: input → control → model → view → output

• but in graphical interface– input only has meaning in relation to output

e.g. mouse click– need to know what was clicked– controller has to decide what to do with click– but view knows what is shown where!

• in practice controller ‘talks’ to view– separation not complete

PAC model

• PAC model closer to Seeheim– abstraction – logical state of component– presentation – manages input and output– control – mediates between them

• manages hierarchy and multiple views– control part of PAC objects communicate

• PAC cleaner in many ways …but MVC used more in practice

(e.g. Java Swing)

PACpresentation - abstraction - control

abstraction presentation

control

A PC

A PC

A PC A P

C

... and on the web

on the websalami sliced Seeheim

between browser and serverNOT the obvious split

presentation dialogue semantics

browser

web server


between browser and server


browser

web server

css andbrowserstyling

templates, XLST

data updated onpage (esp. AJAX)

links andjavascript

serverscripts

updatingbackend

databases


between pages/scripts– consistency and control?


script 1

CSS andtemplates

script 2

script 3

script 4

dialogue flow business logic

maintaining state

Summary

Levels of programming support tools• Windowing systems

– device independence, resource sharing– managing multiple user tasks and applications

• Toolkits– programming interaction objects

• Programming the application– event paradigms: read-evaluation vs. notification– paint paradigms

• Architecture– conceptual architectures for separation– need to think about dialogue and state

implementation

Education

screen apps

toolkit window manager

windowing systems resource

toolkit worries

windowing os servicese

things window manager

screen spacededicated

code toolkit layersvoid