UI Issues, Neural Nets, RTS

CSE 490RAJanuary 27, 2005

Lecture outline

Leftovers from UI Lecture Neural Networks RTS

Uses of the stylus

Direct writing Abstract writing Pointing Selecting Gesture Direct manipulation Control

Mode issues in pen computing Adding modes to the pen

Barrel button Secondary button with non-dominant hand Eraser tip Pressure Explicit mode buttons

Area based modes (writing area, gesture area, control area, etc.)

Cursor feedback Importance of visual cues for informing user Errors in crossing mode boundaries

Control

Widgets design for stylus use

Flow Menu

Use movement through octants for control information

ItemText

Shape

Item

Move

ZoomHighlight

Custom

400%

200%

100%

66.6%

50%

800%25%

Custom

400%

200%

100%

66.6%

50%

800%25%

75.0 100.0

Interaction with direct manipulation

ItemMove

ZoomHighlight

CrossY: Crossing based UI Specify

operations by drawing through

Gestures Commands issued with a single

stroke May be drawn or invisible Support from SDK

Register gestures to be recognized UI Issues

Similar to keyboard short cuts Speed up for experts Hard to learn / remember

Gestures Ambiguity

Distinction between gestures Distinction between gesture and other ink

Robustness Handling misrecognized gestures

False positive False negative

Gesture initiated actions should be undoable

Neural Networks

Fundamentals for Handwriting Reco Lecture (Jay Pittman)

Recognition algorithm

Learning based recognition algorithm

General considerations for learning algorithms Training sets

Collection Evaluation

Training cost Time and space

Algorithm cost Time and space

Robustness to error

Neural networksPerceptrons Motivated by considerations of the brain

Single layer neural networks

Bias weights Threshold activation function

Step function Sigmoid function: 1/(1 + e-x)

What you can do with single layer networks

Any linearly separable dataset can be recognized with a single layer neural network

Gradient descent algorithm

Choose initial weights While not at optimum

Compute derivative Move along derivative

It can be proved this converges

However, single layer networks are very limited

Multilayer networks with hidden nodes

Can recognized much wider range of data set

The gradient descent algorithm generalizes to this case

Real Time Stylus

Allow for user computation on the ink thread

Architecture (Overview)

Pen Service

RealTime Event Sink

RealTime Event Sink

RealTimeStylus

queue

RealTime Event Sink

“Inking” Thread UI Thread

Ink CollectingObject

storage

InkCollector

Substroke operations

Examples Custom Inking Multiple Ink Distributed Ink

Rendering Dynamic – draw on ink packet Static – draw on paint event

Custom Inking Create plugin to

listen for packets Registor for

Packets Draw triangle on

each packet

public void Packets(RealTimeStylus sender, PacketsData data){ for (int i = 0; i < data.Count; i += data.PacketPropertyCount){ Point point = new Point(data[i], data[i+1]); // Packet data always has x, y

// followed by the rest

Point convertedPoint = new Point(); // We need to convert to Pixels... convertedPoint.X = (int)Math.Round((float)point.X * (float)myGraphics.DpiX/2540.0F); convertedPoint.Y = (int)Math.Round((float)point.Y * (float)myGraphics.DpiY/2540.0F);

if (this.pointCount == 0)this.firstPoint = convertedPoint;

if (this.pointCount > 1){ Color color = pointCount % 2 == 0 ? Color.Red : Color.Yellow;Point[] triangle = new Point[3];triangle[0] = this.firstPoint;triangle[1] = this.previousPoint;triangle[2] = convertedPoint;

myGraphics.FillPolygon(new SolidBrush(color), triangle); } this.pointCount++; this.previousPoint = convertedPoint; }}

Distributed Ink

Capture ink packets on machine 1 Send packets to machine 2 Reconstruct ink stroke by inserting

packets

Sample application

Distributed Ink

Sender: Collect packets in packet, when count

is above threshold, send message Receiver:

Insert packets as custom data Render custom data as it is received Assemble custom data as an ink

stroke