I think we should work on recognizing gestures in a

really constrained situation.

Why gestures?

• They can reveal themes of discourse Gestures “give clues to the underlying thematic organization of … the speaker's perspective”

(Cassell, 1998)

• They can reveal hidden reasoningGestures “identify underlying reasoning processes that the speaker cannot articulate”

(Church and Goldin-Meadow, 1986)

• They can point to given tasksGestures can be made for functional reasons: to pick up objects, to use the phone.

Which gestures?Gestures made on the desktop.

Why these?• Because they can indicate office tasks:

(typing, sorting papers, using the stapler)

• Because they can be used to command:(acknowledgement, drill down, deferral, delegation)

• Because the situation is constrained.(planar desktop, limited workspace)

• Because maybe they can indicate more.

OK, so how?

1. Track an arm on the desktop.

Fod, Mataric and Jenkins (2000)

OK, so how?

2. Track the motion of the hand and make segments where velocities are zero.

Fod, Mataric and Jenkins (2000)

And what does this get you?1. Information regarding the principal hand

motions made on the desktop (‘primitives’).2. Information regarding principal regions of the

desk that are most frequently used.3. Information that allows you to recognize new

gestures quickly. 4. Information as to what motions are rare, and

hence easy to recognize (for the purpose of deferral, etcetera).

5. A base upon which to think about: hierarchical segmentations, probabilistic gesture models, and so on.