SM1205 Interactivity

Topic 11: Motion Tracking Part III

Spring 2010 SCM-CityU 1

Button Click

• Usually done via mouse click

Spring 2010 SCM-CityU 2

Button Click • Can we click digital buttons by our fingers directly?

• Answer: Yes!– Through motion

– Of course, you canalso rely on touchsensors• Need extra hardware

Spring 2010 SCM-CityU 3

Today’s Examples

• Motion-driven button interaction

Spring 2010 SCM-CityU 4

Today’s Examples

• Motion-driven button interaction

Spring 2010 SCM-CityU 5

Motion Tracking Library

• In cityu.scm package (developed by Hongbo Fu)

– Webcam class • An easy-to-use wrapper for webcam access

– MotionTracker class• Main class for motion tracking

–MotionTrackerButton class • Use motion to trigger button clicking

Spring 2010 SCM-CityU 6

Key Idea• Analyze motion over a button on the stage– If motion is detected over it, dispatch OVER event

• Cf. MouseEvent.MOUSE_OVER

– If motion continuously exists for a while, dispatch CLICKED event• Cf. MouseEvent.MOUSE_DOWN• To avoid undesired clicks after dispatching CLICKED event, clicking is

disabled for a short period (say 2 sec)

– If clicking is re-enabled, dispatch CLICKABLE event• Cf. MouseEvent.MOUSE_UP

Spring 2010 SCM-CityU 7

• To create a motion-driven button, you need – Step 1: create some display object (e.g., btn) on stage

• By either design tool or ActionScript programming

– Step 2: create a MotionTrackerButton object and associate it with btn

– Step 3: add event listeners for motionBtn • MotionTrackerButton.OVER MotionTrackerButton.CLICKED MotionTrackerButton.CLICKABLE

MotionTrackerButton Class

Spring 2010 SCM-CityU 8

var motionBtn:MotionTrackerButton = new MotionTrackerButton(webcam,

btn);

var motionBtn:MotionTrackerButton = new MotionTrackerButton(webcam,

btn);

Step 1: Create Display Object• Draw some shape on stage• Convert it to symbol • Assign an instance name (e.g., btn)

Spring 2010 SCM-CityU 9

Review: Instance Name vs Class Name

• Symbol instance name (cf. variable in AS) – Use this when you want to directly access this symbol

instance in AS (e.g., btn.x = 20;)

• Symbol class name (cf. complex data type in AS)– Use this when you want to dynamically create (possibly

many) symbol instances in AS• E.g., var btn2: MyButton = new MyButton();

Spring 2010 SCM-CityU 10

Step 2: Create MotionTrackerButton Object

• Before adding a MotionTrackerButton object, we need to initialize a Webcam object– We don’t need to explicitly create a MotionTracker object,

since it is automatically generated for the associated button

Spring 2010 SCM-CityU 11

import cityu.scm.Webcam; // import Webcam class

var webcam:Webcam = new Webcam(640, 480); webcam.addEventListener(Webcam.READY, onCameraReady);

function onCameraReady(e:Event): void { addChild(webcam); // to avoid occlusion of stage objects by webcam webcam.sendToBack(); webcam.alpha = 0.4;

}

import cityu.scm.Webcam; // import Webcam class

var webcam:Webcam = new Webcam(640, 480); webcam.addEventListener(Webcam.READY, onCameraReady);

function onCameraReady(e:Event): void { addChild(webcam); // to avoid occlusion of stage objects by webcam webcam.sendToBack(); webcam.alpha = 0.4;

}

Step 2: Create MotionTrackerButton Object

• First import MotionTrackerButton class

• Then create a button with motion control

– 3rd parameter: amount of motion needed to activate clicking• How easy a button can be clicked

– 4th parameter: time (in milliseconds) needed to re-enable clicing after dispatching CLICKED event • How often a button can be clicked

Spring 2010 SCM-CityU 12

import cityu.scm.MotionTrackerButton; import cityu.scm.MotionTrackerButton;

var motionBtn:MotionTrackerButton =

new MotionTrackerButton(webcam, btn, 10, 500);

var motionBtn:MotionTrackerButton =

new MotionTrackerButton(webcam, btn, 10, 500);

Step 2: Create MotionTrackerButton Object

• Motion detection is clipped to the button area only – To see tracking visualization, make the button a bit

transparent

Spring 2010 SCM-CityU 13

Step 3: Add Event Listeners• Event types – MotionTrackerButton.OVER– MotionTrackerButton.CLICKED– MotionTrackerButton.CLICKABLE

Spring 2010 SCM-CityU 14

motionBtn.addEventListener(MotionTrackerButton.OVER, onButtonOver);

motionBtn.addEventListener(MotionTrackerButton.CLICKED, onButtonClick);

motionBtn.addEventListener(MotionTrackerButton.CLICKABLE, onButtonClickableAgain);

motionBtn.addEventListener(MotionTrackerButton.OVER, onButtonOver);

motionBtn.addEventListener(MotionTrackerButton.CLICKED, onButtonClick);

motionBtn.addEventListener(MotionTrackerButton.CLICKABLE, onButtonClickableAgain);

function onButtonClick(e:Event): void {

trace("button clicked"); }

function onButtonClick(e:Event): void {

trace("button clicked"); }

Add More Visual Effects• Idea: create multiple frames for the button and go to

certain frame when a certain condition is satisfied – The button in our example is just a movie clip

• Review: MovieClip’s properties and methods (ref)– Note: we can access a frame by either frame number or label

Spring 2010 SCM-CityU 15

Properties•currentFrame: int•currentLabel: String •totalFrames: int

Properties•currentFrame: int•currentLabel: String •totalFrames: int

Methods•stop(), play()•nextFrame(), prevFrame()•gotoAndStop(frame)

• frame is a number or string•gotoAndPlay(frame)

Methods•stop(), play()•nextFrame(), prevFrame()•gotoAndStop(frame)

• frame is a number or string•gotoAndPlay(frame)

Add More Visual Effects• Modify button symbol to have two key frames• Frame 1– Use the original shape – Frame label: up

• Frame 2– Add some shadow filter– Frame label: down

Spring 2010 SCM-CityU 16

Class Exercise• Task 1: when button is clicked by motion – If btn’s current frame label is “up”, go to frame “down”– If btn’s current frame label is “down”, go to frame “up”– Tips

• Add code in onButtonClick • Use currentLabel, gotoAndStop

Spring 2010 SCM-CityU 17

Class Exercise• Task 2: play with different values for 3rd and 4th

parameters of MotionTrackerButton. – 3rd parameter: how easy a button can be clicked– 4th parameter: how often a button can be clicked

• Compare – MotionTrackerButton(webcam, btn, 1, 500);– MotionTrackerButton(webcam, btn, 20, 500);

• Compare – MotionTrackerButton(webcam, btn, 5, 50);– MotionTrackerButton(webcam, btn, 5, 2000);

Spring 2010 SCM-CityU 18

Example: Piano with Motion Control

• How to interact multiple MotionTrackerButton objects?

Spring 2010 SCM-CityU 19

Class Exercise • Open MotionTracker-Piano-Skeleton.fla and insert a new symbol

• Frame 1 of this new symbol

• Frame 2 of this new symbol – No AS code needed for this frame

Spring 2010 SCM-CityU 20

Class Exercise

• Create 7 instances of the new symbol on stage– Instance names from left to right: key1, key2, key3,

key4, key5, key6, key7• Alternatively, you can create those instances in AS using

for loop

Spring 2010 SCM-CityU 21

Music Notes• We use an array to store music notes

Spring 2010 SCM-CityU 22

var notes:Array = [new S1(), new S2(), new S3(), new S4(),

new S5(), new S6(), new S7()];

var notes:Array = [new S1(), new S2(), new S3(), new S4(),

new S5(), new S6(), new S7()];

Create Motion Buttons • We use arrays to store both piano keys and their

corresponding motion buttons – Corresponding key and motion button have same index

Spring 2010 SCM-CityU 23

// a list of motion buttons. e.g., keysMB[0] is for keys[0] var keysMB:Array = new Array(); var keys:Array = [key1, key2, key3, key4, key5, key6, key7];

// a list of motion buttons. e.g., keysMB[0] is for keys[0] var keysMB:Array = new Array(); var keys:Array = [key1, key2, key3, key4, key5, key6, key7];

for (var i:uint = 0; i < keys.length; i++) { keysMB.push(new MotionTrackerButton(webcam, keys[i], 5, 500)); }

for (var i:uint = 0; i < keys.length; i++) { keysMB.push(new MotionTrackerButton(webcam, keys[i], 5, 500)); }

Event Listeners

Spring 2010 SCM-CityU 24

for (var i:uint = 0; i < keys.length; i++) { // motion-driven

keysMB[i].addEventListener(MotionTrackerButton.CLICKED, onKeyPressed);

keysMB[i].addEventListener(MotionTrackerButton.CLICKABLE, onKeyReleased);

}

for (var i:uint = 0; i < keys.length; i++) { // motion-driven

keysMB[i].addEventListener(MotionTrackerButton.CLICKED, onKeyPressed);

keysMB[i].addEventListener(MotionTrackerButton.CLICKABLE, onKeyReleased);

}

function onKeyPressed(e:Event): void { for (var i:uint = 0; i < keys.length; i++) {

if (keysMB[i] == e.target) { trace("index of pressed key", i);

} }

}

function onKeyPressed(e:Event): void { for (var i:uint = 0; i < keys.length; i++) {

if (keysMB[i] == e.target) { trace("index of pressed key", i);

} }

}

Extension to Other Instruments

Spring 2010 SCM-CityU 25