Computer

Scie

nce

—U

niv

ersi

ty

of

Amst

erdam

Supervisor(s): Daniel Fontijne (UvA)Rein van den Boomgaard (UvA)

Signed:

Roeland Jago Douma0518476rdouma@science.uva.nl

October 1, 2008

The WII remote forObject Tracking

Bachelor Computer Science

Abstract

Nintendo’s WiiMote, the controler of Nintendo’s latest game console, contains a camera whichcan be used for a variety of applications. However all of these applications are two dimensionalsince it contains only one camera. Here we discuss a way to use two or more WiiMotes toreconstruct three dimensional data. In order to reconstruct three dimensional data we fist haveto calibrate the WiiMotes, internally and externally. We conclude that object tracking in threedimensions is possible and will allow more people to experience 3D interaction at home.

Contents

1 Introduction 31.1 Object Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1.1 Two Dimensional . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.1.2 Three Dimensional . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.2 Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

2 Hardware 72.1 Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 Buttons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.3 Player Leds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.4 Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.5 IR Camera . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.6 Unused Hardware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

3 Communication 93.1 Technical Aspect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2 Protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3.2.1 Input . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.2.2 Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.2.3 Enabling IR . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

4 Calibration 124.1 Internal Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.2 External Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

4.2.1 Essential matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134.2.2 Optimization Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144.2.3 Optimization Cost Function . . . . . . . . . . . . . . . . . . . . . . . . . . 144.2.4 Global Scaling Factor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.2.5 Calibration results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174.2.6 3D Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

5 Conclusion 205.1 WiiMote . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205.2 Calibraton . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

5.2.1 Internal Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205.2.2 External Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205.2.3 Calibration results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

5.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

A WiiMote SDP output A

1

B Geometric Algebra CB.1 Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CB.2 Geometric Product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . CB.3 Outer product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . DB.4 Subspaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . DB.5 Rotors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . D

2

CHAPTER 1

Introduction

At the end of 2006 Nintendo introduced the Nintendo Wii. The Wii is not a revolutionarymachine in terms of graphical power but it does has a revolutionary controller for a console. TheWii is controlled by the WiiMote (short for Wii Remote).

It didn’t take long before reversed engineered specification of the WiiMote appeared on theInternet. For example it appeared to connect through Bluetooth and had an infrared camera.This made it possible to track infrared sources in 2D. Great examples are given by Johnny Lee[7]. With the WiiMote Nintendo gave us a low-cost tracking device that opened a lot of newdoors.

Figure 1.1: The WiiMote from various viewpoints. The infrared camera is located at the frontof the WiiMote, behind the black IR-pass filter.

1.1 Object Tracking

The WiiMote has a built-in camera (see Figure 1.1) with an onboard processor which allows usto track up to four infrared objects. When using the WiiMote to control the Nintendo Wii it isused to track two bright (sets of) leds which are placed on top of the television set. The user holdthe WiiMote in such a way that the leds can be seen by the WiiMote. This allows the WiiMoteto (approximately) compute to which point on the television the WiiMote is pointing.

However, in this thesis we turn the WiiMote around and we will use it to look at the motionsof the user. We do this by moving leds around in front of the WiiMote camera.

The Nintendo WiiMote limits us in our freedom for object tracking since only four objects

3

can be tracked at the same time and the objects have to emit infrared light. However for use athome or in simple experiments this should not be a problem.

The advantage of the WiiMote is that the calculations for the tracking are done onboard.This means we do not have to do this ourselves. Also the use of infrared light is a positive thingsince not many things in a normal house emit strong infrared light, at least not as strong as aninfrared led.

1.1.1 Two Dimensional

When tracking in two dimensions you can suffice with one WiiMote, since one WiiMote sees thex,y plane. Great examples of two dimensional object tracking are given by Johnny Lee ([7]).

Object tracking in two dimensions limits us in several ways. We can not see how far theinfrared dot is away from the WiiMote. This is because of the fact that the WiiMote can onlysee in the x,y-plane and the estimations of the size of the blob are very inaccurate.

Also when using multiple infrared blobs, for example attacjed to the side of a pair of glasses(to use with headtracking), it is not possible to tell the difference between a person walkingaway from the the WiiMote, or turning his head. In both cases the WiiMote will only notice thedistance between the two leds getting smaller.

Figure 1.2: Two leds(red dots). Left close to the WiiMote, right far from the WiiMote

This is illustated in Figure 1.2 and Figure 1.3. The blue lines indicated the field of view ofthe WiiMote. So you can see that in both cases the WiiMote does only see two blobs gettingcloser to each other. So it is impossible to rotate your head and see objects from a different view.

1.1.2 Three Dimensional

To see in three dimensions we need at least two WiiMotes. If we know the location of theWiiMotes we can (from each WiiMote) project a ray through the center of the blobs seen. Theplace where the rays intersect is the three dimensional point where the led was located. This isvisualised in Figure 1.4.

To be able to compute the position of objects in 3D using two WiiMotes we need to find arotation and a translation from one WiiMote to the other. This is called the external calibration.See Figure 1.5. The gray area is the area where we can say something about the three dimensionaldata.

4

Figure 1.3: Two leds (red dots). Left not rotated, right rotated

Figure 1.4: Intersection of two rays from two WiiMotes. Gray stippled lines are the rays, thegreen dot is the point of intersection. i.e. the led.

5

We also need to know how ‘wide’ the WiiMotes can see (their focal length) and their opticalcenter. This called called the internal calibration.

R

t

Figure 1.5: Rotation and translation we would need to find for the external calibration.

1.2 Goal

This projects aims to use multiple WiiMotes to track objects in three dimensions. In order todo so we first describe the hardware of the WiiMote, the communication with the WiiMote andfinally the calibration.

We will show that it is possible to use two WiiMotes for three dimensional object tracking,and this will make it much more affordable to interact in three dimensions with your computerusing the WiiMote.

6

CHAPTER 2

Hardware

The WiiMote contains a variety of hardware, some of it is used in this project and some are not.The used parts are described in as much detail as possible. Since Nintendo has not released anyspecification of the hardware of the WiiMote no official reference is available. However severalpeople have opened their WiiMote and share the information they found. For more info see [2]and [1].

2.1 Communication

All the communication with the WiiMote is done through Bluetooth. This is provided by aBroadcom BCM2042 chip. The whole WiiMote is build around this chip since it is the only wayfor the WiiMote to communicate with the outside world. The chip is designed to be used withBluetooth Human Interface Device (HID) which is build on top of USB Human Interface Device(HID).

2.2 Buttons

The WiiMote consists of twelve buttons:

• 4 buttons on a D-Pad,

• A-button and B-button,

• minus-button, home button and plus-button,

• 1-button and 2-button,

• power-button.

These are binary buttons. They can only be pressed or released.

2.3 Player Leds

There are four leds on the button front of the WiiMote. Normally they are used to identify theplayers when connected to the Wii. When the leds are blinking this means the WiiMote is inBluetooth discoverable mode. The number of leds which blink also indicates the state of thebattery.

7

2.4 Memory

The WiiMote has a build in EEPROM memory. Parts of this memory are accessible to the user.A part is used to store calibration constants and personal data. Other parts are used for theperipherals on the WiiMote, a part of this memory WiiMote are the control registers. Theyallow us to write values to them which the WiiMote then reads out and will perform a certainoperations, like for example enabling the IR camera.

2.5 IR Camera

On the front of the WiiMote an monochrome camera is located. This camera has a resolutionof 128x96 pixels. This camera has an IR-pass filter in front of it, this makes sure we are onlytracking IR sources. The camera contains a built-in processor capable of tracking up to 4 objectsat the same time. The camera contains 8 times subpixel analysis which brings the “resolution”of the camera to 1024x768.

Measurements indicate that the horizontal field of view is approximately 41 degrees and thevertical field of view is approximately 31 degrees. This means that one increment in both verticalor horizontal direction corresponds with 0.04 degrees.

940nm sources are detected about twice as strong as 850nm sources, however on short distance940nm leds are not detected at all, this is probably caused by the fact that the leds used byNintendo are somewhere between 850nm and 940nm. However the exact cause is not determendsince the specification are not public. So distances of at least 80 centimeter are recommended.For better detection of dots clustering of sources can be used. Since if sources are close togetherthe WiiMotes detects 1 “big” source.

2.6 Unused Hardware

The WiiMote also contains several hardware parts that are not used in this project. They aresummed up here so it is clear that the WiiMote can do more than described in this paper.

• Motion Sensor, provides information about movements with the WiiMote.

• Rumble, lets the WiiMote vibrate to notify the user of events.

• Speaker, play sounds on the WiiMote.

• Expansion Port, you can connect several expansions to the WiiMote.

8

CHAPTER 3

Communication

The communication with the WiiMote is one of the most important aspects of this project, butit is also the weakest link in the chain since everything is reversed engineered.

3.1 Technical Aspect

As described in section 2.1 the WiiMote uses Bluetooth HID. Because of this we can connectfairly easy to the WiiMote using standardized protocol.

The WiiMotes also reacts to Bluetooth Service Discovery Protocol (SDP) and the output (seeappendix A) provides some useful information.

Name Nintendo RVL-CNT-01Vendor ID 0x057eProduct ID 0x0306Class 0x002504

The WiiMote does not use any authentication or encryption supported by Bluetooth. Inorder to connect to the WiiMote the WiiMote first has to be placed in discoverable mode bypressing the 1-button and 2-button at the same time (see section 2.2). Once the WiiMote is indiscoverable mode any Bluetooth host can connect to the WiiMote within 20 seconds.

A connection with the WiiMote is done with the L2CAP protocol. L2CAP is by default aconnection oriented protocol that reliably sends individual datagrams of fixed maximum length.The default behaviour if no acknowledgement has been received is to retransmit a package untiltotal connection failure.

L2CAP can be configured for various levels of reliability. However we want each package toarrive until the connection drops, so the default settings are used in this project.

3.2 Protocol

Because the usage of default Bluetooth HID there are some aspects about the protocol that needclarification. The connection that is established to the WiiMote consists of two channels (defaultBluetooth HID profile). A control channel and an interrupt channel. The control channel is usedto send data from the host to the WiiMote and the interrupt channel is used to send data fromthe WiiMote to the host. This is asynchronous so data can be transmitted at the same time onboth channels.

After connecting to the two channels it is important to read from the interrupt channel.The WiiMote will not listen to any commands on the control channel before you read from theinterrupt channel the first time.

When protocol examples are given each byte is written out in hexadecimal without the leading0x with spaces between the bytes.

9

3.2.1 Input

Input commands are commands sent from the WiiMote to the host. All input commands startwith 0xA1.

Default reading

In some cases for example when writing data to a control register (see 2.4) the WiiMote returnssome information. However this is not yet reversed engineered. However it is assumed that ifthe WiiMote returns something the write was successful.

IR Data

When the IR cameras are enabled IR Data is transmitted at 100 hertz. The reporting modeis 0x33. The first two data bytes are the buttons in a bitmask. The next 3 bytes are for theaccelerometer and the rest of the data bytes contain IR information. So the protocol looks like:� �

1 (A1) 33 BB BB AA AA AA II II II II II II II II II II II II� �Listing 3.1: 0x33: IR Data Reporting

Since we are only interested in the IR data part the rest will not be displayed now. The IRdata can contain up to 4 dots. So 3 bytes per dots which have the following representation:� �

1 xxxxxxxx yyyyyyyy yyxxssss� �Listing 3.2: 3 byte dot representation

The last byte contains the most significant bytes for the x and the y coordinates. Furthermoreit contains the size. This is an indication of the size of the dot detected.

3.2.2 Output

Output commands are command send from the host to the WiiMote. In other words we controlthe WiiMote with these commands. All output commands start with 0x52.

Enable IR Camera’s

There is only IR camera. However the WiiMote needs two switches to be enabled so we will callthem IR camera 1 and IR camera 2. The first camera resides on ID 0x13 and the second cameraon 0x1A. Both can be enables by writing 0x04 to their ID.� �

1 (52) 13 042 (52) 1A 04� �

Listing 3.3: Enable IR Camera 1 and IR Camera 2

Writing to control registers

There are several ways to write to several of the registers. However only writing to the controlregisters of the IR cameras explained.� �

1 (52) 04 AA AA AA NN XX� �Listing 3.4: Writing to control Register

• AA AA AA is the address to write to.

• NN is the number of bytes following.

10

• XX is a sequence of NN bytes with the data to write to AA AA AA

For examples see 3.2.3.

Setting data reporting mode

There are several data reporting modes on the WiiMote. We want the mode with IR Data (see3.2.1). To do this we have to set the data reporting mode on the WiiMote. This has the followingsyntax:� �

1 (52) 12 TT MM� �Listing 3.5: Syntax setting data reporting mode

Bit two of TT specifies whether continuous data reporting is desired. This means that if bittwo is set data is reported all the time. And if the bit is not set only data is reported when thereis data to report.

MM specifies the data reporting mode.

3.2.3 Enabling IR

To enable the IR data reporting several steps are needed. This is first displayed in text:� �1 Enable IR Camera 12 Enable IR Camera 23 Write 0x08 to register 0xb000304 Write Sensitivity Block 1 to registers at 0xb000005 Write Sensitivity Block 2 to registers at 0xb0001a6 Write Mode Number to register 0xb000337 Write 0x08 to register 0xb000308 Set data reporting mode to 0x33� �

Listing 3.6: Enable IR Data reporting - text

Now the commands send to the WiiMote:� �1 (52) 13 042 (52) 1A 043 (52) 04 B0 00 30 01 084 (52) 04 B0 00 00 09 00 00 00 00 00 00 90 00 415 (52) 04 B0 00 1A 02 40 006 (52) 04 B0 00 33 01 337 (52) 04 B0 00 30 01 088 (52) 12 00 33� �

Listing 3.7: Enable IR Data reporting - commands

11

CHAPTER 4

Calibration

To compute 3D information using the WiiMotes we need to calibrate them. This means we needto find a translation and a rotation from one WiiMote to the other so three dimensional datacan be reconstructed. In order to do this we need the rotation and translation from WiiMote 1to WiiMote 2. This is illustrated in Figure 4.1. The gray area is the space we want to retrieve3D data from.

R

t

Figure 4.1: Rotation and translation we would like to find

4.1 Internal Calibration

Before we can compute the translation and rotation we need a suitable internal calibration foreach camera. The internal calibration specifies properties of the camera such as focal length andthe barrel distortion of the camera image.

To internally calibrate ordinary cameras for stereo vision you can (for example) use thecommon chessboard method as described by Zhang in [8]. This however is not straightforwardwith a WiiMote since it cannot output actual camera images; only 4 IR dots can be reported atthe same time. Thus we estimated the internal calibration parameters using the specificationsand the measured field of view.

12

An internal calibration matrix typically looks like:

C =

fx 0 cx0 fy cy0 0 1

(4.1)

• fx: focal length

• fy: focal length

• cx: x coordinate of the optical center of the camera

• cy: y coordinate of the optical center of the camera

The matrix in Equation 4.1 gives the relation between world point x and image point u:

u = Cx. (4.2)

The values for matrix 4.1 can be calculated from the hardware specification we have (see Sec-tion 2.5). We assume the optical center is in the center of the visible field, i.e., at (512, 384).The fx and fy are generally the same so we also assume that they are. We calculate the focallength (in pixels) as follows:

fx =0.5 ∗ “horizontal resolution′′

tan(0.5 ∗ π∗“horizontal field of view′′

180 )(4.3)

When we insert the values of Section 2.5 into Equation 4.3 the focal length is calculated. Sothe estimated internal matrix for a WiiMote is approximately:

C =

1369.41 0 5120 1369.41 3840 0 1

(4.4)

We assume no pin cushion or barrel distortion in the camera image. This estimation turned outto work good enough for our purposes (see Section 4.2.5).

4.2 External Calibration

We can externally calibrate the WiiMotes by waiving an IR marker through the field of view.The WiiMotes record the 2D image location of the dots at 100 frames per second. One dotshould be visible for each frame for each camera. If we capture enough frames over a suitablevolume these measurement contain enough information to compute the translation and rotationof the WiiMotes.

Computing the external calibration is performed in two stages. First the essential matrix iscomputed to get a rough estimation of the translation and rotation. Secondly the translation androtation parameters are optimized. So far we have only used two WiiMotes but the optimizationalgorithm works for any number of cameras.

4.2.1 Essential matrix

The essential matrix is a 3x3 matrix which relates corresponding image points in stereo imagesassuming that the cameras satisfy the pinhole camera model. The essential matrix is calculatedby using the Eight-point algorithm (see [4]). We feed all the corresponding points from bothcameras to the algorithm. The more points provided the better the estimation of the essentialmatrix will be.

From the essential matrix we extract an estimate of the translation and rotation of theWiiMotes.

13

4.2.2 Optimization Algorithm

The essential matrix gives us a rough estimate of the translation and rotation. If more than twoWiiMotes are used, it pays off to optimize these estimates because the essential matrix can onlybe computed for pairs of cameras. But even in the case of just two WiiMotes, optimization isuseful because the essential matrix finds an algebraic minimum, instead of a geometric minimum(see [4]).

Notation

Below, N is the number of points and m is the number of cameras. Note that m = 2 is our case,but the method works for any number of cameras larger than 1.

The variables used are the following:

• xij is the 2D location of the point in the camera image.

• Xi is the point in 3D in world coordinates.

• tj is the translation we want to know.

• Rj is the rotation we want to know.

• Xij3 is the distance we have to move along the ray going from the optical center of thecamera j through the image point xij in order to arrive at the 3-D point Xi. Following

this we want: tj +RjXij3xij∼Rj= Xi.

Here i is the index of the points and j is the index of the camera.The optimization of the translation and rotation is done using geometric algebra. For a short

introduction to geometric algebra see appendix B.

4.2.3 Optimization Cost Function

The optimization is performed as described in Lasenby et al. [6], which initially provides thefollowing equation to minimize:

S1 =m∑j=1

N∑i=1

[xij −∼Rj (Xi − tj)Rj

[∼Rj (Xi − tj)Rj ] ∗ e3

]2. (4.5)

Equation 4.5 minimizes the sum of the squared distances in the images plane of each camera.However this does not mean that the error of the reconstructed point is also minimized. Also thepresence of the parameters we are trying to estimate in the denominator of the right hand sideof the equation does not make this equation an easy one to solve. So instead Lasenby proposesto use the following function:

S2 =m∑j=1

N∑i=1

[Xij3xij−∼Rj (Xi − tj)Rj ]2. (4.6)

With Equation 4.6 we try to minimize the distance between the multi camera reconstructionpoint Xi and the single camera reconstruction points Xij3xij . Equation 4.6 represents a costfunction in the world. It is the sum of the squared distances between the world points (Xi) andtheir closest points on the camera rays projecting out of the optical center through the point xij .

This means while Equation 4.5 minimizes the error for each camera but not necessarilyminimizing the error in the world. The Equation 4.6 minimizes the error in the world (seeFigure 4.2). To minimize this function we can differentiate the function with respect to each ofthe parameters and equate it to 0. I.e., we use partial differentiation, hence assuming all othervariables are correct when we are differentiating. We perform partial differentiation with respectto each of the four different sets of variables:

• tj : the translation.

14

S2

S1

Figure 4.2: Cost function S1 minimizes the error in each camera while cost function S2 minimizesthe error in the world.

• Rj : the rotation.

• Xij : the distance from the camera to the 3D points.

• Xi: the 3D coordinates.

We have an initial estimation (see Section 4.2.1) and optimize each variable in turn, keeping theothers constant. The optimization is done by equating the partial derivative of Equation 4.6with respect to each of the four sets of variables to zero. We loop over all parameters until theyconverge.

Differentation with respect to tk

Let Yik ≡ {Xik3xik−∼Rk (Xi−)Rk}. Then the partial equation, ∂tkS2 = 0, gives us:

∂tkS2 = 2N∑i=1

ej∂

∂tjk{∼Rk tkRk} ∗Yik

= 2Rk[N∑i=1

Yik]∼Rk= 0

⇒N∑i=1

Yik = 0. (4.7)

This gives us equation 4.8.

tk =1nk

N∑i=1

[Xi −Xik3Rkxik∼Rk]. (4.8)

I.e., tk is the average position of the camera with respect to the 3D points X. This is illustated inFigure 4.3. The figure is in 2D but the real problem is in three dimensions, however the problemremains the same in principle. We want to translate the WiiMote (the blue dots) so that the

15

Figure 4.3: Optimisation of tk. Red dots are Xi and blue dots are Xik3Rkxik∼Rk.

distance between the red and blue dots is minimized. Equation 4.8 takes the 3D points (Xi) andsubstracts the reconstruction of the 3D points (Xik3Rkxik

∼Rk). This is done for each WiiMote.

As you can see in figure 4.3 there is no exact match. This is because the input data is noisy.However we try to find the best fit using least square estimation.

Differentation with respect to Rk

Rk is the rotation of the WiiMote. We want to find a rotation so that the sum of the distancesbetween the 3D points and the reconstructed points is minimized. This problem is known as theorthogonal Procrustes problem which is described in [3].

∂RkS2 = ∂Rk

N∑i=1

(vik−∼Rk uikRk)2 = 0, (4.9)

where vik = Xik3xik and uik = Xi − tk. As [6] shows, this ultimately leads the well known(Procustes) solution, which uses the SVD:

∼Rk = V UT ,

F k = USV T ,

F kαβ =N∑i=1

(eα∗∼uik)(eβ∗

∼v ik), (4.10)

where F and R are a 3× 3 matrices. Note that we need to convert R from matrix to a rotor.The problem is visualised in figure 4.4. Again the real problem is 3D which means we have

three axes (the x-axis, the y-axis and the z-axis) to rotate the WiiMote. So the problem in threedimensions is more difficult. Also you can see that we do not have an exact match, again becauseof noisy input data.

This optimalisation is also done for each WiiMote.

Differentation with respect to Xpq3

Here we calculate the distance of each the camera to each of the 3D points.

∂Xpq3S2 = ∂Xpq3{Xpq3xpq−∼Rq (Xp − tq)Rq}2

= 2{Xpq3xpq−∼Rq (Xp − tq)Rq} ∗ xpq

= 0. (4.11)

16

Figure 4.4: Optimisation of Rk. The red dots are 3D points, the blue dots are reconstructedpoints and the green dots are the position after rotation.

Equation 4.11 gives us:

Xpq3 =[∼Rq (Xp − tq)Rq] ∗ xpq

x2pq

≡ (Xp − tq)[Rqxpq∼Rq]

x2pq

. (4.12)

The problem is visualised in figure 4.5. We are trying to find the distance of the WiiMotefrom the 3D points (Xi). This is done by using the rotation to determine the direction we arelooking at. Then we substract the translation of the WiiMote from the 3D point.

Differentation with respect to Xk

This is reconstruction of the 3D points. By expanding S2 (equation 4.6) we can get the derivativewith respect to Xk, for k from 1 to N.

∂XkS2 = ∂Xk

m∑j=1

{−2Xkj3(Rjxkj∼Rj) ∗Xk + (Xk − tj)2}

= 2m∑j=1

[−Xkj3Rjxkj∼Rj +(Xk − tj)] = 0 (4.13)

Equation 4.13 can be rearranged to give us:

Xk =1mk

m∑j=1

[tj +Xkj3Rjxkj∼Rj ]. (4.14)

Of course this must satisfy mk 6= 0.

4.2.4 Global Scaling Factor

Note that after external calibration do not know anything about the world scale. So all resultsare relative to each other. If you want to know the scale of the wolrd you can measure (forexample) the distance between the WiiMotes and use that in you calculations.

4.2.5 Calibration results

When we have successfully calibrated the WiiMotes there is of course the question of how goodthis calibration is. We do this by using the calibration data.

17

Figure 4.5: Optimisation of Xpq3. The reds dots are Xp. The blue dots are xpq. Xpq3 is thewhole line from the WiiMote to the red dots.

18

We project the rays from the WiiMotes through the calibration points. Most of the time theserays do not intersect exactly but cross each other. By calculating the average of the shortestdistance between the rays projecting from each WiiMote we can get an indication of the precisionof our setup.

In order to give a metric meaning to the result we need to know how far the WiiMotes areaway from each other. In this experiment 60 cm. We performed a calibration and measured anaverage distance between the rays is 1.6 mm. This seems is good enough to be useable in threedimensional interaction applications.

Note that this value says something about both the noise and the precision of the internalcalibration of the WiiMotes. If there was no noise and the internal calibration was perfect, theexpected distance between the rays would be 0 mm. However, when the internal calibration isoff, the result of the external calibration can never be perfect. Noise will also increase the averagedistance between the rays.

4.2.6 3D Reconstruction

Once the external calibration is known we can directly reconstruct 3D points. Using Equa-tion 4.14 and substituting Equation 4.12 in that to eliminate the Xpq3. To simplify the notation

we write wij = Rjxkj∼Rj which gives the following equation:

Xk =1mk

m∑j=1

[tj +1x2kj

{(Xk − tj) ∗ wij}wij ]. (4.15)

If we take the inner product of the above equation with ei with i = 1, 2, 3, we can rearrange thatto:

Xk∗ [ ei −1mk

m∑j=1

1x2kj

(wkj ∗ ei)wkj ] =

1mk

m∑j=1

[tj ∗ ei −1x2kj

(wkj ∗ tj)(wkj ∗ ei)]. (4.16)

We have 3xN of these equations (since k = 1, .., N and i = 1, 2, 3). For each k we can constructa matrix equation for Xk.

AkXk = bk =⇒ Xk = A−1k bk, (4.17)

where the matrix Ak and the vector bk are given by

Akip = δip −1mk

m∑j=1

1x2kj

(wkj ∗ ei)(wkj ∗ ep) (4.18)

bk ∗ ei = bki =1mk

m∑j=1

tj [ei −1x2kj

(wkj ∗ ei)wkj ]. (4.19)

This shows that is we have our external calibration, rotation (Rs) and translation (ts), thenthrough Equation 4.17 we can quickly reconstruct 3D world point with a method that uses allof the available data at once.

19

CHAPTER 5

Conclusion

The goal of tracking objects with the WiiMote in three dimensions is successful. However cali-brating the WiiMote is not a trivial case and calibrating is (not yet) user-friendly. But we havesuccessfully calibrated the two WiiMotes to generate 3D data. So it is possible to track 3Dobject. However, we have not yet written any applications which actually perform tracking.

5.1 WiiMote

When an abstraction is made the WiiMote can be seen as an infrared camera that can track upto 4 infrared sources at the same time. The resolution of the camera is 1024x768.

The WiiMote communicates with a host through standard Bluetooth Human Interface Device(HID). Which makes connecting with the WiiMote an easy task. The protocol is for a large partreversed engineered which allows us to get the infrared data from the WiiMote in a relativelyeasy way.

5.2 Calibraton

To reconstruct three dimensional data we need to calibrate the WiiMotes. This is done in twosteps. Internal calibration which is the same for both WiiMotes and external calibration whichis the rotation and translation from one WiiMote to the other.

We generate a bulk of data from the two WiiMotes by waiving an infrared light source throughthe space we want to calibrate the WiiMotes on.

5.2.1 Internal Calibration

Internal calibration for the WiiMote cannot be done using the now widely used techniques de-scribed by Zhang [8]. However an estimation can be made with the hardware information of theWiiMote. There is no simple, practical way of experimentally verifying this matrix, however theresult at the end of the calibration process seems to be good enough (see section 5.2.3).

5.2.2 External Calibration

Since we initially have no “real” knowledge about the three dimensional position and orientationof the WiiMotes, we need to estimate the translation and rotation. This is done using theEight-point algorithm ([4]).

However this estimation is not optimal so we want to optimize it. This is done as describedin Lasenby et al. ([6]). We in turn optimize one set of variables: the translation, the rotation,the distance of the camera from the three dimensional points and the actual three dimensionalpoints.

20

5.2.3 Calibration results

We measured the precision of the calibration. We did this by reusing the calibraton data andcalculating the average distance of the rays projected from the WiiMotes. This turns out to be1.6 mm which makes this setup useable for use in three dimensional interaction.

5.3 Future Work

Now that the calibration is done. Applications using two (or more) WiiMotes can be developed.These applications can let the user interact in three dimensions with them.

For example a program for high school students which will allow them to drop an infraredball. The program can then return various things like the speed of the ball, the path of the balletc. When combined with head tracking the same path can be viewed from slightly differentangles.

Of course this can already be done with other setups however the WiiMotes are relativelycheap and will allow a lot more people to experience 3D interaction at home.

21

APPENDIX A

WiiMote SDP output

� �1 Service RecHandle: 0x02 Service Class ID List:3 "SDP Server" (0x1000)4 Protocol Descriptor List:5 "L2CAP" (0x0100)6 PSM: 17 "SDP" (0x0001)8 Language Base Attr List:9 code_ISO639: 0x656e

10 encoding: 0x6a11 base_offset: 0x10012 Profile Descriptor List:13 "" (0x0100)14 Version: 0x01001516 Service Name: Nintendo RVL -CNT -0117 Service Description: Nintendo RVL -CNT -0118 Service Provider: Nintendo19 Service RecHandle: 0x1000020 Service Class ID List:21 "Human Interface Device" (0x1124)22 Protocol Descriptor List:23 "L2CAP" (0x0100)24 PSM: 1725 "HIDP" (0x0011)26 Language Base Attr List:27 code_ISO639: 0x656e28 encoding: 0x6a29 base_offset: 0x10030 Profile Descriptor List:31 "Human Interface Device" (0x1124)32 Version: 0x01003334 Service RecHandle: 0x1000135 Service Class ID List:36 "PnP Information" (0x1200)37 Protocol Descriptor List:38 "L2CAP" (0x0100)39 PSM: 140 "SDP" (0x0001)

A

41 Profile Descriptor List:42 "PnP Information" (0x1200)43 Version: 0x0100\right)� �

Listing A.1: Output of ”sdptool browse”

B

APPENDIX B

Geometric Algebra

In order to understand the calibration method of Chapter 4, some basic knowledge of geometricalgebra is required. We will give a short introduction to geometric algebra in this appendix.Most information is gathered from Dorst et. al ([5]). Only the aspects which matter to thecalibration algorithm will be covered, which is a small subset of geometric algebra.

B.1 Basics

A geometric algebra is a Clifford algebra constructed over a vector space Vn that includes ageometric product. The elements of geometric algebra are multivectors. In our case, the originalvector space V is constructed over the real numbers. A vector v is an element of V.

The geometric product allows us to combine vectors to new elements. These element can beused in further computation as a whole.

B.2 Geometric Product

Given three multivectors A, B and C a geometric product has to following defining properties:

1. Distributivity

• A(B + C) = AB + AC

• (A + B)C = AC + BC

2. Associativity

• (A + B) + C = A + (B + C)

3. 1A = A

4. For any vector v, v2 is a scalar.

5. Commutativity of a geometric product by a scalar

• λA = Aλ

Note that the geometric product is not commutative in general.For vectors, the geometric product can also be defined as the sum of a inner product and a

outer productab = a · b+ a ∧ b, (B.1)

where the outer product will be described shortly. The definition (equation B.1) of the geometricproduct and the associativity allows us to solve vector algebra equations.

A geometric object like a ∧ b is called a blade. A blade can be visualised as a segment of aplane (a parallellogram) with orientation.

C

B.3 Outer product

The outer (or wedge) product has the following properties:

1. Anti-symmetry

• a ∧ b = −b ∧ a

2. Scaling

• a ∧ (βb) = β(a ∧ b)

3. Distributivity

• a ∧ (b+ c) = (a ∧ b) + (a ∧ c)

4. Associativity

• (a ∧ b) ∧ c = a ∧ (b ∧ c)

Note that if a = b this gives a∧a = −a∧a this can only be true if the square of a is zero. Whichis true since a does not span any planear element with itself.

The outer product of vectors is the anti-symmetric part of the geometric product:

a ∧ b =12

(ab− ba).

With the blades constructed using the outer product we can represent subspaces in ourmultivector space.

B.4 Subspaces

A subspace is created by using the outer product on multiple vectors. This can for example giveus a volume element. We do this by first combining two vectors to a planar element (2-blade),and then combining that plane with an other vector to create a volume (3-blade).

This shows the power of subspaces. We can use subspaces as elements of computation. Notonly when combining them, but we can rotate the whole subspace at once.

B.5 Rotors

With a rotor we can rotate for example vectors or general subspaces. Geometrically, the trans-formation a′ = rar represents a reflection in a plane perpendicular to r. To create a rotation weneed two of such reflections.

a′ = rar

a′′ = sa′s

a′′ = (sr)a(rs) =R a∼R (B.2)

R≡ sr is called a rotor. It rotates in the plane spanned by r and s, over twice the angle betweenthem.

∼R≡ rs is called the reverse of R, this because is it obtained by reversing the order of the

geometric products.

D

Bibliography

[1] Wiibrew. http://wiibrew.org/w/index.php?title=Wiimote.

[2] Wiili.org wii linux. http://www.wiili.org/index.php/Wiimote.

[3] G.H. Golub and C.F. Van Loan. Matrix Computations. JohnsHopkinsPress, Baltimore, MD,third edition, 1989.

[4] R. I. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision, 2nd edition.Cambridge University Press, 2004.

[5] D. Fontijne L. Dorst and S. Mann. Geometric Algebra for Computer Science. Elsevier Inc.,2007.

[6] J. Lasenby and A. Stevenson. Using geometric algebra in optical motion capture. GeometricAlgebra with Applications in Science and Engineering, 2001.

[7] J. Chung Lee. Wiimote tracking. http://www.cs.cmu.edu/~johnny/projects/wii/.

[8] Z. Zhang. A flexible new technique for camera calibration. Technical Report MSR-TR-98-71,1998.

i