intro.to .kinect.sdk

7/29/2019 Intro.to .Kinect.sdk

1/46

Introduction to Kinect SDK Beta

CS4201 Interactive System Project Lab 1

Instructor: Dr. Shengdong Zhao

NUS-HCI Lab

Slides prepared by Lan Ziquan

1


2/46

Agenda

Introduction

Preparation

Explanation for sample code Practice together

Show your understanding

2


3/46

From http://www.xbox.com/en-US/Kinect/Home

3


4/46

INTRODUCTION

4


5/46

All about Kinect

First announced on 01/06/09 at E3 2009

Release Date

US: 4/11/10 SG: 18/11/10

Application development

Microsoft released a non-commercial Kinect for

Windows SDK on 16/06/11

You are about to learn and use it

5


6/46

Programming environment

Windows 7

machine withsoftware

installed

Kinect sensor

powersupply

6


7/46

Kinect Sensor

RGB CAMERA

MULTI-ARRAY MIC MOTORIZED TILT

3D DEPTH SENSORS

7


8/46

Programming environment

Windows 7

machine withsoftware

installed

Kinect sensor

powersupply

8


9/46

System Requirements (1/2)

Hardware

Windows 7 (x86 is recommended)

Dual-core, 2.66-Hz (recommended)

Graphics card that supports DirectX 9.0c 2GB of RAM (recommended)

Software

Microsoft Visual Studio 2010 (Prof/Express/)

Microsoft .NET Framework 4.0 Audio control (x86only) is not covered here

9


10/46

System Requirement (2/2)

Software for audio control (not covered in this

lecture)

Microsoft Speech Platform Server Runtime,

version 10.2 (x86)

Microsoft Speech Platform Software

Development Kit, version 10.2 (x86)

Kinect for Windows Runtime Language Pack,version 0.9

10


11/46

PREPARATION

11


12/46

Download and Install Kinect SDK Beta

Download Kinect SDK Beta

X86:http://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK32.msi

X64:http://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK64.msi

Caution:

Uninstall ALL other Kinect drivers before installation Only install on NATIVE Windows 7 instead of Virtual Machine

DISCONNECT Kinect device while installing

12
http://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK32.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK32.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK64.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK64.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK64.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK64.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK64.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK64.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK64.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK64.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK64.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK64.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK64.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK64.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK64.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK32.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK32.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK32.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK32.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK32.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK32.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK32.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK32.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK32.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK32.msihttp://download.microsoft.com/download/3/C/B/3CB546D6-C04E-4097-9393-C81C51F7FDA2/KinectSDK32.msi


13/46

Skills Requirements

IDE Integrated DevelopmentEnvironment Visual Studio 2010

http://msdn.microsoft.com/en-us/vstudio/ff431702 (official)

Programming Language Program logic

C# (or C++)

http://www.homeandlearn.co.uk/csharp/csharp.html

UI Design (with C# only) WPF Windows Presentation Foundation

http://www.codeproject.com/KB/WPF/BeginWPF1.aspx

13
http://msdn.microsoft.com/en-us/vstudio/ff431702http://msdn.microsoft.com/en-us/vstudio/ff431702http://www.homeandlearn.co.uk/csharp/csharp.htmlhttp://www.homeandlearn.co.uk/csharp/csharp.htmlhttp://www.codeproject.com/KB/WPF/BeginWPF1.aspxhttp://www.codeproject.com/KB/WPF/BeginWPF1.aspxhttp://www.codeproject.com/KB/WPF/BeginWPF1.aspxhttp://www.codeproject.com/KB/WPF/BeginWPF1.aspxhttp://www.codeproject.com/KB/WPF/BeginWPF1.aspxhttp://www.homeandlearn.co.uk/csharp/csharp.htmlhttp://www.homeandlearn.co.uk/csharp/csharp.htmlhttp://msdn.microsoft.com/en-us/vstudio/ff431702http://msdn.microsoft.com/en-us/vstudio/ff431702http://msdn.microsoft.com/en-us/vstudio/ff431702http://msdn.microsoft.com/en-us/vstudio/ff431702


14/46

DEMO TIME14


15/46

Kinect Basic Functions

Color stream

Depth stream

Skeletal tracking

Audio processing (not covered here)

15


16/46

Color Stream

16


17/46

Depth stream

17


18/46

Skeletal tracking

18


19/46

EXPLANATION FOR SAMPLE CODE

19

Piece 1

Data field

Piece 2

Three event handlers Piece 3

Helper methods


20/46

MainWindow

video

depth

skeleton

frameRate

20

Piece 1


21/46

Piece 1

Piece 3

Helper methods

Piece 2

Three event

handlers

MainWindow class starts

Data field initialization

Using Microsoft.Research.Kinect.Nui

21


22/46

nui represents the Kinect sensor instance

Initialize variables tocalculate the # of fps

Define constants for

indexing array

Define an array that is large enough to

hold an entire frame of depth data

at 32bits per pixel 22

f d l f h


23/46

Define a dictionary to store colors for each joint

23

Piece 1


24/46

Piece 1

Piece 3

Helper methods

Piece 2

Three event

handlers

Window_Loadedmethod

24


25/46

Before calling any other methods on the Runtime object, i.e. nui,

an application must call Runtime.Initialize to initialize the Kinect sensor

The Initialize method throws InvalidOperationException

if it fails to find a Kinect sensor

Parameter is a bitwise OR combination of the following values

from RuntimeOpitions enumeration:

- UseDepthAndPlayerIndex [recommended]

- UseColor [recommended]

- UseSkeletalTracking [recommended]- UseDepth25


26/46

The value of VideoStream and DepthStream are both ImageStream objects

The ImageStream.Open has 4 parameters

- ImageSteamType is either Video or Depth

- Number of lookahead buffers. By using 2 buffers, the application can

draw with one buffer and the capture driver can fill the other

- ImageResolution is a value of the enumeration- ImageType is also a value of the enumeration

VideoStream.Open DepthStream.Open

UseColor UseDepthAndPlayerIndex

Resolution1280x1024

Resolution640x480

Resolution320x240

Resolution80x60

Color,ColorYUV,ColorYUVRaw DepthAndPlayerIndex

Required RuntimeOptions

Valid Image Resolution

Valid Image Type 26


27/46

Application can use either apolling modelor an event-driven modelto retrieve frames

Here we introduce the event-driven model in this sample

To implement the event-driven model, we creates DepthFrameReady,SkeletonFrameReadyand VideoFrameReadyevents for nui, then associates

them with the event handlers respectively

When a frame from the Kinect sensor is ready for the application, the runtime

signals the relevant event and the associated event handler method runs

Event handler methods to

be explained later

27

Piece 2


28/46

Piece 2

Piece 1

Data field

Piece 3

Helper methods

nui_ColorFrameReady

method

28


29/46

Get the image to be displayed from parameter e

Use the WPF BitmapSource.Create method to create a bitmap

for the display

Check up the library to know the parameters here

NOTE: Image.Bits is an array of bits that form the image

The next method is about to construct that array for the display

29

Piece 2


30/46

Piece 2

Piece 1

Data field

Piece 3

Helper methods

nui_DepthFrameReady

method

30

Convert the image array to the desired display format


31/46

Convert the image array to the desired display format

Then display the convertedDepthFrame

Depth data (Image.Bits) is an array of 16-bit values

Update variables to calculate the number of frames per second

This means a time span of 1 second

31

Piece 3


32/46

Piece 3Piece 1

Data field

Piece 2

Three

eventhandlers

convertDepthFrame

method

32

This method is to convert 16-bit gray-scale data into 32-bit


33/46

This method is to convert 16-bit gray-scale data into 32-bit

RGB data in order to be well displayed

depthFrame16 is an array of 16-bit values, as follows:

- The low-order 3 bits contain a skeleton ID

- The remaining bits contain the depth value in millimeters

Get the player

skeleton ID

- Retrieve the 11 bits of actual depthdata

- Invert the result to generate an 8-bit

intensity value that is appropriate for

display

Zero-initialize the RGB

data in 32-bit frame

33


34/46

Color different players

with different RGB values

Player 0 indicates that NO

player is present

34

Piece 2


35/46

Piece 1

Data field

Piece 3

Helper methods

nui_SkeletonFrameReady

method

35


36/46

Retrieve SkeletonFrame object from parameter e

Initialize player

skeleton index

Create an array of SolidColorBrush

objects for drawing different player

skeletons (according to iSkeleton)

36


37/46

Clear any current displayed skeleton

The TrackingState field of the SkeletonData structure

indicates that this skeleton is being tracked

The two pieces of code is explained next

37

tB d S t th d h th f ll i 3 t


38/46

getBodySegment method has the following 3 parameters

- data.Joints is a collection of joints for the skeleton

- Brush is to draw the (poly)line

- a variable number of JointID values identify a sequence

of joints on the skeleton

The code is explained later

38


39/46

jointPos

39

Piece 3

Piece 1


40/46

Piece 1

Data field

Piece 2

Three

eventhandlersgetDisplayPosition

method

40

Convert skeleton coordinates in the [-1.0, 1.0] range to


41/46

Convert skeleton coordinates in the [ 1.0, 1.0] range to

depth coordinates, which returns x and y coordinates as

floating point number in the range from 0.0 to 1.0

Convert the floating

point coordinates tovalues in the 320x240

depth coordinate

space, which will be

used next

Convert depth coordinates to color

image coordinates, which return

color image coordinates as values in

the 640x480 color image space

41

Piece 3

Piece 1


42/46

Piece 1

Data field

Piece 2

Three

eventhandlers

getBodySegment

method

42


43/46

Construct a point

collection that

contains all joints

Construct a polyline

using data required

43


44/46

PRACTICE TOGETHER

44


45/46

SHOW YOUR UNDERSTANDING

45


46/46

Quick Assignment

Write a small program that can recognize the

following gestures

Left/Right Arm +

UP/DOWN/FORWARD/LEFT/RIGHT

E.g. Left arm FORWARD &

Right arm DOWN

Image from www.123fr.com

intro.to .kinect.sdk

Documents