programming with kinect v2

Matteo Valoriani

Nice to Meet You

Matteo Valoriani

PhD at Politecnico of Milano

CEO of Fifth Element

Speaker and Consultant

Microsoft MVP for Kinect

Intel Software Innovator: RealSense

email: mvaloriani@gmail.com

twitter: @MatteoValoriani

linkedin: https://it.linkedin.com/in/matteovaloriani

2

Usage Scenarios

TRAININGRETAIL HEALTHCARETHERAPY EDUCATION

You have to be a

magician…

… or at least a

good illusionist

“Any sufficiently

advanced

technology is

indistinguishable

from magic”

(Arthur C. Clarke)

Sensor Anatomy

Kinect 2 - Specs

3D DEPTH SENSOR

RGB CAMERA

MULTI-ARRAY MIC

Hardware:

Depth resolution:512×424

RGB resolution:1920×1080 (16:9)

FrameRate:60 FPS

Latency:60 ms

Kinect Adapter

Kinect 2 vs Kinect 1

Version 1 Version 2

Depth range 0.4m → 4.0m 0.4m → 4.5m

Color stream 640×480 1920×1080

Depth stream 320×240 512×424

Infrared stream None 512×424

Type of Light Light coding ToF

Audio stream 4-mic array 16 kHz 4-mic array 48 kHz

USB 2.0 3.0

# Bodies Traked 2 (+4) 6

# Joints 20 25

Hand Traking External tools Yes

Face Traking Yes Yes+Expressions

FOV 57° H 43° V 70° H 60° V

Tilt Motorized Manual

System / Software Requirements

OS Windows 8, 8.1, Embedded 8, Embedded 8.1 (x64)

CPU Intel Core i7 3.1GHz (or higher)

RAM 4GB (or more)

GPU DirectX 11 supported

USB USB 3.0

Compiler Visual Studio 2012, 2013 (Supported Express)

Language Native (C++), Managed (C#,VB.NET), WinRT (C#,HTML)

Other Unity Pro (Add-in), Cinder, openFrameworks (wrapper)

DEMOKinect Evolution

Accessing Kinect Data Sources

High-level architecture

Basic Flow of Programming

Sensor Stream Frame Data

Sensor Source Reader Frame Data

Kinect for Windows SDK v1

Kinect for Windows SDK v2

Source independent to each Data(e.g. ColorSource, DepthSource, InfraredSource, BodyIndexSource, BodySource, …)

Doesn’t depend on each other Source(e.g. Doesn't need to Depth Source when retrieve Body Data)

• In “New Project” create a new Windows Store app

• Enable Microphone and Webcam capabilities

• Add a reference to Microsoft.Kinect

• Use the Microsoft.Kinect namespace in your code

Creating a new store app using Kinect

Represents a single physical sensor

Always valid: when device is disconnected no more frame are generated.

Use IsAviable Property to verify if the device is connected

The KinectSensor class

this KinectSensorthis// Make the world a better place with Kinectthis

Sources

Give access to frames– Events

– Polling

Multiple readers may be

created on a single source

Readers can be paused

Readers

InfraredFrameReader reader = sensor.InfraredFrameSource.OpenReader();

reader.FrameArrived += InfraredReaderFrameArrived;

...

Frame references

void InfraredFrameReaderInfraredFrameArrivedEventArgs

using (InfraredFrame frame = args.FrameReference.AcquireFrame())

{if (frame != null){

// Get what you need from the frame}

}}

Sent in frame event args

AcquireFrame gives access to the actual frame

• Gives access to the frame data– Make a local copy or access the underlying buffer directly

• Contains metadata for the frame– e.g. Color: format, width, height, etc.

• Important: Minimize how long you hold onto the frame– Not Disposing frames will cause you to not receive more frames

Frames

• Allows the app to get a matched set of frames from multiple

sources on a single event

• Delivers frames at the lowest FPS of the selected sources

MultiSourceFrameReader

MultiSourceFrameReader MultiReader =Sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color |

FrameSourceTypes.BodyIndex |FrameSourceTypes.Body);

var frame = args.FrameReference.AcquireFrame(); if (frame != null) {

using (colorFrame = frame.ColorFrameReference.AcquireFrame())using (bodyFrame = frame.BodyFrameReference.AcquireFrame())using (bodyIndexFrame = frame.BodyIndexFrameReference.AcquireFrame()){

//}

}

System

Kinect

Driver

Kinect SDK

Application

Kinect

Driver

Kinect SDK

Application

Kinect

Service

Kinect SDK

Application

Kinect SDK

Application

Kinect for Windows v1 Kinect for Windows v2

Connection

Kinect for Windows v1 Kinect for Windows v2

PCPC

Example Multiple Connection

PCPC PC

Hub

Server

DEMODisplay frame in a Windows Store App

Data Source Details

Physical image sensors, ColorFrameSource,

InfraredFrameSource, DepthFrameSource,

BodyIndexFrameSource, BodyFrameSource, AudioFrameSource

Kinect Data Sources

• 30 or 15 fps, based on lighting

conditions

ColorFrameSource

InfraredFrameSource

– 16-bit distance in millimeters from the

sensor’s focal plane

DepthFrameSource

– 0 to 5: Index of the corresponding body,

as tracked by the body source

– > 5: No tracked body at that pixel

BodyIndexFrameSource

255

0 1

• Range is 0.5-4.5 meters

• Frame data is a collection of Body objects each

with 25 joints– Each joint has position in 3D space and an

orientation

• Up to six simultaneous bodies

• 30fps

• Hand State on 2 bodies

• Lean

BodyFrameSource

Skeleton vs Body

Coordinate System

ColorSpace (Coordinate System of the Color

Image)

– … Color

DepthSpace (Coordinate System of the

Depth Data)

– … Depth, Infrared, BodyIndex

CameraSpace (Coordinate System with the

origin located the Depth Sensor)

– … Body (Joint)

• Three coordinate systems

• Coordinate mapper provides conversions between each system

• Convert single or multiple points

Coordinate mapping

Name Applies to Dimensions Units Range Origin

ColorSpacePoint Color 2 pixels 1920x1080 Top left corner

DepthSpacePoint Depth,

Infrared,

Body index

2 pixels 512x424 Top left corner

CameraSpacePoint Body 3 meters – Infrared/depth

camera

DEMOHandle body frames and coordinate mapping

Kinect Studio

Recording and Playback

Recordable Data Sources

Infrared

13 MB/s

Depth

13 MB/s

BodyFrame

BodyIndex

Color

120 MB/s

Audio

32 KB/s

Legend

Record/Play

Record Only

DEMOKinect Studio 2.0Recording | Playback

Gesture Recognition

Heuristic + Machine Learning Approaches

DEMOProgrammatic detection of a gesture/pose

Gesture Builder

• New tool, shipping with v2 SDK

• Organize data using projects and solutions

• Give meaning to data by tagging gestures

• Build gestures using machine learning technology– Adaptive Boosting (AdaBoost) Trigger

• Determines if player is performing gesture

– Random Forest Regression (RFR) Progress

• Determines the progress of the gesture performed by player

• Analyze / test the results of gesture detection

• Live preview of results

Visual Gesture Builder

Your Application

DEMOMachine Learning Demo

Heuristic

• Gesture is a coding problem

• Quick to do simple

gestures/poses (hand over head)

• ML can also be useful to find

good signals for Heuristic

approach

Machine Learning (ML) with G.B.

• Gesture is a data problem

• Signals which may not be easily

human understandable (progress

in a baseball swing)

• Large investment for production

• Danger of over-fitting, causes you

to be too specific – eliminating

recognition of generic cases

Gesture Recognition

Hand Pointer Gestures

Hand pointer gestures

Engagement Targeting

Press

Panning/Zoom

DEMOControls Basics – XAMLUserViewer | Engagement | Targeting | Press

DemoBuilding ControlsBasics-XAML from Scratch

• File New Project > Grid App

• Add Reference to Microsoft.Kinect.Xaml.Controls

• Enable Microphone + Camera capabilities for app

• Add KinectRegion as container for rootFrame in

App.xaml.cs

• Run it!

Window

Frame

Page

Window

Frame

Page

Grid

KinectRegionKinect

UserViewer

Adding Hand Cursor Support

System Engagement Manual Engagement

• Hands over head

• In certain location of room

• Etc…

“Engagement” for hand cursor

KinectRegion.SetOnePersonSystemEngagement()KinectRegion.SetTwoPersonSystemEngagement()

KinectRegion.SetOnePersonManualEngagement(…)KinectRegion.SetTwoPersonManualEngagement(…)

DEMOPHiZ (Physical Interaction Zone) & KinectPointerPoint

PHIZ – Physical Interaction Zone

X / Y - For Cursor Rendering

KinectPointerPoint->Position->X– Between 0.0 and 1.0

KinectPointerPoint->Position->Y– Between 0.0 and 1.0

X / Y – Allows offscreen info

KinectPointerPoint->Properties->UnclampedPosition->X

KinectPointerPoint->Properties->UnclampedPosition->Y

Z

KinectPointerPoint->Properties->HandReachExtent (raw)

KinectPointerPoint->Properties->PressExtent (affected by user/UI)

Correlating PointerPoint to NUI Data

KinectPointerPoint->Properties->BodyTrackingId

KinectPointerPoint->Properties->HandType

KinectPointerPoint->Properties->BodyTimeCounter

Press

Recommended Minimum Size:• 208 x 208 (in 1080p resolution)• Press attraction towards center

• Larger buttons will just attract away from the edge

Adapting to smaller visual sizes:• Make large-size hit testable• Set KinectRegion.KinectPressInset

(Thickness) to non-visible part of button• Attraction region is smaller

Cursors

DemoControls Basics – XAML

Scroll

Zoom

Interacting with Controls (in Preview release)

• UI Frameworks– XAML (Store)

– WPF

• DirectX (Store)ControlsBasics-

XAML

Microsoft.Kinect.Xaml.Controls

Microsoft.Kinect.Toolkit.Input

WindowsPreview.Kinect

ControlsBasics-WPF

Microsoft.Kinect.Wpf.Controls

Microsoft.Kinect

ControlsBasics-DX

KinectCoreWindow.PointerMoved,KinectPointerPoint, KinectGestureRecognizer

InputPointerManager, PressableModel, ManipulatableModel

Controls Layer – Cursor rendering, Hit Test, Route PointerPoint via InputPointerManager

App (host KinectRegion, etc…)

Interacting with Content

• Navigating a camera through a 3d space

• Rotating a 3d object

• “Scrubbing” a video

Kinect Resources

• General Info & Blog -> http://kinectforwindows.com

• Purchase Sensor -> http://aka.ms/k4wv2purchase

• v2 Preview SDK -> http://aka.ms/k4wv2previewsdk

• Developer Forums -> http://aka.ms/k4wv2forum

Q&A

Tutto il nateriale di questa sessione sul sito

http://www.dotnetcampus.it

@MatteoValoriani

©2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Office, Azure, System Center, Dynamics and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.