Download - Programming with kinect v2
![Page 1: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/1.jpg)
Matteo Valoriani
![Page 2: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/2.jpg)
Nice to Meet You
Matteo Valoriani
PhD at Politecnico of Milano
CEO of Fifth Element
Speaker and Consultant
Microsoft MVP for Kinect
Intel Software Innovator: RealSense
email: [email protected]
twitter: @MatteoValoriani
linkedin: https://it.linkedin.com/in/matteovaloriani
2
![Page 3: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/3.jpg)
Usage Scenarios
TRAININGRETAIL HEALTHCARETHERAPY EDUCATION
![Page 4: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/4.jpg)
You have to be a
magician…
… or at least a
good illusionist
![Page 5: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/5.jpg)
“Any sufficiently
advanced
technology is
indistinguishable
from magic”
(Arthur C. Clarke)
![Page 6: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/6.jpg)
Sensor Anatomy
![Page 7: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/7.jpg)
Kinect 2 - Specs
3D DEPTH SENSOR
RGB CAMERA
MULTI-ARRAY MIC
Hardware:
Depth resolution:512×424
RGB resolution:1920×1080 (16:9)
FrameRate:60 FPS
Latency:60 ms
![Page 8: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/8.jpg)
Kinect Adapter
![Page 9: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/9.jpg)
Kinect 2 vs Kinect 1
![Page 10: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/10.jpg)
Version 1 Version 2
Depth range 0.4m → 4.0m 0.4m → 4.5m
Color stream 640×480 1920×1080
Depth stream 320×240 512×424
Infrared stream None 512×424
Type of Light Light coding ToF
Audio stream 4-mic array 16 kHz 4-mic array 48 kHz
USB 2.0 3.0
# Bodies Traked 2 (+4) 6
# Joints 20 25
Hand Traking External tools Yes
Face Traking Yes Yes+Expressions
FOV 57° H 43° V 70° H 60° V
Tilt Motorized Manual
![Page 11: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/11.jpg)
System / Software Requirements
OS Windows 8, 8.1, Embedded 8, Embedded 8.1 (x64)
CPU Intel Core i7 3.1GHz (or higher)
RAM 4GB (or more)
GPU DirectX 11 supported
USB USB 3.0
Compiler Visual Studio 2012, 2013 (Supported Express)
Language Native (C++), Managed (C#,VB.NET), WinRT (C#,HTML)
Other Unity Pro (Add-in), Cinder, openFrameworks (wrapper)
![Page 12: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/12.jpg)
DEMOKinect Evolution
![Page 13: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/13.jpg)
Accessing Kinect Data Sources
![Page 14: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/14.jpg)
High-level architecture
![Page 15: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/15.jpg)
Basic Flow of Programming
Sensor Stream Frame Data
Sensor Source Reader Frame Data
Kinect for Windows SDK v1
Kinect for Windows SDK v2
Source independent to each Data(e.g. ColorSource, DepthSource, InfraredSource, BodyIndexSource, BodySource, …)
Doesn’t depend on each other Source(e.g. Doesn't need to Depth Source when retrieve Body Data)
![Page 16: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/16.jpg)
• In “New Project” create a new Windows Store app
• Enable Microphone and Webcam capabilities
• Add a reference to Microsoft.Kinect
• Use the Microsoft.Kinect namespace in your code
Creating a new store app using Kinect
![Page 17: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/17.jpg)
Represents a single physical sensor
Always valid: when device is disconnected no more frame are generated.
Use IsAviable Property to verify if the device is connected
The KinectSensor class
this KinectSensorthis// Make the world a better place with Kinectthis
![Page 18: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/18.jpg)
Sources
![Page 19: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/19.jpg)
Give access to frames– Events
– Polling
Multiple readers may be
created on a single source
Readers can be paused
Readers
InfraredFrameReader reader = sensor.InfraredFrameSource.OpenReader();
reader.FrameArrived += InfraredReaderFrameArrived;
...
![Page 20: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/20.jpg)
Frame references
void InfraredFrameReaderInfraredFrameArrivedEventArgs
using (InfraredFrame frame = args.FrameReference.AcquireFrame())
{if (frame != null){
// Get what you need from the frame}
}}
Sent in frame event args
AcquireFrame gives access to the actual frame
![Page 21: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/21.jpg)
• Gives access to the frame data– Make a local copy or access the underlying buffer directly
• Contains metadata for the frame– e.g. Color: format, width, height, etc.
• Important: Minimize how long you hold onto the frame– Not Disposing frames will cause you to not receive more frames
Frames
![Page 22: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/22.jpg)
• Allows the app to get a matched set of frames from multiple
sources on a single event
• Delivers frames at the lowest FPS of the selected sources
MultiSourceFrameReader
MultiSourceFrameReader MultiReader =Sensor.OpenMultiSourceFrameReader(FrameSourceTypes.Color |
FrameSourceTypes.BodyIndex |FrameSourceTypes.Body);
var frame = args.FrameReference.AcquireFrame(); if (frame != null) {
using (colorFrame = frame.ColorFrameReference.AcquireFrame())using (bodyFrame = frame.BodyFrameReference.AcquireFrame())using (bodyIndexFrame = frame.BodyIndexFrameReference.AcquireFrame()){
//}
}
![Page 23: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/23.jpg)
System
Kinect
Driver
Kinect SDK
Application
Kinect
Driver
Kinect SDK
Application
Kinect
Service
Kinect SDK
Application
Kinect SDK
Application
Kinect for Windows v1 Kinect for Windows v2
![Page 24: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/24.jpg)
Connection
Kinect for Windows v1 Kinect for Windows v2
PCPC
![Page 25: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/25.jpg)
Example Multiple Connection
PCPC PC
Hub
Server
![Page 26: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/26.jpg)
DEMODisplay frame in a Windows Store App
![Page 27: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/27.jpg)
Data Source Details
Physical image sensors, ColorFrameSource,
InfraredFrameSource, DepthFrameSource,
BodyIndexFrameSource, BodyFrameSource, AudioFrameSource
![Page 28: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/28.jpg)
Kinect Data Sources
![Page 29: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/29.jpg)
• 30 or 15 fps, based on lighting
conditions
ColorFrameSource
![Page 30: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/30.jpg)
InfraredFrameSource
![Page 31: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/31.jpg)
– 16-bit distance in millimeters from the
sensor’s focal plane
DepthFrameSource
![Page 32: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/32.jpg)
– 0 to 5: Index of the corresponding body,
as tracked by the body source
– > 5: No tracked body at that pixel
BodyIndexFrameSource
255
0 1
![Page 33: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/33.jpg)
• Range is 0.5-4.5 meters
• Frame data is a collection of Body objects each
with 25 joints– Each joint has position in 3D space and an
orientation
• Up to six simultaneous bodies
• 30fps
• Hand State on 2 bodies
• Lean
BodyFrameSource
![Page 34: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/34.jpg)
Skeleton vs Body
![Page 35: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/35.jpg)
Coordinate System
ColorSpace (Coordinate System of the Color
Image)
– … Color
DepthSpace (Coordinate System of the
Depth Data)
– … Depth, Infrared, BodyIndex
CameraSpace (Coordinate System with the
origin located the Depth Sensor)
– … Body (Joint)
![Page 36: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/36.jpg)
• Three coordinate systems
• Coordinate mapper provides conversions between each system
• Convert single or multiple points
Coordinate mapping
Name Applies to Dimensions Units Range Origin
ColorSpacePoint Color 2 pixels 1920x1080 Top left corner
DepthSpacePoint Depth,
Infrared,
Body index
2 pixels 512x424 Top left corner
CameraSpacePoint Body 3 meters – Infrared/depth
camera
![Page 37: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/37.jpg)
DEMOHandle body frames and coordinate mapping
![Page 38: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/38.jpg)
Kinect Studio
Recording and Playback
![Page 39: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/39.jpg)
Recordable Data Sources
Infrared
13 MB/s
Depth
13 MB/s
BodyFrame
BodyIndex
Color
120 MB/s
Audio
32 KB/s
Legend
Record/Play
Record Only
![Page 40: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/40.jpg)
DEMOKinect Studio 2.0Recording | Playback
![Page 41: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/41.jpg)
Gesture Recognition
Heuristic + Machine Learning Approaches
![Page 42: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/42.jpg)
DEMOProgrammatic detection of a gesture/pose
![Page 43: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/43.jpg)
Gesture Builder
• New tool, shipping with v2 SDK
• Organize data using projects and solutions
• Give meaning to data by tagging gestures
• Build gestures using machine learning technology– Adaptive Boosting (AdaBoost) Trigger
• Determines if player is performing gesture
– Random Forest Regression (RFR) Progress
• Determines the progress of the gesture performed by player
• Analyze / test the results of gesture detection
• Live preview of results
![Page 44: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/44.jpg)
Visual Gesture Builder
Your Application
![Page 45: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/45.jpg)
DEMOMachine Learning Demo
![Page 46: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/46.jpg)
Heuristic
• Gesture is a coding problem
• Quick to do simple
gestures/poses (hand over head)
• ML can also be useful to find
good signals for Heuristic
approach
Machine Learning (ML) with G.B.
• Gesture is a data problem
• Signals which may not be easily
human understandable (progress
in a baseball swing)
• Large investment for production
• Danger of over-fitting, causes you
to be too specific – eliminating
recognition of generic cases
Gesture Recognition
![Page 47: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/47.jpg)
Hand Pointer Gestures
![Page 48: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/48.jpg)
Hand pointer gestures
Engagement Targeting
Press
Panning/Zoom
![Page 49: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/49.jpg)
DEMOControls Basics – XAMLUserViewer | Engagement | Targeting | Press
![Page 50: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/50.jpg)
DemoBuilding ControlsBasics-XAML from Scratch
• File New Project > Grid App
• Add Reference to Microsoft.Kinect.Xaml.Controls
• Enable Microphone + Camera capabilities for app
• Add KinectRegion as container for rootFrame in
App.xaml.cs
• Run it!
![Page 51: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/51.jpg)
Window
Frame
Page
Window
Frame
Page
Grid
KinectRegionKinect
UserViewer
Adding Hand Cursor Support
![Page 52: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/52.jpg)
System Engagement Manual Engagement
• Hands over head
• In certain location of room
• Etc…
“Engagement” for hand cursor
KinectRegion.SetOnePersonSystemEngagement()KinectRegion.SetTwoPersonSystemEngagement()
KinectRegion.SetOnePersonManualEngagement(…)KinectRegion.SetTwoPersonManualEngagement(…)
![Page 53: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/53.jpg)
DEMOPHiZ (Physical Interaction Zone) & KinectPointerPoint
![Page 54: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/54.jpg)
PHIZ – Physical Interaction Zone
X / Y - For Cursor Rendering
KinectPointerPoint->Position->X– Between 0.0 and 1.0
KinectPointerPoint->Position->Y– Between 0.0 and 1.0
X / Y – Allows offscreen info
KinectPointerPoint->Properties->UnclampedPosition->X
KinectPointerPoint->Properties->UnclampedPosition->Y
Z
KinectPointerPoint->Properties->HandReachExtent (raw)
KinectPointerPoint->Properties->PressExtent (affected by user/UI)
Correlating PointerPoint to NUI Data
KinectPointerPoint->Properties->BodyTrackingId
KinectPointerPoint->Properties->HandType
KinectPointerPoint->Properties->BodyTimeCounter
![Page 55: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/55.jpg)
Press
Recommended Minimum Size:• 208 x 208 (in 1080p resolution)• Press attraction towards center
• Larger buttons will just attract away from the edge
Adapting to smaller visual sizes:• Make large-size hit testable• Set KinectRegion.KinectPressInset
(Thickness) to non-visible part of button• Attraction region is smaller
![Page 56: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/56.jpg)
Cursors
![Page 57: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/57.jpg)
DemoControls Basics – XAML
Scroll
Zoom
![Page 58: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/58.jpg)
Interacting with Controls (in Preview release)
• UI Frameworks– XAML (Store)
– WPF
• DirectX (Store)ControlsBasics-
XAML
Microsoft.Kinect.Xaml.Controls
Microsoft.Kinect.Toolkit.Input
WindowsPreview.Kinect
ControlsBasics-WPF
Microsoft.Kinect.Wpf.Controls
Microsoft.Kinect
ControlsBasics-DX
KinectCoreWindow.PointerMoved,KinectPointerPoint, KinectGestureRecognizer
InputPointerManager, PressableModel, ManipulatableModel
Controls Layer – Cursor rendering, Hit Test, Route PointerPoint via InputPointerManager
App (host KinectRegion, etc…)
![Page 59: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/59.jpg)
Interacting with Content
• Navigating a camera through a 3d space
• Rotating a 3d object
• “Scrubbing” a video
![Page 60: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/60.jpg)
Kinect Resources
• General Info & Blog -> http://kinectforwindows.com
• Purchase Sensor -> http://aka.ms/k4wv2purchase
• v2 Preview SDK -> http://aka.ms/k4wv2previewsdk
• Developer Forums -> http://aka.ms/k4wv2forum
![Page 61: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/61.jpg)
Q&A
Tutto il nateriale di questa sessione sul sito
http://www.dotnetcampus.it
@MatteoValoriani
![Page 62: Programming with kinect v2](https://reader036.vdocument.in/reader036/viewer/2022082309/55c41bf1bb61ebfb5f8b470a/html5/thumbnails/62.jpg)
©2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, Office, Azure, System Center, Dynamics and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.