kinect v2 introduction and tutorial
DESCRIPTION
2014/10/13 中部大学 ロボット理工学科 機械知覚&ロボティクスグループTRANSCRIPT
![Page 1: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/1.jpg)
Introduction to Kinect v2
Tsukasa Sugiura
@UnaNancyOwen
![Page 2: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/2.jpg)
Self-Introduction
Tsukasa SugiuraMicrosoft MVP for Kinect for Windows
@UnaNancyOwen
http://UnaNancyOwen.com
(July 2014 - June 2015)
![Page 3: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/3.jpg)
Agenda
What is Kinect v2
Specifications
New Features
Demo
Tutorial
![Page 4: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/4.jpg)
Disclaimer
“This is preliminary software and/or hardware and APIs
are preliminary and subject to change.”(Kinect for Windows v2は暫定的なものであり、ソフトウェア、ハードウェア、およびAPIは製品版で変更される可能性があります。)
![Page 5: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/5.jpg)
Kinect for Windows v1 Sensor
MULTI-ARRAY MIC MOTORIZED TILT
3D DEPTH SENSORS
RGB CAMERA
![Page 6: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/6.jpg)
Kinect for Windows v2 Sensor
MULTI-ARRAY MIC
3D DEPTH SENSOR
( IR Camera + IR Emitters )
RGB CAMERA
![Page 7: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/7.jpg)
Kinect for Windows v2 Sensor
Image by iFixit
IR EMITTERS
IR CAMERA
![Page 8: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/8.jpg)
Specifications
Kinect for Windows v1 Kinect for Windows v2
Color 640×480 @ 30fps 1920×1080 @ 30fps
Depth 320×240 @ 30fps 512×424 @ 30fps
Sensor Structured Light(PrimeSense Light Coding)
Time of Flight(ToF)
Range 0.8~4.0 m 0.5~4.5 m
Angle of ViewHorizontal / Vertical
57 / 43 degree 70 / 60 degree
Microphone Array ◯ ◯
![Page 9: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/9.jpg)
Specifications
Kinect for Windows v1 Kinect for Windows v2
BodyIndex 6 people 6 people
Body 2 people 6 people
Joint 20 joint/people 25 joint/people
Hand State Open / Closed Open / Closed / Lasso
Gesture ☓ ◯
Face ◯
Speech / Beamforming ◯ ◯
![Page 10: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/10.jpg)
Basic Features
Color
1920×1080@30fps / 15fps (Lighting Condition)
RGBA, YUV, BGRA, Bayer, YUY2
![Page 11: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/11.jpg)
Basic Features
Depth
512×424@30fps
500~4500[mm]
ToF (Time of Flight)
![Page 12: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/12.jpg)
Basic Features
Infrared / LongExposureInfrared
512×424@30fps
16bit (higher 8 bits)
![Page 13: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/13.jpg)
Basic Features
BodyIndex
512×424@30fps
6 people
Body Area : 0~5, Other Area : 255 (5 < Index)
255
0 1
![Page 14: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/14.jpg)
Basic Features
Body
6 people
25 joint / people (Add Tip, Thumb, Neck)
Orientation (Quaternion)
Hand Type (Right, Left),Hand State (Open, Closed, Lasso), Lean (-1.0f~1.0f)
![Page 15: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/15.jpg)
Basic Features
Audio
Beamforming (+/-50 degree)
Speaker Estimation
Speech Recognition
![Page 16: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/16.jpg)
Application Features
Gesture
Gesture Recognition using Machine Learning
Discrete (detected true/false), Continuous (progress 0.0f~1.0f)
Learning Classifier Tool “Visual Gesture Builder”
Video by http://youtu.be/-XYoblrnDpg
![Page 17: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/17.jpg)
Application Features
Face
Bounding Box, Rotation, Points (Eye, Nose, Mouth Corner)
Activity, Appearance, Expression Activity … Eye Closed, Mouth Open / Moved, Looking Away
Appearance … Wearing Glasses
Expression … Happy
![Page 18: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/18.jpg)
Application Features
HDFace
For Creating 3D Face Model
Points (1347), Triangles (2340), Hair Color, Skin Color
Fitting Face Model
![Page 19: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/19.jpg)
Application Features
Other
Kinect Fusion (3D Shape Reconstruction)
Controls (Assist in implementation of NUI)
![Page 20: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/20.jpg)
Demo
![Page 21: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/21.jpg)
System / Software Requirements
OS * Windows 8, 8.1, Embedded 8, Embedded 8.1 (x64)
CPU Intel Core i7 3.1GHz (or higher)
RAM 4GB (or more)
GPU * DirectX 11 supported
USB * USB 3.0 (Intel or Renesas Host Controller)
Compiler * Visual Studio 2012, 2013 (Supported Express)
Language Native (C++), Managed (C#,VB.NET), WinRT (C#,HTML)
Other Unity Pro (Add-in), Cinder, openFrameworks (wrapper)
![Page 22: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/22.jpg)
Connection
Kinect for Windows v1 Kinect for Windows v2
PCPC
![Page 23: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/23.jpg)
Example Multiple Connection
PCPC PC
Hub
Server
![Page 24: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/24.jpg)
System
Kinect
Driver
Kinect SDK
Application
Kinect
Driver
Kinect SDK
Application
Kinect
Service
Kinect SDK
Application
Kinect SDK
Application
Kinect for Windows v1 Kinect for Windows v2
![Page 25: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/25.jpg)
Tutorial
Basic Flow of Programming (C++)
Sensor Stream Frame Data
Sensor Source Reader Frame Data
Kinect for Windows SDK v1
Kinect for Windows SDK v2
Source independent to each Data(e.g. ColorSource, DepthSource, InfraredSource, BodyIndexSource, BodySource, …)
Doesn’t depend on each other Source(e.g. Doesn't need to Depth Source when retrieve Body Data)
![Page 26: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/26.jpg)
Tutorial
Sensor
Sensor Source Reader Frame Data
// Sensor
IKinectSensor* pSensor;
HRESULT hResult = S_OK;
hResult = GetDefaultKinectSensor( &pSensor );
if( FAILED( hResult ) ){
std::cerr << "Error : GetDefaultKinectSensor" << std::endl;
return -1;
}
hResult = pSensor->Open();
if( FAILED( hResult ) ){
std::cerr << "Error : IKinectSensor::Open()" << std::endl;
return -1;
}
IKinectSensor
![Page 27: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/27.jpg)
Tutorial
Source
Sensor Source Reader Frame Data
// Source
I***FrameSource* pSource;
hResult = pSensor->get_***FrameSource( &pSource );
if( FAILED( hResult ) ){
std::cerr << "Error : IKinectSensor::get_FrameSource()" << std::endl;
return -1;
}
IColorFrameSource
IDepthFrameSource
IInfraredFrameSource
IBodyIndexFrameSource
IBodyFrameSource
![Page 28: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/28.jpg)
Tutorial
Reader
Sensor Source Reader Frame Data
// Reader
I***FrameReader* pReader;
hResult = pSource->OpenReader( &pReader );
if( FAILED( hResult ) ){
std::cerr << "Error : IFrameSource::OpenReader()" << std::endl;
return -1;
}
IColorFrameReader
IDepthFrameReader
IInfraredFrameReader
IBodyIndexFrameReader
IBodyFrameReader
![Page 29: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/29.jpg)
Tutorial
Frame
Sensor Source Reader Frame Data
while( 1 ){
// Frame
I***Frame* pFrame = nullptr;
hResult = pReader->AcquireLatestFrame( &pFrame );
if( SUCCEEDED( hResult ) ){
/* Data */
}
SafeRelease( pFrame );
}
IColorFrame
IDepthFrame
IInfraredFrame
IBodyIndexFrame
IBodyFrame
![Page 30: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/30.jpg)
Tutorial
Data (Depth, Infrared, BodyIndex)
Sensor Source Reader Frame Data
while( 1 ){
// Frame
IDepthFrame* pDepthFrame = nullptr;
hResult = pDepthReader->AcquireLatestFrame( &pDepthFrame );
if( SUCCEEDED( hResult ) ){
unsigned int bufferSize = 0;
unsigned short* pBuffer = nullptr;
hResult = pDepthFrame->AccessUnderlyingBuffer( &bufferSize, &pBuffer );
if( SUCCEEDED( hResult ) ){
/* Processing*/
}
}
SafeRelease( pDepthFrame );
}
Depth, Infrared … unsigned short
BodyIndex … unsigned char
![Page 31: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/31.jpg)
Tutorial
Data (Color)
Sensor Source Reader Frame Data
while( 1 ){
// Frame
IColorFrame* pColorFrame = nullptr;
hResult = pColorReader->AcquireLatestFrame( &pColorFrame );
if( SUCCEEDED( hResult ) ){
unsigned int bufferSize = 0;
unsigned char* pBuffer = nullptr;
hResult = pColorFrame->AccessUnderlyingBuffer( &bufferSize, &pBuffer );
if( SUCCEEDED( hResult ) ){
/* Processing*/
}
}
SafeRelease( pColorFrame );
}
Color … unsigned char
Default Color Image Format … YUY2
![Page 32: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/32.jpg)
Tutorial
Data (Color with Converted Format)
Sensor Source Reader Frame Data
unsigned int buffeerSize = width * height * channel * sizeof(unsigned char);
unsigned char* pBuffer = new unsigned char[bufferSize];
while( 1 ){
// Frame
IColorFrame* pColorFrame = nullptr;
hResult = pReader->AcquireLatestFrame( &pColorFrame );
if( SUCCEEDED( hResult ) ){
hResult = pColorFrame->CopyConvertedFrameDataToArray( bufferSize, pBuffer,
ColorImageFormat::ColorImageFormat_Bgra );
if( SUCCEEDED( hResult ) ){
/* Processing */
}
}
SafeRelease( pColorFrame );
}
delete[] pBuffer;
![Page 33: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/33.jpg)
Tutorial
Sensor Source Reader Frame Data
while( 1 ){
IBodyFrame* pFrame = nullptr;
hResult = pBodyReader->AcquireLatestFrame( &pBodyFrame );
if( SUCCEEDED( hResult ) ){
IBody* pBody[BODY_COUNT] = { 0 };
hResult = pBodyFrame->GetAndRefreshBodyData( BODY_COUNT, pBody );
if( SUCCEEDED( hResult ) ){
/* Processing */
}
for( int count = 0; count < BODY_COUNT; count++ ){
SafeRelease( pBody[count] );
}
}
SafeRelease( pBodyFrame );
}
Data (Body)
![Page 34: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/34.jpg)
Tutorial
Sensor Source Reader Frame Data
for( int count = 0; count < BODY_COUNT; count++ ){
BOOLEAN bTracked = false;
hResult = pBody[count]->get_IsTracked( &bTracked );
if( SUCCEEDED( hResult ) && bTracked ){
Joint joint[JointType::JointType_Count];
hResult = pBody[count]->GetJoints( JointType::JointType_Count, joint );
if( SUCCEEDED( hResult ) ){
for( int type = 0; type < JointType::JointType_Count; type++ ){
if( joint[type].TrackingState != TrackingState::TrackingState_NotTracked ){
CameraSpacePoint cameraSpacePoint = joint[type].Position;
cameraSpacePoint.x; // x (+/- 1.0f)
cameraSpacePoint.y; // y (+/- 1.0f)
cameraSpacePoint.z; // z (500~4500[mm])
}
}
}
}
}
Data (Joint)
![Page 35: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/35.jpg)
Tutorial
Coordinate System
ColorSpace (Coordinate System of the Color Image)
… Color
DepthSpace (Coordinate System of the Depth Data)
… Depth, Infrared, BodyIndex
CameraSpace (Coordinate System with the origin located the Depth Sensor)
… Body (Joint)
![Page 36: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/36.jpg)
Tutorial
Coordinate Mapper// Coordinate Mapper
ICoordinateMapper* pCoordinateMapper;
hResult = pSensor->get_CoordinateMapper( &pCoordinateMapper );
if( FAILED( hResult ) ){
std::cerr << "Error : IKinectSensor::get_CoordinateMapper()" << std::endl;
return -1;
}
ICoordinateMapper::Map○○○FrameTo△△△Space()… Mapping Coordinate System in the Frame
ICoordinateMapper::Map○○○PointsTo△△△Space()… Mapping Coordinate System in the Array of Points
ICoordinateMapper::Map○○○PointTo△△△Space()… Mapping Coordinate System in the Point
![Page 37: Kinect v2 Introduction and Tutorial](https://reader034.vdocument.in/reader034/viewer/2022042602/559446271a28ab0e738b4629/html5/thumbnails/37.jpg)
Reference
Sample Program
Kinect2Sample | GitHubhttps://github.com/UnaNancyOwen/Kinect2Sample
Article Series
Introduction to Kinect for Windows v2 | Build Insider http://www.buildinsider.net/small/kinectv2cpp
Blog
Kinect | Summary?Bloghttp://unanancyowen.com/?cat=3