connect with your
Post on 23-Jan-2016
42 Views
Preview:
DESCRIPTION
TRANSCRIPT
Connect with your
• Hi I’m Justin Weinberg• I’m a Carnegie Mellon Graduate• Manager Consultant at Sogeti• Founder of the North Texas Silverlight User
Group
Wow!
Motorized TiltRGB Camera
3D Depth SensorsMicrophone Array
That’s not USB?
You’ll need this..
http://openkinect.org/ http://www.openni.org/
http://research.microsoft.com/kinectsdk/
http://research.microsoft.com/kinect/
http://www.microsoft.com/express
Color Camera Streams
Depth / Depth with Player Streams
Skeletal Tracking
Audio Functionality
http://research.microsoft.com/kinect/
Demo 0Hello Tilt
Kinect NUI Big Picture Runtime nui = new Runtime();
nui.VideoFrameReady += ...
nui.VideoStream.Open(...)
void nui_VideoFrameReady(object sender, ImageFrameReadyEventArgs e) { //Process e.ImageFrame. }
1 2 3 4
9 10 11 12
8765
13 14 15 16
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
ImageFrame.Image.BitsWith camera data
you might expect Byte[,]
What you actually get is Byte[]:
BGR32 Formatwidth in pixels
heig
ht in
pix
els Stride: Bytes Per Row
Bytes per pixel * Width
4 Bytes Per pixel
Byte Array:
Demo 1Hello Colorful World
Depth data500mm 500mm500mm 500mm
2000mm2030mm2010mm2000mm
500mm500mm500mm2500mm
Depth dataMax resolution for raw depth data is 640 x 480
2 bytes per pixel of depth data
distance(0,0) = byte0 | byte1 << 8
11011110 pixel pixel pixel
0000101011101010000001011110101000001110
11011110
00001010
Demo 2Depth
Depth + Player Data500mm 500mm500mm 500mm
2000mm2030mm2010mm2000mm
500mm500mm500mm2500mm
player 1
player 1
player 1 player 1player 1
player 1 player 1
Depth + Player DataMax resolution is 320 x 240
11011001 00001011
Player: byte0 & 7
00000001
0000011111011001
&
Depth(0,0) = byte0 >> 3 | byte1 << 5
Still 2 bytes per pixel
But player is last 3 bits in first byte
11011 00001011001
Depth + Player Indexdistance
In mm11 bits per pixel
distance In mm
distance In mm
11011001 XXXX0111player player player
distance In mmplayer
distance In mmplayer
distance In mmplayer
distance In mmplayer player
distance In mmplayer
distance In mm
distance In mm
player
distance In mmplayer
distance In mmplayer
000 = No player001 = Player 1010 = Player 2
11011001 XXXX010111011001 XXXX0111
Demo 3Depth + Player Data
Skeletal Tracking
skeletonFrame.Skeletons where skeleton.TrackingState == SkeletonTrackingState.Tracked
skeletonFrame.Skeletons
from skeleton in skeletonFrame.Skeletons where skeleton.TrackingState == SkeletonTrackingState.PositionOnly
hand_left
wrist_left
elbow_left
shoulder_left
hand_right
wrist_right
elbow_right
shoulder_right
hip_left
head
hip_right
knee_left knee_right
ankle_rightankle_left
foot_left foot_right
hip center
spine
shoulder_center
Skeleton.Joints[JointId enum]Joint.Position (X,Y,Z,W) Skeletons are mirrored
X, Y, Z in left handed cartesian system+Z is out from sensor.W always returns 1 at this moment.
Demo 3Skeletal Tracking
Demo 4Putting Visuals
Together
Kinect Audio
Kinect Audio API
Microphone Array DMO
Implementation
Beamforming and Sound Localization
Features
Beam Forming
Beam Forming-0.875 radians-50 degrees
0.875 radians 50 degrees
.0175 radians
10 degrees
MicArrayModeMicArrayAdaptiveBeam Adaptive beam forming. Raises
beamforming eventsMicArrayExternBeam Your app chooses a beam (0 -10), May be
broke. Use with MicArrayBeamAngle.MicArrayFixedBeam Always use the center beam.MicArraySimpleSum All inputs are used (No beamforming)MicArraySingleBeam Perform beam forming. An internal source
localizer selects the beam.MicArraySingleChan One input is used. Set last 8 bits to choose
the input source.
SoundSourcePosition
SoundSourcePositionConfidence BeamChanged Event
Accousting Echo Cancellation
AEC
bool NoiseFill // Clip removes residual audio but leaves gaps. This fills with noise if true
bool CenterClip // After AEC, some echo residuals will remain. Setting to true removes.
int EchoLength – Sets an adaptive filter to the AEC algorithm (in MS). Recommended values are 128, 256, 512, 1024. For most offices and homes, use 256ms
int FrameSize // Size of a frame in samples Allowed values are : 80, 128, 160, 240, 256, 320. System default for microphone array only is 320All other modes default is 160
int SpeakerIndex – Device index of speaker apply AEC process to
Noise Suppression
Noise Supressionbool NoiseSuppression – True enables, false disables.
Automatic Gain Control
Automatic Gain Control
bool AutomaticGainControl – True enables, false disables.
Demo 5Recording Audio
Audio @!%!#!! Gotchas
Use Microsoft.Speech not System.Speech
Must run in an MTA
AEC SystemModes crash if no speaker output.
Must plug Kinect into Wall
DevicePairGuid / Qualitymetrics = crash
Should run in ThreadMode Priority Highest
Demo 6Speech recognition
Demo 7Kinect Effect
(Thanks SoulSolutions!)
Creative Commons Acknowledgments• Flickr by Ian Lewis• Flickr by major nelson• Flickr by markodesign• Flickr by stephantom• Flickr by Abstract Machine• Flickr By Aixo.co.kr• Flickr By Andresmh• Flickr by Antony Bennison• Flickr by Ars Electronica• Flickr By bm.iphone• Flickr by campuspartymexico• Flickr by Chris Jagers• Flickr By ci_polla Paola Sucato• Flickr by creepyhalloweenimages• Flickr by cvander• Flickr by IMAL.org• Flickr by Institut Douglas• Flickr by Joshua and AmberFlickr by Logon Sakai• Flickr by openexhibits• Flickr by Paul_Clarke
• Flickr by petyr.rahl• Flickr by pt• Flickr by Stilldavid• Flickr by y3rdua• Flickr by yesyesnono• Flickr by yum9me• Flickr by Jeff Howard• Flickr by jhscreenwm• Flickr by Kyle McDonald• Flickr by Morten Rand-Hendriksen• Flickr by Stuart Herbert• Flickr by Visiophone• Flickr by orkomedix• Michael Zöllner and Stephan Huber at the University of Konstanz• Openclip by J_Alves• Openclip by jetxee• Openclip by molumen• Openclip by shokunin• Openclip by Jhnri• Openclip by Machovka• Adafruit image of Hector MartinThanks!!
top related