microso( kinect · • 2010: release for xbox 360 • 2011: kinect for windows sdk 1.x • 2012:...
TRANSCRIPT
1
Microso(KinectDepthSensing
2
PeterSperlC#developeratAnylineStudentatTUWien(MedicalInformaCcs)GamedeveloperhobbyistEmail:[email protected]
3
Agenda– Microso(Kinect
• About• Timeline• Components• SpecificaCons
– Depthsensing• V1
– Codedlight– TriangulaCon
• V2– TimeofFlight
– LiveDemo
4
• MoConsensinginputdevice• Naturaluserinterfacethroughvoiceandgestures• InnovaCveuse-cases
– Gaming,HealthCare,Fitness,Medicine,VR,AR,RoboCcs,etc.
• Sensor&SDK:ForWindowsPCandXbox–Codewri[eninC#,C++/CLIorVB.NET
5
Timeline
– 1stgeneraCon:• 2010:ReleaseforXbox360• 2011:KinectforWindowsSDK1.x• 2012:KinectforWindowsV1
– 2ndgeneraCon:• 2013:ReleaseforXboxOne• 2014:KinectforWindowsV2+SDK2.0• 2015:
– KinectforWindowsV2productwasdisconCnued,BUT:– WindowsuserssimplyuseKinectforXboxOnesensor+Kinectadapter
6
Components• Colorsensor:
• commoncamerasensortoacquireRGBimages
• Depthcamera:• IRemi[er+IRsensoruseinfraredlightto
acquiredepthimages• V1:StructuredLight/CodedLight• V2:TimeofFlight
• Audiosensor:• Microphonearraytocapturesoundsignal
andsoundposiCon
7
KinectSpecificaCons
Feature V1 V2
Colorstream 640x480at30FPS 1920x1280at30FPS
Depthstream 320x240 512x424
Infraredstream --- 512x424
Depthdistance 0.4m(nearmode)–4m 0.4m–4.5m
Horizontalfieldofview 57° 70°
VerCcalfieldofview 43° 60°
#definedskeletonjoints 20 25
#bodiestracked 2(+4recognized) 6
OperaCngsystem Windows7/8 Windows8
8
KinectSpecificaCons
V1 V2
9
DepthSensing(V1) 1. IREmi[er(E)projectsaknowninfraredlightpa[ern
(“codedlight”)intothescene2. IRSensor(S)retrievesreflectedlightfromthe
environment3. DepthmapiscomputedusingtriangulaConandlight
codingtechnology(byPrimeSense)4. DepthdataavailableviaUSBtotheAPI
10
CodedLight(alsocalledstructuredlight)
• Emi[erprojectspseudo-randompa[ernoflight(“Speckles”)
• Pa[ernis“coded”:– hastobeuniqueper
posiConsoeachpointcanberecognizedinthepa[ern
11
CodedLight• Lightisdistorted
onthesurface• Pa[ernisunique
ineveryposiCononthescene
• AllowstocomputedepthinformaConthroughtriangulaCon
12
TriangulaCon
• UsedinstereovisiontoreconstructapointPin3Dspace,giventwoormoreimages
• MulCpleimageprojecConsofthesamesceneyielddepthinformaCon
• PosiCons,orientaCons,andopCcalproperCesoftheimagesourcesmustbeknown
13
TriangulaCon
• Problem:– TriangulaConrequirestwoormoreimagesources,butKinecthasonlyonedepthsensor
14
TriangulaCon
• Trick:Kinectactuallyusestwoimages:– TheimagecapturedfromtheIRsensor– The“invisible”hardwiredreferencepa[ern
• TriangulaConapplicablebecausethereisadistancebetweenIRsensorandIRprojector
15
TriangulaCon
• Simplifiedmodel:– CamerasareparallelalongbaselineB– Cameracentersareatthesameheight– Camerashavesamefocallengthf
• Goal:calculatedisparityforeachpixel
B
16
TriangulaCon
D=x–x’=(B*f)/z
z=(B*f)/(x–x’)
• DisparityD:DifferenceinimagelocaConofthesamepointXwhenprojectedfromdifferentperspecCves
• DisparityisinverselyproporConaltodepth
f Focallength
D Measureddisparity
B Baselengthbetweenemi[erandcamera
z DistancetoX
17
TriangulaCon
• IRPa[erniscorrelatedagainstthereferencepa[ern• Projectedspeckles:
– ifdistancetothesensorissmallerorlargerthanonthereferenceplane:shi(edindirecConofthebaselinebetweenprojectorandperspecCvecenteroftheIRcamera
– x-direcConshi(saremeasuredforeachspeckle
• Distancefromeachpixeltothesensorisretrievedfromcorrespondingdisparity
xk,yk Coordinatesofpointk
xo,yo Principalpointoffsets
σx,σy LensdistorConcorrecCons
Z0 DistancebetweenobjectandSensoronreferenceplane
Zk DistancebetweenpointKandSensorinobjectspace
17
TriangulaCon
• IRSensorandIRCameraareepipolarrec6fiedØ BothhavematchingFieldofViewØ Correspondencesearchreducedto1DproblemØ IfreflecCngsurfacegetsfartheraway,lightfromprojectorgets“larger”sinceeach
dotisaconeoflightØ Lightexpandingrateandcamerapixelexpansionarethesame
• Whenobjectdistancechanges,dotsappeartobeshi(edonlyinx-direcCon
• Trackingadot’sx-coordinateyieldsthepixelsobserveddepth
18
DepthSensing(V2)
• 2ndgeneraConofKinectusesTimeofFlighttomeasuredepth
• IndirectmeasurementofCmeittakesforlaserpulsestoasurfaceandbacktothesensor
Ø Howisthispossible?
19
TimeofFlight • Speedoflight=299,792,458m/s• 1GHzCPU=1*109Hz=1,000,000,000Hz->Pulseoflighttravels~29cmwithin1CPUcycle
• IRSensor:• divides1pixelinhalf• Halfpixelisturnedonandoffextremelyfast
• On:Absorbphotonsoflaserlight• Off:Rejectphotonsoflaserlight
• Otherhalfofthepixeldoesthesame,but180°phaseshi(ed
20
TimeofFlight • LaserIlluminator(atthesameCme):
• Pulselightsourceinphaseofthefirstpixel-half
• Only“on”whenfirstpixelhalfis“on”
• Distanceat0cm:• Lightwillbeabsorbedinthe1sthalf,
nothingwillhappeninthesecondhalf
• Let’smovelightsourceawayforabit:• TimeofFlightwillslightlyincrease,photons
willreturninthe2ndhalfandberejected• Totalamountoflightabsorbedwill
decreaseslightly,totalamountoflightrejectedwillincreaseslightly
21
TimeofFlight • Movingsourceevenfartheraway:
• Absorbedamountdecreasesfurther,rejectedamountincreasesfurther
• A(erseveralmillisecondsexposure:• Twototalamountsofphotonsabsorbedbythetwofirsthalvesarecompared
tothetwototalamountsofphotonsrejectedbythetwosecondhalves• RaCoisimportant,notthetotalamount–surfacecanabsorblight,butit
affectsbothmeasurementsequally
• Depthismeasuredbytheround-tripdistance
22
TimeofFlight • Movingevenfurtheraway:
• Photonsmightovershoot1stand2ndhalfandarriveinthenextcycle• ->Ambiguity
• Ambiguityeasilyresolvedbyincreasingpixel-halfCmes,butthensmalldepthchangescan’tbemeasured
• SoluCon:• OnelowresoluConmeasurementasesCmate• Onehighprecisionmeasurement,usingesCmatetoeliminateambiguiCes
24
LiveDemo
25
Thankyou!
AndthanksPRIPforlendingmetheKinectforXboxOnesensorJ
Pa[ernRecogniConImageProcessingGroup@TUWien
• Sources:– h[ps://msdn.microso(.com/en-us/library/hh973078.aspx#Depth_Ranges– AccuracyandResoluConofKinectDepthDataforIndoorMappingApplicaCons
h[ps://courses.cs.washington.edu/courses/cse455/09wi/Lects/lect16.pdf– h[p://www.slideshare.net/KouroshKhoshelham/generaCon-and-weighCng-of-3d-point-correspondences-for-improved-registraCon-of-rgbd-data– h[ps://mirror2image.wordpress.com/2010/11/30/how-kinect-works-stereo-triangulaCon/– h[p://people.eng.unimelb.edu.au/kkhoshelham/research_files/kinectaccuracy/isprsarchives-XXXVIII-5-W12-133-2011.pdf– h[p://www.cs.cornell.edu/~hema/rgbd-workshop-2014/papers/Alhwarin.pdf– h[p://www.visionrt.com/content/core-technology
• Images:– h[p://www.qualitymag.com/arCcles/91010--d-imaging-enters-the-machine-vision-world– h[p://www.cnblogs.com/TracePlus/p/4138615.html– h[ps://msdn.microso(.com/en-us/library/jj131033.aspx– h[p://vision.ia.ac.cn/Students/gzp/kinect1.html– h[p://2.bp.blogspot.com/-f4tcmi3LrDE/UZ0uY5f2p6I/AAAAAAAAB0Y/8FQlnBG8suA/s1600/new-kinect-internal.png
• CodeExample:– h[p://kinect.github.io/tutorial/