2/16/2016 eec492/693/793 - iphone application development 1 eec-693/793 applied computer vision with...
DESCRIPTION
3 Working with Kinect Streams Kinect SDK supports two image stream formats Color image stream Depth image stream Both are children of ImageStream class The image frames are stored into a buffer for use by the application. If there is any delay in reading the buffer data and rendering it as images, the buffer will be overwritten => old frame lost Main steps in handling image streams Enabling the stream Capturing the stream frame by frame Processing the image framesTRANSCRIPT
![Page 1: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/1.jpg)
05/04/23EEC492/693/793 - iPhone Application
Development 1
EEC-693/793EEC-693/793Applied Computer Vision Applied Computer Vision
with Depth Cameraswith Depth Cameras
Lecture 4Lecture 4
Wenbing ZhaoWenbing [email protected]@ieee.org
![Page 2: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/2.jpg)
2
OutlineOutline Working with Kinect color stream Types of color images Retrieving color images: event & polling models Steps on handling of color images Event handler: C# delegate Build the KinectCam app
![Page 3: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/3.jpg)
3
Working with Kinect Streams Kinect SDK supports two image stream formats
Color image stream Depth image stream Both are children of ImageStream class
The image frames are stored into a buffer for use by the application. If there is any delay in reading the buffer data and rendering it as images, the buffer will be overwritten => old frame lost
Main steps in handling image streams Enabling the stream Capturing the stream frame by frame Processing the image frames
![Page 4: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/4.jpg)
Types of Color Images RGB: read-green-blue color space (RGB)
Each pixel is an array of four: Blue, Green, Red, Alpha Alpha: transparency degree 32bits per pixel at 640x480 at 30FPS or 1280x960 at 12FPS
YUV: Y: luminance channel; U: blue channel, V: red channel 16 bits per pixel at 640x480 at 15 FPS Uses less memory than RGB
Bayer: raw Bayer color image format with a Bayer color filter array (or Bayer filter): 50% green, 25% red, 25% blue Resolution at 640x480 at 30 FPS or 1280x960 at 12 FPS
![Page 5: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/5.jpg)
Two ways: event model and polling model Event model
Kinect sends the frame to the app whenever a new frame is captured by the sensor
Need to register a event handler for the event
Retrieving Color Image Stream
![Page 6: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/6.jpg)
Polling Model
The application sends a request to the sensor whenever there is a need to get an image frame
Pass the time interval after which the sensor will return the image frame
![Page 7: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/7.jpg)
Capturing Color Images
Main steps Starting Kinect sensor Enable color stream channel with desirable image
format Subscribe (register) an event handler (a method)
with the stream channel Process the image frame Render the image frame on the user interface
![Page 8: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/8.jpg)
Enable Color Stream Channel In class ColorImageStream, two methods public void Enable();
By default, use RgbResolution640x480Fps30 public void Enable(ColorImageFormat format);
RgbResolution640x480Fps30 RgbResolution1280x960Fps12 YuvResolution640x480Fps15 RawYuvResolution640x480Fps15 InfraredResolution640x480Fps30 RawBayerResolution640x480Fps30 RawBayerResolution1280x960Fps12 Undefined
You can enable only one type of color stream at a time
![Page 9: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/9.jpg)
Registering an Event Handler What is colorFrameReady?!
It is the event handler that you will implement You can use any method name as you wish The method must have the following signature:
void colorFrameReady(object sender, ColorImageFrameReadyEventArgs e)
Sender: the object that fires the event e: the object that holds data to be retrieved (i.e., image frame)
this.sensor.ColorFrameReady += colorFrameReady;
![Page 10: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/10.jpg)
Event Handler Event handler is similar to a function pointer in C/C++ In C# (and objective C), the event handler is a delegate with
pedefined signature General format:
public delegate void MyEventHandler(object sender, MyEventArgs e); Sender: files the event e: holds the data to be handled
A delegate allows you to pass methods of one class to objects of other classes that can call those methods
![Page 11: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/11.jpg)
Delegate in C#(http://www.developerfusion.com/article/2137/event-handling-in-net-using-c/2/ )using System; // Step 1. Declare a delegate with the signature of the encapsulated method public delegate void MyDelegate(string input);
//Step 2. Define methods that match with signature of delegate declaration class MyClass1{
public void delegateMethod1(string input){ Console.WriteLine("delegateMethod1 and input to the method is {0}",input); }
public void delegateMethod2(string input){ Console.WriteLine("delegateMethod2 and input to the method is {0}",input); } }
}
![Page 12: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/12.jpg)
Delegate in C#(http://www.developerfusion.com/article/2137/event-handling-in-net-using-c/2/ )
//Step 3. Create delegate object and plug in the methods class MyClass2 { public MyDelegate createDelegate(){ MyClass1 c2=new MyClass1(); MyDelegate d1 = new MyDelegate(c2.delegateMethod1); MyDelegate d2 = new MyDelegate(c2.delegateMethod2); MyDelegate d3 = d1 + d2; return d3; } }
![Page 13: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/13.jpg)
Delegate in C#(http://www.developerfusion.com/article/2137/event-handling-in-net-using-c/2/ )
//Step 4. Call the encapsulated methods through the delegate class MyClass3{ public void callDelegate(MyDelegate d, string input){ d(input); } } class Driver{ static void Main(string[] args){ MyClass2 c2 = new MyClass2(); MyDelegate d = c2.createDelegate(); MyClass3 c3 = new MyClass3(); c3.callDelegate(d,"Calling the delegate"); } }
![Page 14: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/14.jpg)
Event Handler Delegate for ColorImage
In Kinect.Sensor class
That is why we can this:
namespace System {public delegate void EventHandler<TEventArgs>
(object sender, TEventArgs e);}
public event EventHandler<ColorImageFrameReadyEventArgs> ColorFrameReady;
this.sensor.ColorFrameReady += colorFrameReady;
![Page 15: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/15.jpg)
Retrieve the Color Image Frame
What is using(a new object) { }? It defines the scope of the new object Once the last statement in the {} block is executed, the object is
garbage deleted
void colorFrameReady(object sender,ColorImageFrameReadyEventArgs e){
using (ColorImageFrame imageFrame = e.OpenColorImageFrame()) { ……. }}
![Page 16: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/16.jpg)
Handling/Displaying Color Image// copy image to a byte arrayimageFrame.CopyPixelDataTo(colorPixels);
// width in bytes of a single row of pixel data including padding.int stride = imageFrame.Width * imageFrame.BytesPerPixel;// Write the pixel data into our bitmapthis.colorBitmap.WritePixels( new Int32Rect(0, 0, this.colorBitmap.PixelWidth,
this.colorBitmap.PixelHeight),this.colorPixels,stride,
0);
// Don’t do this for stride calculation// If format is changed, it may not work!this.colorBitmap.PixelWidth * sizeof(int);
![Page 17: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/17.jpg)
Color Image Handling
Updates the pixels in the specified region of the bitmap. srcRect: The rectangle of the
System.Windows.Media.Imaging.WriteableBitmap to update pixels: The pixel array used to update the bitmap. stride: The stride of the update region in pixels. offset: The input buffer offset
public void WritePixels(Int32Rect srcRect, Array pixels, int stride, int offset);
![Page 18: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/18.jpg)
WriteableBitmap Class Inherits from BitmapSource Constructor
pixelWidth:width of the bitmap pixelHeight: height of the bitmap dpiX: horizontal dots per inch (dpi) dpiY: vertical dots per inch pixelFormat: pixel format palette: finite set of colors
public WriteableBitmap(int pixelWidth, int pixelHeight, double dpiX, double dpiY, PixelFormat
pixelFormat, BitmapPalette palette);
![Page 19: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/19.jpg)
System.Windows.Media.PixelFormats Bgr32 property: blue, green, red channels (3 bytes),
last byte is set to 0 http://msdn.microsoft.com/en-us/magazine/cc534995.aspx
Bgra32 Bgr24 Gray2 Gray16 Etc.
![Page 20: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/20.jpg)
Build KinectCam App Create a new WFP project named KinectCam Draw the GUI for the app Modify xaml file to add WindowLoaded and
WindowClosing as before Add member variables for displaying color images Start Kinect sensor, enable color stream, register
color stream handler, connect image control to the bitmap for display
Process each color image
![Page 21: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/21.jpg)
Design GUI Add label title Add image control
![Page 22: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/22.jpg)
Add Member Variables
KinectSensor sensor;
// Bitmap that will hold color information private WriteableBitmap colorBitmap;
// Intermediate storage for the color data received from the camera
private byte[] colorPixels;
![Page 23: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/23.jpg)
Initialization (in WindowLoaded())if (KinectSensor.KinectSensors.Count > 0) {
this.sensor = KinectSensor.KinectSensors[0]; if (this.sensor != null && !this.sensor.IsRunning) {
this.sensor.ColorStream.Enable();this.colorPixels = new byte[this.sensor.ColorStream.FramePixelDataLength];this.colorBitmap = new WriteableBitmap(this.sensor.ColorStream.FrameWidth, this.sensor.ColorStream.FrameHeight, 96.0, 96.0, PixelFormats.Bgr32, null);this.image1.Source = this.colorBitmap;
this.sensor.ColorFrameReady += this.colorFrameReady;this.sensor.Start();
} else { MessageBox.Show("No device is connected!"); this.Close(); } }}
![Page 24: 2/16/2016 EEC492/693/793 - iPhone Application Development 1 EEC-693/793 Applied Computer Vision with Depth Cameras Lecture 4 Wenbing Zhao](https://reader035.vdocument.in/reader035/viewer/2022062413/5a4d1b4c7f8b9ab0599a5cca/html5/thumbnails/24.jpg)
Handling/Displaying Color Images void colorFrameReady(object sender, ColorImageFrameReadyEventArgs e) {
using (ColorImageFrame imageFrame = e.OpenColorImageFrame()) { if (null == imageFrame) return;
imageFrame.CopyPixelDataTo(colorPixels); int stride = imageFrame.Width * imageFrame.BytesPerPixel;
// Write the pixel data into our bitmap this.colorBitmap.WritePixels( new Int32Rect(0, 0, this.colorBitmap.PixelWidth,
this.colorBitmap.PixelHeight), this.colorPixels, stride, 0); }}