ios gpuimage introduction
DESCRIPTION
An introduction to GPUImage that is an excellent image processing framework for iOS. The architecture, threading model and how to leverage OpenGL ES are addressed in the slides.TRANSCRIPT
![Page 1: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/1.jpg)
Aug. 5, 2014
GPUImage IntroductionBruce Tsai
![Page 2: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/2.jpg)
Core Image
✤ Included in iOS from 2011!
✤ 100+ built-in filters!
✤ Multithread by CPU
![Page 3: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/3.jpg)
Drawbacks
✤ Customization limited!
- Subclass CIFilter in iOS!
- Create custom filters (shader) only in OSX!
✤ Performance not tuned!
- Address tuning tips in WWDC 2012 (session 511)3
![Page 4: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/4.jpg)
GPUImage
✤ Open source (BSD) iOS framework!
✤ GPU-based image and video processing!
✤ From GPU-accelerated video processing project (2010)!
✤ https://github.com/BradLarson/GPUImage
4
![Page 5: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/5.jpg)
Advantages
✤ Performance!
- High FPS in real-time preview!
✤ Customization!
- OpenGL ES shader code
5
![Page 6: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/6.jpg)
Buffer Process Flow
VideoCamera Sepia
View
MovieWriter
BlendPicture
Source Filter Output
![Page 7: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/7.jpg)
Code Snippet
GPUImageVideoCamera *videoCamera = [[GPUImageVideoCamera alloc] initWithSessionPreset:AVCaptureSessionPreset640x480 cameraPosition:AVCaptureDevicePositionBack];
GPUImageFilter *customFilter = [[GPUImageFilter alloc] initWithFragmentShaderFromFile:@“CustomShader”];
GPUImageView *filteredVideoView = [[GPUImageView alloc] initWithFrame:CGRectMake(0.0, 0.0, viewWidth, viewHeight)];
[videoCamera addTarget:customFilter];
[customFilter addTarget:filteredVideoView];
[videoCamera startCameraCapture];
![Page 8: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/8.jpg)
Class Diagram
![Page 9: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/9.jpg)
Procedures
1. Add effect to input buffer !
2. Save result to output buffer!
3. Set targets’ input buffer attributes!
4. Set targets’ input buffer!
5. Invoke targets’ operation9
![Page 10: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/10.jpg)
![Page 11: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/11.jpg)
Frame Buffer Object (FBO)
glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);
glBindTexture(GL_TEXTURE_2D, renderTexture);
glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, renderTexture, 0);
![Page 12: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/12.jpg)
Buffer Binding in FBO
Frame BufferTexturePixel Buffer
Render destinationActual render target
Main memory buffer
![Page 13: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/13.jpg)
Conversion to and from Texture
CVPixelBufferRef pxBufferRef = CMSampleBufferGetImageBuffer(imageBuffer);
CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault, videoTextureCache, pxBufferRef, NULL, GL_TEXTURE_2D, GL_RGBA, width, height, GL_BGRA, GL_UNSIGNED_BYTE, 0, &videoTextureRef);
...
// renderTargetRef is corresponding to renderTexture of FBO
[writerAdaptor appendPixelBuffer:renderTargetRef withPresentationTime:pts];
![Page 14: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/14.jpg)
Threading Model
✤ Grand Central Dispatch (GCD)!
- Dispatch queue!
✤ runSynchronousOnVideoProcessingQueue!
✤ runAsynchronousOnVideoProcessingQueue!
✤ runSynchronousOnContextQueue!
✤ runAsynchronousOnContextQueue14
![Page 15: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/15.jpg)
![Page 16: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/16.jpg)
Demo
![Page 17: iOS GPUImage introduction](https://reader033.vdocument.in/reader033/viewer/2022051514/54b4d84f4a79592f4e8b462a/html5/thumbnails/17.jpg)
Reference
✤ https://github.com/BradLarson/GPUImage!
✤ http://blog.db-in.com/all-about-opengl-es-2-x-part-2/!
✤ https://developer.apple.com/library/ios/documentation/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/Introduction/Introduction.html#//apple_ref/doc/uid/TP40008793!
✤ https://developer.apple.com/library/ios/documentation/GraphicsImaging/Conceptual/CoreImaging/ci_intro/ci_intro.html#//apple_ref/doc/uid/TP30001185