ios visual f/x using glsl
DESCRIPTION
Original Presentation: http://bit.ly/JdJt0v Tweets: @dugla Email: [email protected]TRANSCRIPT
![Page 1: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/1.jpg)
iOS Visual F/X
Douglass TurnerElastic Image Softwareemail: [email protected] tweets: @duglacell: 781 775 3708
![Page 2: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/2.jpg)
• Cocoa Touch Limitations• Visual f/x Idioms• Show and Tell
![Page 3: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/3.jpg)
Cocoa Touch and its realization in Objective-C is about:
Abstraction. Pattern. Messaging. Generality. Simplicity.
![Page 4: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/4.jpg)
Containers: NSArray. NSDictionary. NSSet.
Data:NSData. NSString. CoreData.
Patterns:MVC. Target/Action. Delegation.
![Page 5: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/5.jpg)
The Objective-C Runtime is Powerful:
NSClassFromString | NSStringFromClass.
![Page 6: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/6.jpg)
When it comes to visual expressive power the limitations of Cocoa Touch become apparent.
![Page 7: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/7.jpg)
UIView lives in flatland ...
// UIViewstruct CGRect { CGPoint origin; CGSize size; };struct CGPoint { CGFloat x; CGFloat y; };
struct CGAffineTransform { CGFloat a, b, c, d; CGFloat tx, ty;};
@property(nonatomic) CGRect frame;@property(nonatomic) CGRect bounds; @property(nonatomic) CGPoint center; @property(nonatomic) CGAffineTransform transform;
- (CGPoint)convertPoint:(CGPoint)point toView:(UIView *)view;- (CGPoint)convertPoint:(CGPoint)point fromView:(UIView *)view;
- (CGRect)convertRect:(CGRect)rect toView:(UIView *)view;- (CGRect)convertRect:(CGRect)rect fromView:(UIView *)view;
![Page 8: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/8.jpg)
... but relies on CALayer which lives in 3D.
struct CATransform3D{ CGFloat m11, m12, m13, m14; CGFloat m21, m22, m23, m24; CGFloat m31, m32, m33, m34; CGFloat m41, m42, m43, m44;};
// CALayer@property CGFloat zPosition;@property CGFloat anchorPointZ;@property CATransform3D transform;
![Page 9: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/9.jpg)
iOS would prefer you ignore pixels and think about resolution independent points instead
![Page 10: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/10.jpg)
The level of visual abstraction is the asset:
Image. Video. Audio.
![Page 11: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/11.jpg)
As long as you are willing to remain at this high level of abstraction you can get a lot done using UIViewAnimation, CIImage, etc.
![Page 12: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/12.jpg)
But perhaps you are interested in going a bit deeper ...
![Page 13: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/13.jpg)
Hello OpenGL
You take the red pill – you stay in Wonderland and I show you how deep the rabbit-hole goes." -Morpheus
![Page 14: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/14.jpg)
OpenGL Cocoa
![Page 15: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/15.jpg)
OpenGL is old school
![Page 16: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/16.jpg)
You are flipping levers on a large state machine.
![Page 17: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/17.jpg)
OpenGL is a dragster. Not a Prius. Just buckle up ...
![Page 18: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/18.jpg)
... and enjoy the ride, baby!
![Page 19: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/19.jpg)
OpenGL websites are a bit different than the Cocoa websites/blogs you are familiar with ...
![Page 20: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/20.jpg)
Isn’t that sweet ...
![Page 21: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/21.jpg)
Um... WTF?
![Page 22: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/22.jpg)
Can you feel the love?
![Page 23: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/23.jpg)
Dude. I think my eyes are bleeding.
![Page 24: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/24.jpg)
Our focus is GLSL the OpenGL Shading Language
![Page 25: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/25.jpg)
The Shader Backstory
• Pixar creates Reyes (Render Everything You Ever Saw)• Pixar creates RenderMan: C-like language for describing a 3D look at the sample (pixel) level.• Boom! Everything changes.• GLSL created in the spirit of RenderMan.
![Page 26: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/26.jpg)
Shaders allow us to ignore the rest of a software system, freeing us to focus on achieving a specific look.
![Page 27: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/27.jpg)
EISParticleSystemhttp://github.com/turner/EISParticleSystem
![Page 28: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/28.jpg)
Texture Atlas Hacks
http://github.com/turner/EISParticleSystem
![Page 29: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/29.jpg)
s
t
Texture Space
![Page 30: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/30.jpg)
s
t
Texture Space
![Page 31: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/31.jpg)
Accelerometer Hacks
http://github.com/turner/EISParticleSystem
![Page 32: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/32.jpg)
touchesBegan:withEvent:
touchesMoved:withEvent:
touchesEnded:withEvent:
touchesCancelled:withEvent:
Initialize model state
Evolve model state
Clean up model state
![Page 33: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/33.jpg)
Demo
![Page 34: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/34.jpg)
GLSL. Think C-lite.
![Page 35: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/35.jpg)
varying!vec2 v_st;
uniform sampler2D hero;
void main() {!! gl_FragColor =
(heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st);!}
varying!vec2 v_st;
uniform sampler2D hero;
void main() {!! gl_FragColor = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st);!}
TextureMapShader
![Page 36: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/36.jpg)
InvertColorShader
vec2 v_st;
uniform int heroChannels;uniform sampler2D hero;
void main() {! ! vec3 rgb; rgb = (heroChannels == 1) ? 1.0 - vec3(raw.a) : 1.0 - raw.rgb; !
gl_FragColor = vec4(rgb, raw.a);!}
![Page 37: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/37.jpg)
![Page 38: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/38.jpg)
![Page 39: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/39.jpg)
LuminanceShader
varying vec2 v_st;
uniform int heroChannels;uniform sampler2D hero;
void main() {! vec4 raw = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st);
! vec3 rgb;! if (heroChannels == 1) {!! ! rgb = vec3(texture2D(hero, v_st).a);! !! } else {!! ! vec3 luminance_weights = vec3(0.30, 0.59, 0.11);!!! ! rgb = vec3( dot( luminance_weights, texture2D(hero, v_st).rgb ) );! }!! gl_FragColor = vec4(rgb, raw.a);
}
![Page 40: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/40.jpg)
![Page 41: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/41.jpg)
![Page 42: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/42.jpg)
MixShader
varying!vec2 v_st;
uniform vec3 overRGB;!uniform vec3 underRGB;
uniform int heroChannels;uniform sampler2D hero;
void main() {
! float mixer;!! mixer = (heroChannels == 1) ? texture2D(hero, v_st).a : texture2D(hero, v_st).g;!! gl_FragColor = vec4(mix(underRGB, overRGB, mixer), texture2D(hero, v_st).a);!}
varying!vec2 v_st;
uniform vec3 overRGB;!uniform vec3 underRGB;
uniform int heroChannels;uniform sampler2D hero;
void main() {
! float mixer;!! mixer = (heroChannels == 1) ? texture2D(hero, v_st).a : texture2D(hero, v_st).g;!! gl_FragColor = vec4(mix(underRGB, overRGB, mixer), texture2D(hero, v_st).a);!}
![Page 43: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/43.jpg)
![Page 44: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/44.jpg)
The GPU is a parallel processing beast. It uses a SIMD architecture (Single Instruction Multiple Data) to achieve massive processing power.
A GLSL shader is a SIMD program. The GPU takes a shader and evaluates it simultaneously - but with different data - at every sample (pixel) in parallel:
On iPad that is:2048 x 1536 * 30 fps = 94,371,840 shader evaluations per sec.
![Page 45: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/45.jpg)
What is exciting about this level of performance on an iOS device is the ability to take tasks previously thought of as desktop tasks done with Photoshop, Final Cut, etc. and do them live in a handheld device equipped with camera, mic, and other sensors.
![Page 46: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/46.jpg)
Demo. GLSL Powered Apps
• Beautiful Panoramas. App Store: http://bit.ly/9KJBLA• BMW Interior. Panoramic hotspots prototype.• RadPad. iPad radiology prototype.
![Page 47: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/47.jpg)
Shader Idioms
To fully “get” the power of shaders and their style of use it helpsknow the idioms guiding their use:
• Multiple Passes• Iteration aka Ping/Pong• Buffers/Channels• Indirection/Remapping
![Page 48: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/48.jpg)
Shader Idioms
These idioms derive directly from Hollywood film production workflows and practices that enable: • complex problem decomposition. Tractability.• rapid turnaround.• maximum flexibility and tweek-ability.• responsiveness to director’s whims
![Page 49: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/49.jpg)
Demo. ElasticImage.
![Page 50: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/50.jpg)
ElasticImage highlights.
• Rapid shader creation & deployment• Shaders and gesture declared in plist• Cocoa Touch for gestures & the usual.• C++ for 3D glue code.
![Page 51: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/51.jpg)
![Page 52: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/52.jpg)
Shader Example: ColourLovers
![Page 53: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/53.jpg)
![Page 54: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/54.jpg)
![Page 55: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/55.jpg)
I selected interesting palettes and created a texture.
Horizontal axis is the color palette. Vertical axis selects between color palettes.
At runtime this palette texture is loaded and attached to the ColourLoverShader.
![Page 56: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/56.jpg)
EISColourLoversShader.fsh
varying!mediump vec2 v_st;
// Palette selectoruniform float paletteDial;
// Contribution of remapped coloruniform float strengthDial;
// Palettes are ganged together into a single texture.// A specific palette is selected with the paletteDialuniform sampler2D colourLoversPalettes;
// herouniform int heroChannels;uniform sampler2D hero;
void main() {!! vec4 raw = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st);!! vec3 cooked;! cooked.r = texture2D(colourLoversPalettes, vec2(raw.r, paletteDial)).r;! cooked.g = texture2D(colourLoversPalettes, vec2(raw.g, paletteDial)).g;! cooked.b = texture2D(colourLoversPalettes, vec2(raw.b, paletteDial)).b;!! gl_FragColor = vec4(mix(raw.rgb, cooked, strengthDial), raw.a);}
![Page 57: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/57.jpg)
EISColourLoversShader.fsh
varying!mediump vec2 v_st;
// Palette selectoruniform float paletteDial;
// Contribution of remapped coloruniform float strengthDial;
// Palettes are ganged together into a single texture.// A specific palette is selected with the paletteDialuniform sampler2D colourLoversPalettes;
// herouniform int heroChannels;uniform sampler2D hero;
void main() {!! vec4 raw = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st);!! vec3 cooked;! cooked.r = texture2D(colourLoversPalettes, vec2(raw.r, paletteDial)).r;! cooked.g = texture2D(colourLoversPalettes, vec2(raw.g, paletteDial)).g;! cooked.b = texture2D(colourLoversPalettes, vec2(raw.b, paletteDial)).b;!! gl_FragColor = vec4(mix(raw.rgb, cooked, strengthDial), raw.a);}
![Page 58: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/58.jpg)
Demo ColourLovers Shader
![Page 59: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/59.jpg)
Hue Shift Shader
RGB HSB
![Page 60: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/60.jpg)
Conceptually simple idea. Use one channel of a photo to select the hue of a resultant image.
![Page 61: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/61.jpg)
Demo Hue Shift Shader
![Page 62: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/62.jpg)
Quantize ST Shader
![Page 63: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/63.jpg)
Beyond Photography - The Digital Darkroom by Gerard J. Holzmann
Sampling a texture at a low rate results in quantization.
![Page 64: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/64.jpg)
Beyond Photography - The Digital Darkroom by Gerard J. Holzmann
Sampling in polar coordinates rather then cartesian is a bit more interesting.
![Page 65: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/65.jpg)
Demo Quantize ST Shader
![Page 66: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/66.jpg)
BurnShader
![Page 67: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/67.jpg)
![Page 68: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/68.jpg)
![Page 69: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/69.jpg)
red channel = drop shadow green channel = cut out
final rgb texture
![Page 70: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/70.jpg)
Demo BurnShader
![Page 71: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/71.jpg)
Links
A previous meetup talk I gave on iOS OpenGL
• iOS OpenGL - http://slidesha.re/Y1MW8
Github - code• HelloGLSL - http://bit.ly/JCcMju
• EISRenderHelpful. Helpful 3D rendering glue code - http://bit.ly/JZ4HW3
Github - people to follow
• Philip Rideout - https://github.com/prideout
• Raphael Sebbe - https://github.com/rsebbe
• Brad Larson - https://github.com/BradLarson
• Jeff LaMarche - https://github.com/jlamarche
Elsewhere:
• Martins Upitis - http://devlog-martinsh.blogspot.com/• Ole Begemann - http://bit.ly/srlCBV
• Daniel Rakos - http://bit.ly/a3QATn
![Page 72: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/72.jpg)
Thank You!
![Page 73: iOS Visual F/X Using GLSL](https://reader033.vdocument.in/reader033/viewer/2022051314/54c7cda14a7959755a8b4572/html5/thumbnails/73.jpg)
Copyright © Douglass Turner
Douglass TurnerElastic Image Softwareemail: [email protected] tweets: @duglacell: 781 775 3708