live photo editing and raw processing with core image

310
© 2016 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple. Media #WWDC16 Session 505 Live Photo Editing and RAW Processing with Core Image David Hayward Pixel Perfectionist

Upload: others

Post on 11-Sep-2021

6 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Live Photo Editing and RAW Processing with Core Image

© 2016 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.

Media #WWDC16

Session 505

Live Photo Editing and RAW Processing with Core Image

David Hayward Pixel Perfectionist

Page 2: Live Photo Editing and RAW Processing with Core Image

What You Will Learn Today

A very brief introduction to Core ImageAdjusting RAW images on iOSEditing Live Photos Extending Core Image using CIImageProcessor

Page 3: Live Photo Editing and RAW Processing with Core Image

A Very Brief Introduction to Core Image

Page 4: Live Photo Editing and RAW Processing with Core Image

A simple, high-performance API to apply filters to imagesA Very Brief Introduction to Core Image

HueFilter

SepiaFilter

Input toWorking Space

Original CIImage Output CIImage

Page 5: Live Photo Editing and RAW Processing with Core Image

A simple, high-performance API to apply filters to imagesA Very Brief Introduction to Core Image

HueFilter

SepiaFilter

Input toWorking Space

Original CIImage

image = image.applyingFilter(

"CISepiaTone",

withInputParameters:

["inputIntensity" : 1.0])

Output CIImage

Page 6: Live Photo Editing and RAW Processing with Core Image

A simple, high-performance API to apply filters to imagesA Very Brief Introduction to Core Image

HueFilter

SepiaFilter

Input toWorking Space

Original CIImage Output CIImage

Page 7: Live Photo Editing and RAW Processing with Core Image

A simple, high-performance API to apply filters to imagesA Very Brief Introduction to Core Image

O Working Spaceto Output

ContrastFilter

HueFilter

SepiaFilter

Input toWorking Space I

Original CIImage Output CIImage

Page 8: Live Photo Editing and RAW Processing with Core Image

OWorking Spaceto Output

ContrastFilter

HueFilter

Automatic color managementA Very Brief Introduction to Core Image

SepiaFilter

Input toWorking Space

Original CIImage Output CIImage

Page 9: Live Photo Editing and RAW Processing with Core Image

OWorking Spaceto Output

ContrastFilter

HueFilter

Automatic color managementA Very Brief Introduction to Core Image

SepiaFilter

Input toWorking Space

Original CIImage Output CIImage

⚠Wide color images and displays are common.

Most open-source image processing librariesdo not support color management.

Page 10: Live Photo Editing and RAW Processing with Core Image

OWorking Spaceto Output

ContrastFilter

HueFilter

Automatic color managementA Very Brief Introduction to Core Image

SepiaFilter

Input toWorking Space

Original CIImage Output CIImage

Page 11: Live Photo Editing and RAW Processing with Core Image

kernel vec4 sepia () kernel vec4 hue () kernel vec4 contrast ()

Each CIFilter has one or more CIKernel functionsA Very Brief Introduction to Core Image

OWorking Spaceto Output

ContrastFilter

HueFilter

SepiaFilter

Input toWorking Space

Original CIImage Output CIImage

Page 12: Live Photo Editing and RAW Processing with Core Image

kernel vec4 sepia ()

kernel vec4 hue ()

kernel vec4 contrast ()

Each CIFilter has one or more CIKernel functionsA Very Brief Introduction to Core Image

O Working Spaceto Output

ContrastFilter

HueFilter

SepiaFilter

Input toWorking Space

Concatenated Program

Original CIImage Output CIImage

Page 13: Live Photo Editing and RAW Processing with Core Image

180 Built-In FiltersAccordionFoldTransition AdditionCompositing AffineClamp AffineTile AffineTransform AreaAverage AreaHistogram AreaMaximum AreaMaximumAlpha AreaMinimum AreaMinimumAlpha AztecCodeGenerator BarsSwipeTransition BlendWithAlphaMask BlendWithMask Bloom BoxBlur BumpDistortion BumpDistortionLinear CheckerboardGenerator CircleSplashDistortion CircularScreen CircularWrap Clamp CMYKHalftone Code128BarcodeGenerator ColorBlendMode ColorBurnBlendMode ColorClamp ColorControls

ColorCrossPolynomial ColorCube ColorCubeWithColorSpace ColorDodgeBlendMode ColorInvert ColorMap ColorMatrix ColorMonochrome ColorPolynomial ColorPosterize ColumnAverage ComicEffect ConstantColorGenerator Convolution3X3 Convolution5X5 Convolution7X7 Convolution9Horizontal Convolution9Vertical CopyMachineTransition Crop Crystallize DarkenBlendMode DepthOfField DifferenceBlendMode DiscBlur DisintegrateWithMaskTransition DisplacementDistortion DissolveTransition DivideBlendMode DotScreen

Droste Edges EdgeWork EightfoldReflectedTile ExclusionBlendMode ExposureAdjust FalseColor FlashTransition FourfoldReflectedTile FourfoldRotatedTile FourfoldTranslatedTile GammaAdjust GaussianBlur GaussianGradient GlassDistortion GlassLozenge GlideReflectedTile Gloom HardLightBlendMode HatchedScreen HeightFieldFromMask HexagonalPixellate HighlightShadowAdjust HistogramDisplayFilter HoleDistortion HueAdjust HueBlendMode HueSaturationValueGradient Kaleidoscope LanczosScaleTransform

LenticularHaloGenerator LightenBlendMode LightTunnel LinearBurnBlendMode LinearDodgeBlendMode LinearGradient LinearToSRGBToneCurve LineOverlay LineScreen LuminosityBlendMode MaskedVariableBlur MaskToAlpha MaximumComponent MaximumCompositing MedianFilter MinimumComponent MinimumCompositing ModTransition MotionBlur MultiplyBlendMode MultiplyCompositing NinePartStretched NinePartTiled NoiseReduction OpTile OverlayBlendMode PageCurlTransition PageCurlWithShadowTransition ParallelogramTile PDF417BarcodeGenerator

PerspectiveCorrection PerspectiveTile PerspectiveTransform PerspectiveTransformWithExtent PhotoEffectChrome PhotoEffectFade PhotoEffectInstant PhotoEffectMono PhotoEffectNoir PhotoEffectProcess PhotoEffectTonal PhotoEffectTransfer PinchDistortion PinLightBlendMode Pixellate Pointillize QRCodeGenerator RadialGradient RandomGenerator RippleTransition RowAverage SaturationBlendMode ScreenBlendMode SepiaTone ShadedMaterial SharpenLuminance SixfoldReflectedTile SixfoldRotatedTile SmoothLinearGradient SoftLightBlendMode

SourceAtopCompositing SourceInCompositing SourceOutCompositing SourceOverCompositing SpotColor SpotLight SRGBToneCurveToLinear StarShineGenerator StraightenFilter StretchCrop StripesGenerator SubtractBlendMode SunbeamsGenerator SwipeTransition TemperatureAndTint Thermal ToneCurve TorusLensDistortion TriangleKaleidoscope TriangleTile TwelvefoldReflectedTile TwirlDistortion UnsharpMask Vibrance Vignette VignetteEffect VortexDistortion WhitePointAdjust XRay ZoomBlur

Page 14: Live Photo Editing and RAW Processing with Core Image

CIHueSaturationValueGradientNew Built-In CIFilters NEW

Page 15: Live Photo Editing and RAW Processing with Core Image

CIHueSaturationValueGradientNew Built-In CIFilters NEW

Page 16: Live Photo Editing and RAW Processing with Core Image

CINinePartStretched and CINinePartTiledNew Built-In CIFilters NEW

Page 17: Live Photo Editing and RAW Processing with Core Image

CINinePartStretched and CINinePartTiledNew Built-In CIFilters NEW

Page 18: Live Photo Editing and RAW Processing with Core Image

CINinePartStretched and CINinePartTiledNew Built-In CIFilters NEW

Page 19: Live Photo Editing and RAW Processing with Core Image

CIEdgePreserveUpsampleFilterNew Built-In CIFilters NEW

6x7 Pixel Input

Page 20: Live Photo Editing and RAW Processing with Core Image

CIEdgePreserveUpsampleFilterNew Built-In CIFilters NEW

6x7 Pixel Input 1024x768 Pixel Guide

+

Page 21: Live Photo Editing and RAW Processing with Core Image

CIEdgePreserveUpsampleFilterNew Built-In CIFilters NEW

6x7 Pixel Input 1024x768 Pixel Guide 1024x768 Pixel Result

+ =

Page 22: Live Photo Editing and RAW Processing with Core Image

CIEdgePreserveUpsampleFilterNew Built-In CIFilters NEW

6x7 Pixel Input 1024x768 Pixel Guide 1024x768 Pixel Result

+ =

Page 23: Live Photo Editing and RAW Processing with Core Image

New Performance Controls

Metal on by default

NEW

Page 24: Live Photo Editing and RAW Processing with Core Image

New Performance Controls

Metal on by defaultNow UIImage(ciImage:) is much faster

NEW

Page 25: Live Photo Editing and RAW Processing with Core Image

New Performance Controls

Metal on by defaultNow UIImage(ciImage:) is much fasterCore Image supports input and output of half-float CGImageRefs

NEW

Page 26: Live Photo Editing and RAW Processing with Core Image

New Performance Controls

Metal on by defaultNow UIImage(ciImage:) is much fasterCore Image supports input and output of half-float CGImageRefs

NEW

Pixel Format Bytes Per Pixel Bit Depth Range QuantizationRGBA8 4 8 0…1 linear

Page 27: Live Photo Editing and RAW Processing with Core Image

New Performance Controls

Metal on by defaultNow UIImage(ciImage:) is much fasterCore Image supports input and output of half-float CGImageRefs

NEW

Pixel Format Bytes Per Pixel Bit Depth Range QuantizationRGBA8 4 8 0…1 linear

RGBAf 16 24 –1038 … 1038 logarithmic

Page 28: Live Photo Editing and RAW Processing with Core Image

New Performance Controls

Metal on by defaultNow UIImage(ciImage:) is much fasterCore Image supports input and output of half-float CGImageRefs

NEW

Pixel Format Bytes Per Pixel Bit Depth Range QuantizationRGBA8 4 8 0…1 linear

RGBAf 16 24 –1038 … 1038 logarithmic

RGBAh 8 10 –65519 … 65519 logarithmic

Page 29: Live Photo Editing and RAW Processing with Core Image

New Performance Controls

Metal on by defaultNow UIImage(ciImage:) is much fasterCore Image supports input and output of half-float CGImageRefs

NEW

Pixel Format Bytes Per Pixel Bit Depth Range QuantizationRGBA8 4 8 0…1 linear

RGBAf 16 24 –1038 … 1038 logarithmic

RGBAh 8 10 –65519 … 65519 logarithmic

CVPixelFormat30RGBLEPackedWideGamut 4 10 –0.37 … 1.62 gamma’d

Page 30: Live Photo Editing and RAW Processing with Core Image

Adjusting RAW Images with Core Image

Page 31: Live Photo Editing and RAW Processing with Core Image

Adjusting RAW Images with Core Image

What is a RAW fileUsing the CIRAWFilter APISupporting wide-gamut outputManaging memory

Page 32: Live Photo Editing and RAW Processing with Core Image

What is a RAW fileAdjusting RAW Images

Page 33: Live Photo Editing and RAW Processing with Core Image

What is a RAW fileAdjusting RAW Images

Most cameras use a color filter arrayand a sensor array

Sensor Array

Color Filter Array

Page 34: Live Photo Editing and RAW Processing with Core Image

What is a RAW fileAdjusting RAW Images

Most cameras use a color filter arrayand a sensor array

Photons from the scene pass through the filter and are counted by the sensor

Sensor Array

Light from Scene

Color Filter Array

Page 35: Live Photo Editing and RAW Processing with Core Image

What is a RAW fileAdjusting RAW Images

Most cameras use a color filter arrayand a sensor array

Photons from the scene pass through the filter and are counted by the sensor

Page 36: Live Photo Editing and RAW Processing with Core Image

What is a RAW fileAdjusting RAW Images

Most cameras use a color filter arrayand a sensor array

Photons from the scene pass through the filter and are counted by the sensor

Page 37: Live Photo Editing and RAW Processing with Core Image

What is a RAW fileAdjusting RAW Images

Most cameras use a color filter arrayand a sensor array

Photons from the scene pass through the filter and are counted by the sensor

Advanced image processing is required to develop the RAW sensor data into an image suitable for output

Page 38: Live Photo Editing and RAW Processing with Core Image

What is a RAW fileAdjusting RAW Images

Most cameras use a color filter arrayand a sensor array

Photons from the scene pass through the filter and are counted by the sensor

Advanced image processing is required to develop the RAW sensor data into an image suitable for output

Page 39: Live Photo Editing and RAW Processing with Core Image

What is a RAW fileAdjusting RAW Images

Page 40: Live Photo Editing and RAW Processing with Core Image

What is a RAW fileAdjusting RAW Images

RAW files store unprocessed scene data

Page 41: Live Photo Editing and RAW Processing with Core Image

What is a RAW fileAdjusting RAW Images

RAW files store unprocessed scene data

JPG files store processed output images

Page 42: Live Photo Editing and RAW Processing with Core Image

What is a RAW fileAdjusting RAW Images

JPG files store processed output images

Page 43: Live Photo Editing and RAW Processing with Core Image

What is a RAW fileAdjusting RAW Images

Page 44: Live Photo Editing and RAW Processing with Core Image

Stages of RAW image processingAdjusting RAW Images

Page 45: Live Photo Editing and RAW Processing with Core Image

Stages of RAW image processingAdjusting RAW Images

Extract critical metadata

Page 46: Live Photo Editing and RAW Processing with Core Image

Stages of RAW image processingAdjusting RAW Images

Extract critical metadataDecode RAW sensor image

Page 47: Live Photo Editing and RAW Processing with Core Image

Stages of RAW image processingAdjusting RAW Images

Extract critical metadataDecode RAW sensor imageDe-mosaic reconstruction

Page 48: Live Photo Editing and RAW Processing with Core Image

Stages of RAW image processingAdjusting RAW Images

Extract critical metadataDecode RAW sensor imageDe-mosaic reconstructionApply lens correction

Page 49: Live Photo Editing and RAW Processing with Core Image

Stages of RAW image processingAdjusting RAW Images

Extract critical metadataDecode RAW sensor imageDe-mosaic reconstructionApply lens correctionReduce noise

Page 50: Live Photo Editing and RAW Processing with Core Image

Stages of RAW image processingAdjusting RAW Images

Extract critical metadataDecode RAW sensor imageDe-mosaic reconstructionApply lens correctionReduce noiseColor-match scene-referred sensor values to output-referred color space

Page 51: Live Photo Editing and RAW Processing with Core Image

Stages of RAW image processingAdjusting RAW Images

Extract critical metadataDecode RAW sensor imageDe-mosaic reconstructionApply lens correctionReduce noiseColor-match scene-referred sensor values to output-referred color spaceAdjust exposure and temperature/tint

Page 52: Live Photo Editing and RAW Processing with Core Image

Stages of RAW image processingAdjusting RAW Images

Extract critical metadataDecode RAW sensor imageDe-mosaic reconstructionApply lens correctionReduce noiseColor-match scene-referred sensor values to output-referred color spaceAdjust exposure and temperature/tintAdd sharpening, contrast, and saturation

Page 53: Live Photo Editing and RAW Processing with Core Image

Advantages of RAWAdjusting RAW Images

Page 54: Live Photo Editing and RAW Processing with Core Image

Advantages of RAWAdjusting RAW Images

Contains linear and deep pixel data which enables great editability

Page 55: Live Photo Editing and RAW Processing with Core Image

Advantages of RAWAdjusting RAW Images

Contains linear and deep pixel data which enables great editabilityImage processing gets better every year

Page 56: Live Photo Editing and RAW Processing with Core Image

Advantages of RAWAdjusting RAW Images

Contains linear and deep pixel data which enables great editabilityImage processing gets better every yearCan be rendered to any color space

Page 57: Live Photo Editing and RAW Processing with Core Image

Advantages of RAWAdjusting RAW Images

Contains linear and deep pixel data which enables great editabilityImage processing gets better every yearCan be rendered to any color spaceUsers can use different software to interpret the image

Page 58: Live Photo Editing and RAW Processing with Core Image

Advantages of JPEGAdjusting RAW Images

Page 59: Live Photo Editing and RAW Processing with Core Image

Advantages of JPEGAdjusting RAW Images

Fast to load and display

Page 60: Live Photo Editing and RAW Processing with Core Image

Advantages of JPEGAdjusting RAW Images

Fast to load and displayContains colors targeting a specific color space

Page 61: Live Photo Editing and RAW Processing with Core Image

Advantages of JPEGAdjusting RAW Images

Fast to load and displayContains colors targeting a specific color space Predictable results

Page 62: Live Photo Editing and RAW Processing with Core Image

Advantages of JPEGAdjusting RAW Images

Fast to load and displayContains colors targeting a specific color space Predictable resultsCameras can provide a great default image for display

Page 63: Live Photo Editing and RAW Processing with Core Image

Advantages of JPEGAdjusting RAW Images

Fast to load and displayContains colors targeting a specific color space Predictable resultsCameras can provide a great default image for display• iOS cameras are a good example of this

Page 64: Live Photo Editing and RAW Processing with Core Image

Platform supportAdjusting RAW Images

Page 65: Live Photo Editing and RAW Processing with Core Image

Platform supportAdjusting RAW Images

Now Core Image fully supports RAW on iOS and tvOS

Page 66: Live Photo Editing and RAW Processing with Core Image

Platform supportAdjusting RAW Images

Now Core Image fully supports RAW on iOS and tvOS• Supports over 400 unique camera models from 16 vendors

Page 67: Live Photo Editing and RAW Processing with Core Image

Platform supportAdjusting RAW Images

Now Core Image fully supports RAW on iOS and tvOS• Supports over 400 unique camera models from 16 vendors• Also supports DNG files captured from iOS devices

Page 68: Live Photo Editing and RAW Processing with Core Image

Platform supportAdjusting RAW Images

Now Core Image fully supports RAW on iOS and tvOS• Supports over 400 unique camera models from 16 vendors• Also supports DNG files captured from iOS devices

- iSight cameras on iPhone 6S, iPhone 6S Plus, iPhone SE, iPad Pro (9.7-inch)

Page 69: Live Photo Editing and RAW Processing with Core Image

Platform supportAdjusting RAW Images

Now Core Image fully supports RAW on iOS and tvOS• Supports over 400 unique camera models from 16 vendors• Also supports DNG files captured from iOS devices

- iSight cameras on iPhone 6S, iPhone 6S Plus, iPhone SE, iPad Pro (9.7-inch)

Advances in iOS Photography Pacific Heights Tuesday 11:00AM

Page 70: Live Photo Editing and RAW Processing with Core Image

Platform supportAdjusting RAW Images

Now Core Image fully supports RAW on iOS and tvOS• Supports over 400 unique camera models from 16 vendors• Also supports DNG files captured from iOS devices

- iSight cameras on iPhone 6S, iPhone 6S Plus, iPhone SE, iPad Pro (9.7-inch)• The same high-performance RAW pipeline as on macOS

Advances in iOS Photography Pacific Heights Tuesday 11:00AM

Page 71: Live Photo Editing and RAW Processing with Core Image

Platform supportAdjusting RAW Images

Now Core Image fully supports RAW on iOS and tvOS• Supports over 400 unique camera models from 16 vendors• Also supports DNG files captured from iOS devices

- iSight cameras on iPhone 6S, iPhone 6S Plus, iPhone SE, iPad Pro (9.7-inch)• The same high-performance RAW pipeline as on macOS• Requires A8 or newer processor (iOS GPU Family 2)

Advances in iOS Photography Pacific Heights Tuesday 11:00AM

Page 72: Live Photo Editing and RAW Processing with Core Image

Platform supportAdjusting RAW Images

Page 73: Live Photo Editing and RAW Processing with Core Image

Platform supportAdjusting RAW Images

We continuously add support for cameras and improve quality

Page 74: Live Photo Editing and RAW Processing with Core Image

Platform supportAdjusting RAW Images

We continuously add support for cameras and improve quality• New cameras are added in software updates

Page 75: Live Photo Editing and RAW Processing with Core Image

Platform supportAdjusting RAW Images

We continuously add support for cameras and improve quality• New cameras are added in software updates• Pipeline improvements are versioned

Page 76: Live Photo Editing and RAW Processing with Core Image

DemoAdjusting images on iOS

Page 77: Live Photo Editing and RAW Processing with Core Image

CIRAWFilter API lets you control the stagesAdjusting RAW Images

CIRAWFilter gives your application:

Page 78: Live Photo Editing and RAW Processing with Core Image

CIRAWFilter API lets you control the stagesAdjusting RAW Images

CIRAWFilter gives your application:• CIImage with wide gamut, extended range, half-float precision

Page 79: Live Photo Editing and RAW Processing with Core Image

CIRAWFilter API lets you control the stagesAdjusting RAW Images

CIRAWFilter gives your application:• CIImage with wide gamut, extended range, half-float precision• Easy control over RAW processing parameters

Page 80: Live Photo Editing and RAW Processing with Core Image

CIRAWFilter API lets you control the stagesAdjusting RAW Images

CIRAWFilter gives your application:• CIImage with wide gamut, extended range, half-float precision• Easy control over RAW processing parameters• Fast, interactive performance using GPU

Page 81: Live Photo Editing and RAW Processing with Core Image

Using the CIRAWFilter APIAdjusting RAW Images

RAW Image File• File URL• File data• CVPixelBuffer

Page 82: Live Photo Editing and RAW Processing with Core Image

Using the CIRAWFilter APIAdjusting RAW Images

RAW Image File• File URL• File data• CVPixelBuffer

CIRAWFilter

Page 83: Live Photo Editing and RAW Processing with Core Image

Using the CIRAWFilter APIAdjusting RAW Images

RAW Image File• File URL• File data• CVPixelBuffer

CIRAWFilter

User Adjustments • Exposure• Temperature, tint• Noise reduction

Page 84: Live Photo Editing and RAW Processing with Core Image

Using the CIRAWFilter APIAdjusting RAW Images

RAW Image File• File URL• File data• CVPixelBuffer

CIImageCIRAWFilter

User Adjustments • Exposure• Temperature, tint• Noise reduction

Page 85: Live Photo Editing and RAW Processing with Core Image

// Using the CIRAWFilter API

func getAdjustedRAW(url: URL) -> CIImage?

{

// Load the image

let f = CIFilter(imageURL: url, options:nil)

// Get the NR amount

if let nr = f.value(forKey: kCIInputLuminanceNoiseReductionAmountKey) {

// Change the NR amount

f.setValue(nr.doubleValue + 0.1,

forKey: kCIInputLuminanceNoiseReductionAmountKey)

}

// Get the adjusted image

return f.outputImage

}

Page 86: Live Photo Editing and RAW Processing with Core Image

// Using the CIRAWFilter API

func getAdjustedRAW(url: URL) -> CIImage?

{

// Load the image

let f = CIFilter(imageURL: url, options:nil)

// Get the NR amount

if let nr = f.value(forKey: kCIInputLuminanceNoiseReductionAmountKey) {

// Change the NR amount

f.setValue(nr.doubleValue + 0.1,

forKey: kCIInputLuminanceNoiseReductionAmountKey)

}

// Get the adjusted image

return f.outputImage

}

Page 87: Live Photo Editing and RAW Processing with Core Image

// Using the CIRAWFilter API

func getAdjustedRAW(url: URL) -> CIImage?

{

// Load the image

let f = CIFilter(imageURL: url, options:nil)

// Get the NR amount

if let nr = f.value(forKey: kCIInputLuminanceNoiseReductionAmountKey) {

// Change the NR amount

f.setValue(nr.doubleValue + 0.1,

forKey: kCIInputLuminanceNoiseReductionAmountKey)

}

// Get the adjusted image

return f.outputImage

}

Page 88: Live Photo Editing and RAW Processing with Core Image

// Using the CIRAWFilter API

func getAdjustedRAW(url: URL) -> CIImage?

{

// Load the image

let f = CIFilter(imageURL: url, options:nil)

// Get the NR amount

if let nr = f.value(forKey: kCIInputLuminanceNoiseReductionAmountKey) {

// Change the NR amount

f.setValue(nr.doubleValue + 0.1,

forKey: kCIInputLuminanceNoiseReductionAmountKey)

}

// Get the adjusted image

return f.outputImage

}

Page 89: Live Photo Editing and RAW Processing with Core Image

// Using the CIRAWFilter API

func getAdjustedRAW(url: URL) -> CIImage?

{

// Load the image

let f = CIFilter(imageURL: url, options:nil)

// Get the NR amount

if let nr = f.value(forKey: kCIInputLuminanceNoiseReductionAmountKey) {

// Change the NR amount

f.setValue(nr.doubleValue + 0.1,

forKey: kCIInputLuminanceNoiseReductionAmountKey)

}

// Get the adjusted image

return f.outputImage

}

Page 90: Live Photo Editing and RAW Processing with Core Image

// Using the CIRAWFilter API

func getAdjustedRAW(url: URL) -> CIImage?

{

// Load the image

let f = CIFilter(imageURL: url, options:nil)

// Get the NR amount

if let nr = f.value(forKey: kCIInputLuminanceNoiseReductionAmountKey) {

// Change the NR amount

f.setValue(nr.doubleValue + 0.1,

forKey: kCIInputLuminanceNoiseReductionAmountKey)

}

// Get the adjusted image

return f.outputImage

}

Page 91: Live Photo Editing and RAW Processing with Core Image

Using the CIRAWFilter APIAdjusting RAW Images

RAW Image File• File URL• File data• CVPixelBuffer

CIImage

User Adjustments • Exposure• Temperature, tint• Noise reduction

CIRAWFilter

Page 92: Live Photo Editing and RAW Processing with Core Image

Using the CIRAWFilter APIAdjusting RAW Images

RAW Image File• File URL• File data• CVPixelBuffer

CIImage

User Adjustments • Exposure• Temperature, tint• Noise reduction

CIRAWFilter

Page 93: Live Photo Editing and RAW Processing with Core Image

Using the CIRAWFilter APIAdjusting RAW Images

RAW Image File• File URL• File data• CVPixelBuffer

CIImage

User Adjustments • Exposure• Temperature, tint• Noise reduction

CIRAWFilter

Page 94: Live Photo Editing and RAW Processing with Core Image

Using the CIRAWFilter APIAdjusting RAW Images

RAW Image File• File URL• File data• CVPixelBuffer

CIImage

User Adjustments • Exposure• Temperature, tint• Noise reduction

CIRAWFilter

Page 95: Live Photo Editing and RAW Processing with Core Image

Using the CIRAWFilter APIAdjusting RAW Images

RAW Image File• File URL• File data• CVPixelBuffer

CIImage

User Adjustments • Exposure• Temperature, tint• Noise reduction

CIRAWFilter

Page 96: Live Photo Editing and RAW Processing with Core Image

Using the CIRAWFilter APIAdjusting RAW Images

RAW Image File• File URL• File data• CVPixelBuffer

CIImage

User Adjustments • Exposure• Temperature, tint• Noise reduction

CIRAWFilter

Page 97: Live Photo Editing and RAW Processing with Core Image

Using the CIRAWFilter APIAdjusting RAW Images

RAW Image File• File URL• File data• CVPixelBuffer

CIImage

User Adjustments • Exposure• Temperature, tint• Noise reduction

CIRAWFilter

Page 98: Live Photo Editing and RAW Processing with Core Image

CGImage

Using the CIRAWFilter APIAdjusting RAW Images

RAW Image File• File URL• File data• CVPixelBuffer

CIImage

User Adjustments • Exposure• Temperature, tint• Noise reduction

CIRAWFilter

Page 99: Live Photo Editing and RAW Processing with Core Image

CGImage

Output Image (JPG, TIF,…)

Using the CIRAWFilter APIAdjusting RAW Images

RAW Image File• File URL• File data• CVPixelBuffer

CIImage

User Adjustments • Exposure• Temperature, tint• Noise reduction

CIRAWFilter

Page 100: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a JPEG or TIFF

class myClass {

lazy var contextForSaving: CIContext = CIContext(options:

[kCIContextCacheIntermediates : false,

kCIContextPriorityRequestLow : true]) // Now this works on macOS too!

Page 101: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a JPEG or TIFF

class myClass {

lazy var contextForSaving: CIContext = CIContext(options:

[kCIContextCacheIntermediates : false,

kCIContextPriorityRequestLow : true]) // Now this works on macOS too!

Page 102: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a JPEG or TIFF

class myClass {

lazy var contextForSaving: CIContext = CIContext(options:

[kCIContextCacheIntermediates : false,

kCIContextPriorityRequestLow : true]) // Now this works on macOS too!

Page 103: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a JPEG or TIFF

class myClass {

lazy var contextForSaving: CIContext = CIContext(options:

[kCIContextCacheIntermediates : false,

kCIContextPriorityRequestLow : true]) // Now this works on macOS too!

Page 104: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a JPEG or TIFF

class myClass {

lazy var contextForSaving: CIContext = CIContext(options:

[kCIContextCacheIntermediates : false,

kCIContextPriorityRequestLow : true]) // Now this works on macOS too!

Page 105: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a JPEG or TIFF

func save(from rawImage: CIImage,

to jpegDestination: URL) throws

{

let cs = CGColorSpace(name: CGColorSpace.displayP3)!

try contextForSaving.writeJPEGRepresentation(

of: rawImage,

to: jpegDestination,

colorSpace: cs,

options: [kCGImageDestinationLossyCompressionQuality: 1.0])

}

Page 106: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a JPEG or TIFF

func save(from rawImage: CIImage,

to jpegDestination: URL) throws

{

let cs = CGColorSpace(name: CGColorSpace.displayP3)!

try contextForSaving.writeJPEGRepresentation(

of: rawImage,

to: jpegDestination,

colorSpace: cs,

options: [kCGImageDestinationLossyCompressionQuality: 1.0])

}

Page 107: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a JPEG or TIFF

func save(from rawImage: CIImage,

to jpegDestination: URL) throws

{

let cs = CGColorSpace(name: CGColorSpace.displayP3)!

try contextForSaving.writeJPEGRepresentation(

of: rawImage,

to: jpegDestination,

colorSpace: cs,

options: [kCGImageDestinationLossyCompressionQuality: 1.0])

}

Page 108: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a JPEG or TIFF

func save(from rawImage: CIImage,

to jpegDestination: URL) throws

{

let cs = CGColorSpace(name: CGColorSpace.displayP3)!

try contextForSaving.writeJPEGRepresentation(

of: rawImage,

to: jpegDestination,

colorSpace: cs,

options: [kCGImageDestinationLossyCompressionQuality: 1.0])

}

Page 109: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a JPEG or TIFF

func save(from rawImage: CIImage,

to jpegDestination: URL) throws

{

let cs = CGColorSpace(name: CGColorSpace.displayP3)!

try contextForSaving.writeJPEGRepresentation(

of: rawImage,

to: jpegDestination,

colorSpace: cs,

options: [kCGImageDestinationLossyCompressionQuality: 1.0])

}

Page 110: Live Photo Editing and RAW Processing with Core Image

// Share a RAW to a JPEG or TIFF

// Useful if the receiver doesn't support color management

func share(from rawImage: CIImage,

to jpegDestination: URL) throws

{

let cs = CGColorSpace(name: CGColorSpace.displayP3)!

try contextForSaving.writeJPEGRepresentation(

of: rawImage,

to: jpegDestination,

colorSpace: cs,

options: [kCGImageDestinationLossyCompressionQuality: 1.0,

kCGImageDestinationOptimizeColorForSharing: true])

}

Page 111: Live Photo Editing and RAW Processing with Core Image

// Share a RAW to a JPEG or TIFF

// Useful if the receiver doesn't support color management

func share(from rawImage: CIImage,

to jpegDestination: URL) throws

{

let cs = CGColorSpace(name: CGColorSpace.displayP3)!

try contextForSaving.writeJPEGRepresentation(

of: rawImage,

to: jpegDestination,

colorSpace: cs,

options: [kCGImageDestinationLossyCompressionQuality: 1.0,

kCGImageDestinationOptimizeColorForSharing: true])

}

Page 112: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a CGImageRef

func createCGImage(from rawImage: CIImage) -> CGImage?

{

return contextForSaving.createCGImage(

rawImage,

from: rawImage.extent,

format: kCIFormatRGBA8,

colorSpace: CGColorSpace(name: CGColorSpace.displayP3),

deferred: true) // process the RAW when returned CGImage is drawn

}

Page 113: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a CGImageRef

func createCGImage(from rawImage: CIImage) -> CGImage?

{

return contextForSaving.createCGImage(

rawImage,

from: rawImage.extent,

format: kCIFormatRGBA8,

colorSpace: CGColorSpace(name: CGColorSpace.displayP3),

deferred: true) // process the RAW when returned CGImage is drawn

}

Page 114: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a CGImageRef

func createCGImage(from rawImage: CIImage) -> CGImage?

{

return contextForSaving.createCGImage(

rawImage,

from: rawImage.extent,

format: kCIFormatRGBAh,

colorSpace: CGColorSpace(name: CGColorSpace.extendedLinearSRGB),

deferred: true) // process the RAW when returned CGImage is drawn

}

Page 115: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a CGImageRef

func createCGImage(from rawImage: CIImage) -> CGImage?

{

return contextForSaving.createCGImage(

rawImage,

from: rawImage.extent,

format: kCIFormatRGBAh,

colorSpace: CGColorSpace(name: CGColorSpace.extendedLinearSRGB),

deferred: true) // process the RAW when returned CGImage is drawn

}

Page 116: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a CGImageRef

func createCGImage(from rawImage: CIImage) -> CGImage?

{

return contextForSaving.createCGImage(

rawImage,

from: rawImage.extent,

format: kCIFormatRGBAh,

colorSpace: CGColorSpace(name: CGColorSpace.extendedLinearSRGB),

deferred: false) // process the RAW once before this returns

}

Page 117: Live Photo Editing and RAW Processing with Core Image

// Saving a RAW to a CGImageRef

func createCGImage(from rawImage: CIImage) -> CGImage?

{

return contextForSaving.createCGImage(

rawImage,

from: rawImage.extent,

format: kCIFormatRGBAh,

colorSpace: CGColorSpace(name: CGColorSpace.extendedLinearSRGB),

deferred: false) // process the RAW once before this returns

}

Page 118: Live Photo Editing and RAW Processing with Core Image

Using the CIRAWFilter APIAdjusting RAW Images

RAW Image File• File URL• File data• CVPixelBuffer

CIImage

User Adjustments • Exposure• Temperature, tint• Noise reduction

CIRAWFilter

Page 119: Live Photo Editing and RAW Processing with Core Image

Using the CIRAWFilter APIAdjusting RAW Images

RAW Image File• File URL• File data• CVPixelBuffer

CIImage

User Adjustments • Exposure• Temperature, tint• Noise reduction

CIRAWFilter

Linear Space Filter

Page 120: Live Photo Editing and RAW Processing with Core Image

Supporting wide gamutAdjusting RAW Images

Page 121: Live Photo Editing and RAW Processing with Core Image

Supporting wide gamutAdjusting RAW Images

CIKernel Language uses float precision

Page 122: Live Photo Editing and RAW Processing with Core Image

Supporting wide gamutAdjusting RAW Images

CIKernel Language uses float precision• When needed, intermediate buffers use the CIContext’s current working format

Page 123: Live Photo Editing and RAW Processing with Core Image

Supporting wide gamutAdjusting RAW Images

CIKernel Language uses float precision• When needed, intermediate buffers use the CIContext’s current working format• On macOS, the default working format is kCIFormatRGBAh

Page 124: Live Photo Editing and RAW Processing with Core Image

Supporting wide gamutAdjusting RAW Images

CIKernel Language uses float precision• When needed, intermediate buffers use the CIContext’s current working format• On macOS, the default working format is kCIFormatRGBAh• On iOS/tvOS, the default working format is kCIFormatBGRA8

Page 125: Live Photo Editing and RAW Processing with Core Image

Supporting wide gamutAdjusting RAW Images

CIKernel Language uses float precision• When needed, intermediate buffers use the CIContext’s current working format• On macOS, the default working format is kCIFormatRGBAh• On iOS/tvOS, the default working format is kCIFormatBGRA8• RAW pipeline CIKernels always use kCIFormatRGBAh working format

Page 126: Live Photo Editing and RAW Processing with Core Image

Supporting wide gamutAdjusting RAW Images

CIKernel Language uses float precision• When needed, intermediate buffers use the CIContext’s current working format• On macOS, the default working format is kCIFormatRGBAh• On iOS/tvOS, the default working format is kCIFormatBGRA8• RAW pipeline CIKernels always use kCIFormatRGBAh working format

Create your CIContext with a kCIContextWorkingFormat optionset to kCIFormatRGBAh ensure wide gamut won’t be clipped.

Page 127: Live Photo Editing and RAW Processing with Core Image

Supporting wide gamutAdjusting RAW Images

CIKernel Language uses float precision• When needed, intermediate buffers use the CIContext’s current working format• On macOS, the default working format is kCIFormatRGBAh• On iOS/tvOS, the default working format is kCIFormatBGRA8• RAW pipeline CIKernels always use kCIFormatRGBAh working format

Create your CIContext with a kCIContextWorkingFormat optionset to kCIFormatRGBAh ensure wide gamut won’t be clipped.Core Image supports wide gamut output color spaces

Page 128: Live Photo Editing and RAW Processing with Core Image

Supporting wide gamutAdjusting RAW Images

CIKernel Language uses float precision• When needed, intermediate buffers use the CIContext’s current working format• On macOS, the default working format is kCIFormatRGBAh• On iOS/tvOS, the default working format is kCIFormatBGRA8• RAW pipeline CIKernels always use kCIFormatRGBAh working format

Create your CIContext with a kCIContextWorkingFormat optionset to kCIFormatRGBAh ensure wide gamut won’t be clipped.Core Image supports wide gamut output color spaces• Such as Extended Linear sRGB, Adobe RGB, or Display P3

Page 129: Live Photo Editing and RAW Processing with Core Image

Warning: “Objects are larger than they appear”Saving RAW Images

Page 130: Live Photo Editing and RAW Processing with Core Image

Warning: “Objects are larger than they appear”Saving RAW Images

RAW files can be very large and require several intermediate buffers to render

Page 131: Live Photo Editing and RAW Processing with Core Image

Warning: “Objects are larger than they appear”Saving RAW Images

RAW files can be very large and require several intermediate buffers to renderTo reduce memory high water-mark use these new APIs:

Page 132: Live Photo Editing and RAW Processing with Core Image

Warning: “Objects are larger than they appear”Saving RAW Images

RAW files can be very large and require several intermediate buffers to renderTo reduce memory high water-mark use these new APIs:CIContext(options: [kCIContextCacheIntermediates: false])

Page 133: Live Photo Editing and RAW Processing with Core Image

Warning: “Objects are larger than they appear”Saving RAW Images

RAW files can be very large and require several intermediate buffers to renderTo reduce memory high water-mark use these new APIs:CIContext(options: [kCIContextCacheIntermediates: false])

context.writeJPEGRepresentationOfImage()

Page 134: Live Photo Editing and RAW Processing with Core Image

Warning: “Objects are larger than they appear”Saving RAW Images

RAW files can be very large and require several intermediate buffers to renderTo reduce memory high water-mark use these new APIs:CIContext(options: [kCIContextCacheIntermediates: false])

context.writeJPEGRepresentationOfImage()

context.createCGImage(... deferred: true)

Page 135: Live Photo Editing and RAW Processing with Core Image

Warning: “Objects are larger than they appear”Saving RAW Images

Application Type Supports RAWs

Page 136: Live Photo Editing and RAW Processing with Core Image

Warning: “Objects are larger than they appear”Saving RAW Images

Application Type Supports RAWs

Apps on ≥2GB RAM Devices Up to 120 Megapixels

Page 137: Live Photo Editing and RAW Processing with Core Image

Warning: “Objects are larger than they appear”Saving RAW Images

Application Type Supports RAWs

Apps on ≥2GB RAM Devices Up to 120 Megapixels

Apps on 1GB RAM Devices Up to 60 Megapixels

Page 138: Live Photo Editing and RAW Processing with Core Image

Warning: “Objects are larger than they appear”Saving RAW Images

Application Type Supports RAWs

Apps on ≥2GB RAM Devices Up to 120 Megapixels

Apps on 1GB RAM Devices Up to 60 Megapixels

Photo Editing Extensions Up to 60 Megapixels

Page 139: Live Photo Editing and RAW Processing with Core Image

Editing Live Photos

Etienne Guerard Live Photo Editor-in-Chief

Page 140: Live Photo Editing and RAW Processing with Core Image

AgendaEditing Live Photos

Page 141: Live Photo Editing and RAW Processing with Core Image

AgendaEditing Live Photos

Introduction

Page 142: Live Photo Editing and RAW Processing with Core Image

AgendaEditing Live Photos

IntroductionWhat Can be Edited?

Page 143: Live Photo Editing and RAW Processing with Core Image

AgendaEditing Live Photos

IntroductionWhat Can be Edited?Obtaining a Live Photo for Editing

Page 144: Live Photo Editing and RAW Processing with Core Image

AgendaEditing Live Photos

IntroductionWhat Can be Edited?Obtaining a Live Photo for EditingSetting Up a Live Photo Editing Context

Page 145: Live Photo Editing and RAW Processing with Core Image

AgendaEditing Live Photos

IntroductionWhat Can be Edited?Obtaining a Live Photo for EditingSetting Up a Live Photo Editing ContextApplying Core Image Filters

Page 146: Live Photo Editing and RAW Processing with Core Image

AgendaEditing Live Photos

IntroductionWhat Can be Edited?Obtaining a Live Photo for EditingSetting Up a Live Photo Editing ContextApplying Core Image Filters Previewing an Edited Live Photo

Page 147: Live Photo Editing and RAW Processing with Core Image

AgendaEditing Live Photos

IntroductionWhat Can be Edited?Obtaining a Live Photo for EditingSetting Up a Live Photo Editing ContextApplying Core Image Filters Previewing an Edited Live PhotoSaving to the PhotoLibrary

Page 148: Live Photo Editing and RAW Processing with Core Image

AgendaEditing Live Photos

IntroductionWhat Can be Edited?Obtaining a Live Photo for EditingSetting Up a Live Photo Editing ContextApplying Core Image Filters Previewing an Edited Live PhotoSaving to the PhotoLibraryDemo

Page 149: Live Photo Editing and RAW Processing with Core Image

IntroductionLive Photo

Page 150: Live Photo Editing and RAW Processing with Core Image

IntroductionLive Photo

Live Photos include audio, photo, and video media

Page 151: Live Photo Editing and RAW Processing with Core Image

IntroductionLive Photo

Live Photos include audio, photo, and video mediaLive Photos can be captured on recent devices

Page 152: Live Photo Editing and RAW Processing with Core Image

IntroductionLive Photo

Live Photos include audio, photo, and video mediaLive Photos can be captured on recent devicesNew this year:

NEW

Page 153: Live Photo Editing and RAW Processing with Core Image

IntroductionLive Photo

Live Photos include audio, photo, and video mediaLive Photos can be captured on recent devicesNew this year:• Users can fully edit Live Photos in Photos

NEW

Page 154: Live Photo Editing and RAW Processing with Core Image

IntroductionLive Photo

Live Photos include audio, photo, and video mediaLive Photos can be captured on recent devicesNew this year:• Users can fully edit Live Photos in Photos• New API to capture Live Photos

NEW

Advances in iOS Photography Pacific Heights Tuesday 11:00AM

Page 155: Live Photo Editing and RAW Processing with Core Image

IntroductionLive Photo

Live Photos include audio, photo, and video mediaLive Photos can be captured on recent devicesNew this year:• Users can fully edit Live Photos in Photos• New API to capture Live Photos• New API to edit Live Photos!

NEW

Advances in iOS Photography Pacific Heights Tuesday 11:00AM

Page 156: Live Photo Editing and RAW Processing with Core Image

What can be edited?Live Photo

Page 157: Live Photo Editing and RAW Processing with Core Image

What can be edited?Live Photo

Photo

Page 158: Live Photo Editing and RAW Processing with Core Image

What can be edited?Live Photo

PhotoVideo frames

Page 159: Live Photo Editing and RAW Processing with Core Image

What can be edited?Live Photo

PhotoVideo framesAudio volume

Page 160: Live Photo Editing and RAW Processing with Core Image

What can be edited?Live Photo

PhotoVideo framesAudio volumeDimensions

Page 161: Live Photo Editing and RAW Processing with Core Image

Photo editing extensionObtaining a Live Photo for Editing

<!—- Info.plist —>

<key>NSExtension</key>

<dict>

<key>NSExtensionAttributes</key>

<dict>

<key>PHSupportedMediaTypes</key>

<array>

<string>LivePhoto</string>

</array>

</dict>

</dict>

Page 162: Live Photo Editing and RAW Processing with Core Image

Photo editing extensionObtaining a Live Photo for Editing

<!—- Info.plist —>

<key>NSExtension</key>

<dict>

<key>NSExtensionAttributes</key>

<dict>

<key>PHSupportedMediaTypes</key>

<array>

<string>LivePhoto</string>

</array>

</dict>

</dict>

Page 163: Live Photo Editing and RAW Processing with Core Image

Photo editing extensionObtaining a Live Photo for Editing

// Called automatically by Photos when your extension starts

func startContentEditing(input: PHContentEditingInput, placeholderImage: UIImage) {

// See if we have a Live Photo

if input.mediaType == .image && input.mediaSubtypes.contains(.photoLive) {

// Edit Live Photo

// ...

}

else {

// Not a Live Photo

}

}

Page 164: Live Photo Editing and RAW Processing with Core Image

Photo editing extensionObtaining a Live Photo for Editing

// Called automatically by Photos when your extension starts

func startContentEditing(input: PHContentEditingInput, placeholderImage: UIImage) {

// See if we have a Live Photo

if input.mediaType == .image && input.mediaSubtypes.contains(.photoLive) {

// Edit Live Photo

// ...

}

else {

// Not a Live Photo

}

}

Page 165: Live Photo Editing and RAW Processing with Core Image

Photo editing extensionObtaining a Live Photo for Editing

// Called automatically by Photos when your extension starts

func startContentEditing(input: PHContentEditingInput, placeholderImage: UIImage) {

// See if we have a Live Photo

if input.mediaType == .image && input.mediaSubtypes.contains(.photoLive) {

// Edit Live Photo

// ...

}

else {

// Not a Live Photo

}

}

Page 166: Live Photo Editing and RAW Processing with Core Image

PhotoKit AppObtaining a Live Photo for Editing

// Request a content editing input for a PHAsset

asset.requestContentEditingInput(options) {

(input: PHContentEditingInput?, info: [NSObject: AnyObject]) in

guard let input = input else { print("Error: \(info)"); return }

// See if we have a live photo

if input.mediaType == .image && input.mediaSubtypes.contains(.photoLive) {

// Edit Live Photo

// ...

}

else {

// Not a Live Photo

}

}

Page 167: Live Photo Editing and RAW Processing with Core Image

PhotoKit AppObtaining a Live Photo for Editing

// Request a content editing input for a PHAsset

asset.requestContentEditingInput(options) {

(input: PHContentEditingInput?, info: [NSObject: AnyObject]) in

guard let input = input else { print("Error: \(info)"); return }

// See if we have a live photo

if input.mediaType == .image && input.mediaSubtypes.contains(.photoLive) {

// Edit Live Photo

// ...

}

else {

// Not a Live Photo

}

}

Page 168: Live Photo Editing and RAW Processing with Core Image

PhotoKit AppObtaining a Live Photo for Editing

// Request a content editing input for a PHAsset

asset.requestContentEditingInput(options) {

(input: PHContentEditingInput?, info: [NSObject: AnyObject]) in

guard let input = input else { print("Error: \(info)"); return }

// See if we have a live photo

if input.mediaType == .image && input.mediaSubtypes.contains(.photoLive) {

// Edit Live Photo

// ...

}

else {

// Not a Live Photo

}

}

Page 169: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoEditingContextSetting Up a Live Photo Editing Context

Page 170: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoEditingContextSetting Up a Live Photo Editing Context

Info about the Live Photo

Page 171: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoEditingContextSetting Up a Live Photo Editing Context

Info about the Live PhotoFrame processor block

Page 172: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoEditingContextSetting Up a Live Photo Editing Context

Info about the Live PhotoFrame processor blockAudio volume

Page 173: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoEditingContextSetting Up a Live Photo Editing Context

Info about the Live PhotoFrame processor blockAudio volumePrepare Live Photo for playback

Page 174: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoEditingContextSetting Up a Live Photo Editing Context

Info about the Live PhotoFrame processor blockAudio volumePrepare Live Photo for playbackProcess Live Photo for saving

Page 175: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoEditingContextSetting Up a Live Photo Editing Context

Info about the Live PhotoFrame processor blockAudio volumePrepare Live Photo for playbackProcess Live Photo for saving

// Setup Live Photo editing context

self.context = PHLivePhotoEditingContext(livePhotoEditingInput: input)

Page 176: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoFrameWorking with the Frame Processor

Page 177: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoFrameWorking with the Frame Processor

Input image

Page 178: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoFrameWorking with the Frame Processor

Input imageFrame type

Page 179: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoFrameWorking with the Frame Processor

Input imageFrame typeFrame time

Page 180: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoFrameWorking with the Frame Processor

Input imageFrame typeFrame timeRender scale

Page 181: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoFrameWorking with the Frame Processor

Input imageFrame typeFrame timeRender scale

self.livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

// Your adjustments go here...

return frame.image

}

Page 182: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoFrameWorking with the Frame Processor

Input imageFrame typeFrame timeRender scale

self.livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

// Your adjustments go here...

return frame.image

}

Page 183: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoFrameWorking with the Frame Processor

Input imageFrame typeFrame timeRender scale

self.livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

// Your adjustments go here...

return frame.image

}

Page 184: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoFrameWorking with the Frame Processor

Input imageFrame typeFrame timeRender scale

self.livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

// Your adjustments go here...

return frame.image

}

Page 185: Live Photo Editing and RAW Processing with Core Image

647 x 1150

Page 186: Live Photo Editing and RAW Processing with Core Image

// Applying a static adjustment

self.livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

var image = frame.image

// Crop to square

let extent = image.extent

let size = min(extent.width, extent.height)

let rect = CGRect(x: (extent.width - size) / 2, y: (extent.height - size) / 2,

width: size, height: size)

image = image.cropping(to: rect)

return image

}

Page 187: Live Photo Editing and RAW Processing with Core Image

// Applying a static adjustment

self.livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

var image = frame.image

// Crop to square

let extent = image.extent

let size = min(extent.width, extent.height)

let rect = CGRect(x: (extent.width - size) / 2, y: (extent.height - size) / 2,

width: size, height: size)

image = image.cropping(to: rect)

return image

}

Page 188: Live Photo Editing and RAW Processing with Core Image

// Applying a static adjustment

self.livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

var image = frame.image

// Crop to square

let extent = image.extent

let size = min(extent.width, extent.height)

let rect = CGRect(x: (extent.width - size) / 2, y: (extent.height - size) / 2,

width: size, height: size)

image = image.cropping(to: rect)

return image

}

Page 189: Live Photo Editing and RAW Processing with Core Image

// Applying a static adjustment

self.livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

var image = frame.image

// Crop to square

let extent = image.extent

let size = min(extent.width, extent.height)

let rect = CGRect(x: (extent.width - size) / 2, y: (extent.height - size) / 2,

width: size, height: size)

image = image.cropping(to: rect)

return image

}

Page 190: Live Photo Editing and RAW Processing with Core Image

647 x 1150

Page 191: Live Photo Editing and RAW Processing with Core Image

// Applying a time-based adjustment

let tP = CMTimeGetSeconds(self.livePhotoEditingContext.photoTime)

let duration = CMTimeGetSeconds(self.livePhotoEditingContext.duration)

self.livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

var image = frame.image

let tF = CMTimeGetSeconds(frame.time)

// Simple linear ramp function from (0, tP, duration) to (-1, 0, +1)

let dt = (tF < tP) ? CGFloat((tF - tP) / tP) : CGFloat((tF - tP) / (duration - tP))

// Animate crop rect

image = image.cropping(to: rect.offsetBy(dx: dt * rect.minX, dy: dt * rect.minY))

return image

}

Page 192: Live Photo Editing and RAW Processing with Core Image

// Applying a time-based adjustment

let tP = CMTimeGetSeconds(self.livePhotoEditingContext.photoTime)

let duration = CMTimeGetSeconds(self.livePhotoEditingContext.duration)

self.livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

var image = frame.image

let tF = CMTimeGetSeconds(frame.time)

// Simple linear ramp function from (0, tP, duration) to (-1, 0, +1)

let dt = (tF < tP) ? CGFloat((tF - tP) / tP) : CGFloat((tF - tP) / (duration - tP))

// Animate crop rect

image = image.cropping(to: rect.offsetBy(dx: dt * rect.minX, dy: dt * rect.minY))

return image

}

Page 193: Live Photo Editing and RAW Processing with Core Image

// Applying a time-based adjustment

let tP = CMTimeGetSeconds(self.livePhotoEditingContext.photoTime)

let duration = CMTimeGetSeconds(self.livePhotoEditingContext.duration)

self.livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

var image = frame.image

let tF = CMTimeGetSeconds(frame.time)

// Simple linear ramp function from (0, tP, duration) to (-1, 0, +1)

let dt = (tF < tP) ? CGFloat((tF - tP) / tP) : CGFloat((tF - tP) / (duration - tP))

// Animate crop rect

image = image.cropping(to: rect.offsetBy(dx: dt * rect.minX, dy: dt * rect.minY))

return image

}

Page 194: Live Photo Editing and RAW Processing with Core Image

// Applying a time-based adjustment

let tP = CMTimeGetSeconds(self.livePhotoEditingContext.photoTime)

let duration = CMTimeGetSeconds(self.livePhotoEditingContext.duration)

self.livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

var image = frame.image

let tF = CMTimeGetSeconds(frame.time)

// Simple linear ramp function from (0, tP, duration) to (-1, 0, +1)

let dt = (tF < tP) ? CGFloat((tF - tP) / tP) : CGFloat((tF - tP) / (duration - tP))

// Animate crop rect

image = image.cropping(to: rect.offsetBy(dx: dt * rect.minX, dy: dt * rect.minY))

return image

}

Page 195: Live Photo Editing and RAW Processing with Core Image

// Applying a time-based adjustment

let tP = CMTimeGetSeconds(self.livePhotoEditingContext.photoTime)

let duration = CMTimeGetSeconds(self.livePhotoEditingContext.duration)

self.livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

var image = frame.image

let tF = CMTimeGetSeconds(frame.time)

// Simple linear ramp function from (0, tP, duration) to (-1, 0, +1)

let dt = (tF < tP) ? CGFloat((tF - tP) / tP) : CGFloat((tF - tP) / (duration - tP))

// Animate crop rect

image = image.cropping(to: rect.offsetBy(dx: dt * rect.minX, dy: dt * rect.minY))

return image

}

Page 196: Live Photo Editing and RAW Processing with Core Image

647 x 1150

Page 197: Live Photo Editing and RAW Processing with Core Image

647 x 1150

Page 198: Live Photo Editing and RAW Processing with Core Image

// Applying a resolution-dependent adjustment

livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

var image = frame.image

// Apply screen effect

let scale = frame.renderScale

image = image.applyingFilter("CILineScreen", withInputParameters:

[ "inputAngle" : 3 * Double.pi / 4,

"inputWidth" : 50 * scale,

"inputCenter" : CIVector(x: image.extent.midX, y: image.extent.midY)

])

return image

}

Page 199: Live Photo Editing and RAW Processing with Core Image

// Applying a resolution-dependent adjustment

livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

var image = frame.image

// Apply screen effect

let scale = frame.renderScale

image = image.applyingFilter("CILineScreen", withInputParameters:

[ "inputAngle" : 3 * Double.pi / 4,

"inputWidth" : 50 * scale,

"inputCenter" : CIVector(x: image.extent.midX, y: image.extent.midY)

])

return image

}

Page 200: Live Photo Editing and RAW Processing with Core Image

// Applying a resolution-dependent adjustment

livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

var image = frame.image

// Apply screen effect

let scale = frame.renderScale

image = image.applyingFilter("CILineScreen", withInputParameters:

[ "inputAngle" : 3 * Double.pi / 4,

"inputWidth" : 50 * scale,

"inputCenter" : CIVector(x: image.extent.midX, y: image.extent.midY)

])

return image

}

Page 201: Live Photo Editing and RAW Processing with Core Image

// Applying a resolution-dependent adjustment

livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

var image = frame.image

// Apply screen effect

let scale = frame.renderScale

image = image.applyingFilter("CILineScreen", withInputParameters:

[ "inputAngle" : 3 * Double.pi / 4,

"inputWidth" : 50 * scale,

"inputCenter" : CIVector(x: image.extent.midX, y: image.extent.midY)

])

return image

}

Page 202: Live Photo Editing and RAW Processing with Core Image

647 x 1150

Page 203: Live Photo Editing and RAW Processing with Core Image

// Applying an adjustment to the photo only

livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

var image = frame.image

// Add watermark to the photo only

if frame.type == .photo {

// Composite logo

image = logo.applyingFilter("CILinearDodgeBlendMode",

withInputParameters: ["inputBackgroundImage" : image])

}

return image

}

Page 204: Live Photo Editing and RAW Processing with Core Image

// Applying an adjustment to the photo only

livePhotoEditingContext.frameProcessor = {

(frame: PHLivePhotoFrame, error: NSErrorPointer) -> CIImage? in

var image = frame.image

// Add watermark to the photo only

if frame.type == .photo {

// Composite logo

image = logo.applyingFilter("CILinearDodgeBlendMode",

withInputParameters: ["inputBackgroundImage" : image])

}

return image

}

Page 205: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoViewPreviewing a Live Photo

// Prepare Live Photo for playback

self.livePhotoEditingContext.prepareLivePhotoForPlayback(withTargetSize: targetSize,

options: nil) {

(livePhoto: PHLivePhoto?, error: NSError?) in

guard let livePhoto = livePhoto else { print("Prepare error: \(error)"); return }

// Update live photo view

self.livePhotoView.livePhoto = livePhoto

}

Page 206: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoViewPreviewing a Live Photo

// Prepare Live Photo for playback

self.livePhotoEditingContext.prepareLivePhotoForPlayback(withTargetSize: targetSize,

options: nil) {

(livePhoto: PHLivePhoto?, error: NSError?) in

guard let livePhoto = livePhoto else { print("Prepare error: \(error)"); return }

// Update live photo view

self.livePhotoView.livePhoto = livePhoto

}

Page 207: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoViewPreviewing a Live Photo

// Prepare Live Photo for playback

self.livePhotoEditingContext.prepareLivePhotoForPlayback(withTargetSize: targetSize,

options: nil) {

(livePhoto: PHLivePhoto?, error: NSError?) in

guard let livePhoto = livePhoto else { print("Prepare error: \(error)"); return }

// Update live photo view

self.livePhotoView.livePhoto = livePhoto

}

Page 208: Live Photo Editing and RAW Processing with Core Image

PHLivePhotoViewPreviewing a Live Photo

// Prepare Live Photo for playback

self.livePhotoEditingContext.prepareLivePhotoForPlayback(withTargetSize: targetSize,

options: nil) {

(livePhoto: PHLivePhoto?, error: NSError?) in

guard let livePhoto = livePhoto else { print("Prepare error: \(error)"); return }

// Update live photo view

self.livePhotoView.livePhoto = livePhoto

}

Page 209: Live Photo Editing and RAW Processing with Core Image

Photo editing extensionSaving to the Photo Library

// Called automatically by Photos to save the edits

func finishContentEditing(completionHandler: (PHContentEditingOutput?) -> Void) {

let output = PHContentEditingOutput(contentEditingInput: self.contentEditingInput)

self.livePhotoEditingContext.saveLivePhoto(to: output, options: nil) {

(success: Bool, error: NSError?) in

if success {

output.adjustmentData = PHAdjustmentData(/* Your adjustment data */)

completionHandler(output)

}

}

}

Page 210: Live Photo Editing and RAW Processing with Core Image

Photo editing extensionSaving to the Photo Library

// Called automatically by Photos to save the edits

func finishContentEditing(completionHandler: (PHContentEditingOutput?) -> Void) {

let output = PHContentEditingOutput(contentEditingInput: self.contentEditingInput)

self.livePhotoEditingContext.saveLivePhoto(to: output, options: nil) {

(success: Bool, error: NSError?) in

if success {

output.adjustmentData = PHAdjustmentData(/* Your adjustment data */)

completionHandler(output)

}

}

}

Page 211: Live Photo Editing and RAW Processing with Core Image

Photo editing extensionSaving to the Photo Library

// Called automatically by Photos to save the edits

func finishContentEditing(completionHandler: (PHContentEditingOutput?) -> Void) {

let output = PHContentEditingOutput(contentEditingInput: self.contentEditingInput)

self.livePhotoEditingContext.saveLivePhoto(to: output, options: nil) {

(success: Bool, error: NSError?) in

if success {

output.adjustmentData = PHAdjustmentData(/* Your adjustment data */)

completionHandler(output)

}

}

}

Page 212: Live Photo Editing and RAW Processing with Core Image

Photo editing extensionSaving to the Photo Library

// Called automatically by Photos to save the edits

func finishContentEditing(completionHandler: (PHContentEditingOutput?) -> Void) {

let output = PHContentEditingOutput(contentEditingInput: self.contentEditingInput)

self.livePhotoEditingContext.saveLivePhoto(to: output, options: nil) {

(success: Bool, error: NSError?) in

if success {

output.adjustmentData = PHAdjustmentData(/* Your adjustment data */)

completionHandler(output)

}

}

}

Page 213: Live Photo Editing and RAW Processing with Core Image

Photo editing extensionSaving to the Photo Library

// Called automatically by Photos to save the edits

func finishContentEditing(completionHandler: (PHContentEditingOutput?) -> Void) {

let output = PHContentEditingOutput(contentEditingInput: self.contentEditingInput)

self.livePhotoEditingContext.saveLivePhoto(to: output, options: nil) {

(success: Bool, error: NSError?) in

if success {

output.adjustmentData = PHAdjustmentData(/* Your adjustment data */)

completionHandler(output)

}

}

}

Page 214: Live Photo Editing and RAW Processing with Core Image

Photo editing extensionSaving to the Photo Library

// Called automatically by Photos to save the edits

func finishContentEditing(completionHandler: (PHContentEditingOutput?) -> Void) {

let output = PHContentEditingOutput(contentEditingInput: self.contentEditingInput)

self.livePhotoEditingContext.saveLivePhoto(to: output, options: nil) {

(success: Bool, error: NSError?) in

if success {

output.adjustmentData = PHAdjustmentData(/* Your adjustment data */)

completionHandler(output)

}

}

}

Page 215: Live Photo Editing and RAW Processing with Core Image

PhotoKit AppSaving to the Photo Library

let output = PHContentEditingOutput(contentEditingInput: self.contentEditingInput)

self.livePhotoEditingContext.saveLivePhoto(to: output, options: nil) {

(success: Bool, error: NSError?) in

if success {

output.adjustmentData = PHAdjustmentData(/* Your adjustment data */)

PHPhotoLibrary.shared().performChanges({

PHAssetChangeRequest(for: asset).contentEditingOutput = output

}) { (success: Bool, error: NSError?) in

// Completion handler

}

}

}

Page 216: Live Photo Editing and RAW Processing with Core Image

PhotoKit AppSaving to the Photo Library

let output = PHContentEditingOutput(contentEditingInput: self.contentEditingInput)

self.livePhotoEditingContext.saveLivePhoto(to: output, options: nil) {

(success: Bool, error: NSError?) in

if success {

output.adjustmentData = PHAdjustmentData(/* Your adjustment data */)

PHPhotoLibrary.shared().performChanges({

PHAssetChangeRequest(for: asset).contentEditingOutput = output

}) { (success: Bool, error: NSError?) in

// Completion handler

}

}

}

Page 217: Live Photo Editing and RAW Processing with Core Image

DemoLive Photo editing extension

Page 218: Live Photo Editing and RAW Processing with Core Image

SummaryEditing Live Photos

Page 219: Live Photo Editing and RAW Processing with Core Image

SummaryEditing Live Photos

What you’ve learned so far

Page 220: Live Photo Editing and RAW Processing with Core Image

SummaryEditing Live Photos

What you’ve learned so far• How to use the Live Photo editing context and the frame processor

Page 221: Live Photo Editing and RAW Processing with Core Image

SummaryEditing Live Photos

What you’ve learned so far• How to use the Live Photo editing context and the frame processor• How to preview a Live Photo using a Live Photo view

Page 222: Live Photo Editing and RAW Processing with Core Image

SummaryEditing Live Photos

What you’ve learned so far• How to use the Live Photo editing context and the frame processor• How to preview a Live Photo using a Live Photo view• How to save a Live Photo back to the Photo Library

Page 223: Live Photo Editing and RAW Processing with Core Image

SummaryEditing Live Photos

What you’ve learned so far• How to use the Live Photo editing context and the frame processor• How to preview a Live Photo using a Live Photo view• How to save a Live Photo back to the Photo Library

Remember

Page 224: Live Photo Editing and RAW Processing with Core Image

SummaryEditing Live Photos

What you’ve learned so far• How to use the Live Photo editing context and the frame processor• How to preview a Live Photo using a Live Photo view• How to save a Live Photo back to the Photo Library

Remember• Don’t forget to opt-in to Live Photo Editing in your extension’s Info.plist

Page 225: Live Photo Editing and RAW Processing with Core Image

SummaryEditing Live Photos

What you’ve learned so far• How to use the Live Photo editing context and the frame processor• How to preview a Live Photo using a Live Photo view• How to save a Live Photo back to the Photo Library

Remember• Don’t forget to opt-in to Live Photo Editing in your extension’s Info.plist• Make sure to save your adjustment data to the Photo Library

Page 226: Live Photo Editing and RAW Processing with Core Image

SummaryEditing Live Photos

What you’ve learned so far• How to use the Live Photo editing context and the frame processor• How to preview a Live Photo using a Live Photo view• How to save a Live Photo back to the Photo Library

Remember• Don’t forget to opt-in to Live Photo Editing in your extension’s Info.plist• Make sure to save your adjustment data to the Photo Library• Live Photo Editing support should be easy to add to your existing app/extension

Page 227: Live Photo Editing and RAW Processing with Core Image

Extending Core Image Using CIImageProcessor

Alexandre Naaman Lord of Pixelland

Page 228: Live Photo Editing and RAW Processing with Core Image

You can do lots with built-in CIFilters and custom CIKernelsUsing CIImageProcessor

OWorking Spaceto OutputFilterFilter

Output CIImage

FilterInput toWorking SpaceI

Original CIImage

Page 229: Live Photo Editing and RAW Processing with Core Image

Now part of the graph can use something differentUsing CIImageProcessor

OWorking Spaceto OutputFilterProcessorFilterInput to

Working SpaceI

Original CIImage Output CIImage

Page 230: Live Photo Editing and RAW Processing with Core Image

Now part of the graph can use something differentUsing CIImageProcessor

OWorking Spaceto OutputFilterProcessorFilterInput to

Working SpaceI

Original CIImage

Custom CPU code

Output CIImage

Page 231: Live Photo Editing and RAW Processing with Core Image

Now part of the graph can use something differentUsing CIImageProcessor

OWorking Spaceto OutputFilterProcessorFilterInput to

Working SpaceI

Original CIImage

Custom CPU codeCustom Metal code

Output CIImage

Page 232: Live Photo Editing and RAW Processing with Core Image

Core Image WWDC 2014

// Applying a CIKernel in a CIFilter subclass

// Only create the kernel once

static let yourKernel = CIKernel(string:"kernel vec4 your_code_here …”)!

override var outputImage: CIImage!

{

return yourKernel.apply(withExtent: calcExtent(),

roiCallback: { (index, rect) -> CGRect in

return calcROI(rect) },

arguments: [inputImage!, inputArgument!])

}

Page 233: Live Photo Editing and RAW Processing with Core Image

Core Image WWDC 2014

// Applying a CIKernel in a CIFilter subclass

// Only create the kernel once

static let yourKernel = CIKernel(string:"kernel vec4 your_code_here …”)!

override var outputImage: CIImage!

{

return yourKernel.apply(withExtent: calcExtent(),

roiCallback: { (index, rect) -> CGRect in

return calcROI(rect) },

arguments: [inputImage!, inputArgument!])

}

Page 234: Live Photo Editing and RAW Processing with Core Image

Core Image WWDC 2014

// Applying a CIKernel in a CIFilter subclass

// Only create the kernel once

static let yourKernel = CIKernel(string:"kernel vec4 your_code_here …”)!

override var outputImage: CIImage!

{

return yourKernel.apply(withExtent: calcExtent(),

roiCallback: { (index, rect) -> CGRect in

return calcROI(rect) },

arguments: [inputImage!, inputArgument!])

}

Page 235: Live Photo Editing and RAW Processing with Core Image

Core Image WWDC 2014

// Applying a CIKernel in a CIFilter subclass

// Only create the kernel once

static let yourKernel = CIKernel(string:"kernel vec4 your_code_here …”)!

override var outputImage: CIImage!

{

return yourKernel.apply(withExtent: calcExtent(),

roiCallback: { (index, rect) -> CGRect in

return calcROI(rect) },

arguments: [inputImage!, inputArgument!])

}

Page 236: Live Photo Editing and RAW Processing with Core Image

Core Image WWDC 2014

// Applying a CIKernel in a CIFilter subclass

// Only create the kernel once

static let yourKernel = CIKernel(string:"kernel vec4 your_code_here …”)!

override var outputImage: CIImage!

{

return yourKernel.apply(withExtent: calcExtent(),

roiCallback: { (index, rect) -> CGRect in

return calcROI(rect) },

arguments: [inputImage!, inputArgument!])

}

Page 237: Live Photo Editing and RAW Processing with Core Image

// Applying a CIImageProcessor in a CIFilter subclass

override var outputImage: CIImage!

{

return inputImage.withExtent( calcExtent(),

processorDescription: "myProcessor",

argumentDigest: calcDigest(inputArgument: inputArgument),

inputFormat: kCIFormatBGRA8,

outputFormat: kCIFormatRGBAf,

options: [:],

roiCallback: { (rect) -> CGRect in calcROI(rect) },

processor: { ( input: CIImageProcessorInput,

output: CIImageProcessorOutput ) in

// do what you want here

// read from input,

// use inputArgument,

// write to output

})

}

Page 238: Live Photo Editing and RAW Processing with Core Image

// Applying a CIImageProcessor in a CIFilter subclass

override var outputImage: CIImage!

{

return inputImage.withExtent( calcExtent(),

processorDescription: "myProcessor",

argumentDigest: calcDigest(inputArgument: inputArgument),

inputFormat: kCIFormatBGRA8,

outputFormat: kCIFormatRGBAf,

options: [:],

roiCallback: { (rect) -> CGRect in calcROI(rect) },

processor: { ( input: CIImageProcessorInput,

output: CIImageProcessorOutput ) in

// do what you want here

// read from input,

// use inputArgument,

// write to output

})

}

Page 239: Live Photo Editing and RAW Processing with Core Image

// Applying a CIImageProcessor in a CIFilter subclass

override var outputImage: CIImage!

{

return inputImage.withExtent( calcExtent(),

processorDescription: "myProcessor",

argumentDigest: calcDigest(inputArgument: inputArgument),

inputFormat: kCIFormatBGRA8,

outputFormat: kCIFormatRGBAf,

options: [:],

roiCallback: { (rect) -> CGRect in calcROI(rect) },

processor: { ( input: CIImageProcessorInput,

output: CIImageProcessorOutput ) in

// do what you want here

// read from input,

// use inputArgument,

// write to output

})

}

Page 240: Live Photo Editing and RAW Processing with Core Image

// Applying a CIImageProcessor in a CIFilter subclass

override var outputImage: CIImage!

{

return inputImage.withExtent( calcExtent(),

processorDescription: "myProcessor",

argumentDigest: calcDigest(inputArgument: inputArgument),

inputFormat: kCIFormatBGRA8,

outputFormat: kCIFormatRGBAf,

options: [:],

roiCallback: { (rect) -> CGRect in calcROI(rect) },

processor: { ( input: CIImageProcessorInput,

output: CIImageProcessorOutput ) in

// do what you want here

// read from input,

// use inputArgument,

// write to output

})

}

Page 241: Live Photo Editing and RAW Processing with Core Image

// Applying a CIImageProcessor in a CIFilter subclass

override var outputImage: CIImage!

{

return inputImage.withExtent( calcExtent(),

processorDescription: "myProcessor",

argumentDigest: calcDigest(inputArgument: inputArgument),

inputFormat: kCIFormatBGRA8,

outputFormat: kCIFormatRGBAf,

options: [:],

roiCallback: { (rect) -> CGRect in calcROI(rect) },

processor: { ( input: CIImageProcessorInput,

output: CIImageProcessorOutput ) in

// do what you want here

// read from input,

// use inputArgument,

// write to output

})

}

Page 242: Live Photo Editing and RAW Processing with Core Image

// Applying a CIImageProcessor in a CIFilter subclass

override var outputImage: CIImage!

{

return inputImage.withExtent( calcExtent(),

processorDescription: "myProcessor",

argumentDigest: calcDigest(inputArgument: inputArgument),

inputFormat: kCIFormatBGRA8,

outputFormat: kCIFormatRGBAf,

options: [:],

roiCallback: { (rect) -> CGRect in calcROI(rect) },

processor: { ( input: CIImageProcessorInput,

output: CIImageProcessorOutput ) in

// do what you want here

// read from input,

// use inputArgument,

// write to output

})

}

Page 243: Live Photo Editing and RAW Processing with Core Image

// Applying a CIImageProcessor in a CIFilter subclass

override var outputImage: CIImage!

{

return inputImage.withExtent( calcExtent(),

processorDescription: "myProcessor",

argumentDigest: calcDigest(inputArgument: inputArgument),

inputFormat: kCIFormatBGRA8,

outputFormat: kCIFormatRGBAf,

options: [:],

roiCallback: { (rect) -> CGRect in calcROI(rect) },

processor: { ( input: CIImageProcessorInput,

output: CIImageProcessorOutput ) in

// do what you want here

// read from input,

// use inputArgument,

// write to output

})

}

Page 244: Live Photo Editing and RAW Processing with Core Image

Using CIImageProcessor

Page 245: Live Photo Editing and RAW Processing with Core Image

Using CIImageProcessor

Useful when you have an algorithm that isn’t suitable for CIKernel language

Page 246: Live Photo Editing and RAW Processing with Core Image

Using CIImageProcessor

Useful when you have an algorithm that isn’t suitable for CIKernel languageA good example of this is an integral image

Page 247: Live Photo Editing and RAW Processing with Core Image

Using CIImageProcessor

Useful when you have an algorithm that isn’t suitable for CIKernel languageA good example of this is an integral image• Each output pixel contains the sum of all input pixels above and to the left

Page 248: Live Photo Editing and RAW Processing with Core Image

Using CIImageProcessor

Useful when you have an algorithm that isn’t suitable for CIKernel languageA good example of this is an integral image• Each output pixel contains the sum of all input pixels above and to the left• This cannot be calculated as a traditional data-parallel pixel shader

Page 249: Live Photo Editing and RAW Processing with Core Image

What’s an integral image?Using CIImageProcessor

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Input Image Integral Image

1 5 10 13 15

1 7 16 25 30

4 17 34 45 51

10 31 51 66 79

17 40 61 76 92

Page 250: Live Photo Editing and RAW Processing with Core Image

What’s an integral image?Using CIImageProcessor

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Input Image Integral Image

1 5 10 13 15

1 7 16 25 30

4 17 34 45 51

10 31 51 66 79

17 40 61 76 92

Page 251: Live Photo Editing and RAW Processing with Core Image

What’s an integral image?Using CIImageProcessor

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Input Image Integral Image

1 5 10 13 15

1 7 16 25 30

4 17 34 45 51

10 31 51 66 79

17 40 61 76 92

Page 252: Live Photo Editing and RAW Processing with Core Image

What’s an integral image?Using CIImageProcessor

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Input Image Integral Image

1 5 10 13 15

1 7 16 25 30

4 17 34 45 51

10 31 51 66 79

17 40 61 76 92

Page 253: Live Photo Editing and RAW Processing with Core Image

What’s an integral image?Using CIImageProcessor

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Input Image Integral Image

1 5 10 13 15

1 7 16 25 30

4 17 34 45 51

10 31 51 66 79

17 40 61 76 92

Page 254: Live Photo Editing and RAW Processing with Core Image

// CIImageProcessor block of integral image

processor: { ( input: CIImageProcessorInput, output: CIImageProcessorOutput ) in

let inputPointer = UnsafeMutablePointer <UInt8>(input.baseAddress)

let outputPointer = UnsafeMutablePointer <Float>(output.baseAddress)

let outputHeight = UInt(output.region.height)

let outputWidth = UInt(output.region.width) let xShift = UInt(output.region.minX - input.region.minX)

let yShift = UInt(output.region.minY - input.region.minY)

for j in 0..<outputHeight {

for i in 0..<outputWidth {

// ... compute value of output(i,j) from input(i,j,xShift,yShift) }

}

}

Page 255: Live Photo Editing and RAW Processing with Core Image

// CIImageProcessor block of integral image

processor: { ( input: CIImageProcessorInput, output: CIImageProcessorOutput ) in

let inputPointer = UnsafeMutablePointer <UInt8>(input.baseAddress)

let outputPointer = UnsafeMutablePointer <Float>(output.baseAddress)

let outputHeight = UInt(output.region.height)

let outputWidth = UInt(output.region.width) let xShift = UInt(output.region.minX - input.region.minX)

let yShift = UInt(output.region.minY - input.region.minY)

for j in 0..<outputHeight {

for i in 0..<outputWidth {

// ... compute value of output(i,j) from input(i,j,xShift,yShift) }

}

}

Page 256: Live Photo Editing and RAW Processing with Core Image

// CIImageProcessor block of integral image

processor: { ( input: CIImageProcessorInput, output: CIImageProcessorOutput ) in

let inputPointer = UnsafeMutablePointer <UInt8>(input.baseAddress)

let outputPointer = UnsafeMutablePointer <Float>(output.baseAddress)

let outputHeight = UInt(output.region.height)

let outputWidth = UInt(output.region.width) let xShift = UInt(output.region.minX - input.region.minX)

let yShift = UInt(output.region.minY - input.region.minY)

for j in 0..<outputHeight {

for i in 0..<outputWidth {

// ... compute value of output(i,j) from input(i,j,xShift,yShift) }

}

}

Page 257: Live Photo Editing and RAW Processing with Core Image

// CIImageProcessor block of integral image

processor: { ( input: CIImageProcessorInput, output: CIImageProcessorOutput ) in

let inputPointer = UnsafeMutablePointer <UInt8>(input.baseAddress)

let outputPointer = UnsafeMutablePointer <Float>(output.baseAddress)

let outputHeight = UInt(output.region.height)

let outputWidth = UInt(output.region.width) let xShift = UInt(output.region.minX - input.region.minX)

let yShift = UInt(output.region.minY - input.region.minY)

for j in 0..<outputHeight {

for i in 0..<outputWidth {

// ... compute value of output(i,j) from input(i,j,xShift,yShift) }

}

}

Page 258: Live Photo Editing and RAW Processing with Core Image

// CIImageProcessor block of integral image

processor: { ( input: CIImageProcessorInput, output: CIImageProcessorOutput ) in

let inputPointer = UnsafeMutablePointer <UInt8>(input.baseAddress)

let outputPointer = UnsafeMutablePointer <Float>(output.baseAddress)

let outputHeight = UInt(output.region.height)

let outputWidth = UInt(output.region.width) let xShift = UInt(output.region.minX - input.region.minX)

let yShift = UInt(output.region.minY - input.region.minY)

for j in 0..<outputHeight {

for i in 0..<outputWidth {

// ... compute value of output(i,j) from input(i,j,xShift,yShift) }

}

}

Page 259: Live Photo Editing and RAW Processing with Core Image

// CIImageProcessor block of integral image using MPS

processor: { ( input: CIImageProcessorInput, output: CIImageProcessorOutput ) in

let kernel = MPSImageIntegral(device: output.metalCommandBuffer?.device)

let offsetX = output.region.minX - input.region.minX

let offsetY = output.region.minY - input.region.minY

kernel.offset = MPSOffset(x:Int(offsetX), y: Int(offsetY), z: 0)

kernel.encodeToCommandBuffer(output.metalCommandBuffer?,

sourceTexture: input.metalTexture,

destinationTexture: output.metalTexture)

}

Page 260: Live Photo Editing and RAW Processing with Core Image

// CIImageProcessor block of integral image using MPS

processor: { ( input: CIImageProcessorInput, output: CIImageProcessorOutput ) in

let kernel = MPSImageIntegral(device: output.metalCommandBuffer?.device)

let offsetX = output.region.minX - input.region.minX

let offsetY = output.region.minY - input.region.minY

kernel.offset = MPSOffset(x:Int(offsetX), y: Int(offsetY), z: 0)

kernel.encodeToCommandBuffer(output.metalCommandBuffer?,

sourceTexture: input.metalTexture,

destinationTexture: output.metalTexture)

}

Page 261: Live Photo Editing and RAW Processing with Core Image

// CIImageProcessor block of integral image using MPS

processor: { ( input: CIImageProcessorInput, output: CIImageProcessorOutput ) in

let kernel = MPSImageIntegral(device: output.metalCommandBuffer?.device)

let offsetX = output.region.minX - input.region.minX

let offsetY = output.region.minY - input.region.minY

kernel.offset = MPSOffset(x:Int(offsetX), y: Int(offsetY), z: 0)

kernel.encodeToCommandBuffer(output.metalCommandBuffer?,

sourceTexture: input.metalTexture,

destinationTexture: output.metalTexture)

}

Page 262: Live Photo Editing and RAW Processing with Core Image

// CIImageProcessor block of integral image using MPS

processor: { ( input: CIImageProcessorInput, output: CIImageProcessorOutput ) in

let kernel = MPSImageIntegral(device: output.metalCommandBuffer?.device)

let offsetX = output.region.minX - input.region.minX

let offsetY = output.region.minY - input.region.minY

kernel.offset = MPSOffset(x:Int(offsetX), y: Int(offsetY), z: 0)

kernel.encodeToCommandBuffer(output.metalCommandBuffer?,

sourceTexture: input.metalTexture,

destinationTexture: output.metalTexture)

}

Page 263: Live Photo Editing and RAW Processing with Core Image

1176 x 660

Use Integral Image to Do Fast Variable Box Blur

Page 264: Live Photo Editing and RAW Processing with Core Image

1176 x 660

Use Integral Image to Do Fast Variable Box Blur

Page 265: Live Photo Editing and RAW Processing with Core Image

Very fast box sumsHow Can You Use an Integral Image

Input Image

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Page 266: Live Photo Editing and RAW Processing with Core Image

Very fast box sumsHow Can You Use an Integral Image

Input Image

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Page 267: Live Photo Editing and RAW Processing with Core Image

Very fast box sumsHow Can You Use an Integral Image

Input Image

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Page 268: Live Photo Editing and RAW Processing with Core Image

Very fast box sumsHow Can You Use an Integral Image

n² Reads

Input Image

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Page 269: Live Photo Editing and RAW Processing with Core Image

Very fast box sumsHow Can You Use an Integral Image

2n Reads

Input Image

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Page 270: Live Photo Editing and RAW Processing with Core Image

2n Reads

Very fast box sumsHow Can You Use an Integral Image

Input Image

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Integral Image

1 5 10 13 15

1 7 16 25 30

4 17 34 45 51

10 31 51 66 79

17 40 61 76 92

Page 271: Live Photo Editing and RAW Processing with Core Image

2n Reads

Very fast box sumsHow Can You Use an Integral Image

Input Image

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Integral Image

1 5 10 13 15

1 7 16 25 30

4 17 34 45 51

10 31 51 66 79

17 40 61 76 92

Page 272: Live Photo Editing and RAW Processing with Core Image

2n Reads

Very fast box sumsHow Can You Use an Integral Image

Input Image

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Integral Image

1 5 10 13 15

1 7 16 25 30

4 17 34 45 51

10 31 51 66 79

17 40 61 76 92

Page 273: Live Photo Editing and RAW Processing with Core Image

2n Reads

Very fast box sumsHow Can You Use an Integral Image

Input Image

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Integral Image

1 5 10 13 15

1 7 16 25 30

4 17 34 45 51

10 31 51 66 79

17 40 61 76 92

Page 274: Live Photo Editing and RAW Processing with Core Image

2n Reads

Very fast box sumsHow Can You Use an Integral Image

Input Image

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Integral Image

1 5 10 13 15

1 7 16 25 30

4 17 34 45 51

10 31 51 66 79

17 40 61 76 92

Page 275: Live Photo Editing and RAW Processing with Core Image

2n Reads

Very fast box sumsHow Can You Use an Integral Image

4 Reads

Input Image

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Integral Image

1 5 10 13 15

1 7 16 25 30

4 17 34 45 51

10 31 51 66 79

17 40 61 76 92

Page 276: Live Photo Editing and RAW Processing with Core Image

2n Reads

Very fast box sumsHow Can You Use an Integral Image

4 Reads

2 + 4 + 6 + 7 + 8 + 2 + 8 + 3 + 4 == 66 – 10 – 13 + 1

Input Image

1 4 5 3 2

0 2 4 6 3

3 7 8 2 1

6 8 3 4 7

7 2 1 0 3

Integral Image

1 5 10 13 15

1 7 16 25 30

4 17 34 45 51

10 31 51 66 79

17 40 61 76 92

Page 277: Live Photo Editing and RAW Processing with Core Image

// CIKernel box blur from integral image

kernel vec4 boxBlur(sampler image, float radius, vec4 e) {

vec2 c = destCoord();

vec2 lowerLeft = clampToRect(c + vec2(-radius-1.0, -radius), e);

vec2 upperRight = clampToRect(c + vec2(radius, radius+1.0), e);

vec2 diagonal = upperRight - lowerLeft;

float usedArea = abs(diagonal.x * diagonal.y);

float originalArea = (2.0*radius+1.0) * (2.0*radius+1.0);

vec4 ul = sample(image, samplerTransform(image, vec2(lowerLeft.x, upperRight.y)));

vec4 ur = sample(image, samplerTransform(image, upperRight));

vec4 ll = sample(image, samplerTransform(image, lowerLeft));

vec4 lr = sample(image, samplerTransform(image, vec2(upperRight.x, lowerLeft.y)));

return ( ul + lr - ur - ll ) * usedArea / originalArea; }

Page 278: Live Photo Editing and RAW Processing with Core Image

// CIKernel box blur from integral image

kernel vec4 boxBlur(sampler image, float radius, vec4 e) {

vec2 c = destCoord();

vec2 lowerLeft = clampToRect(c + vec2(-radius-1.0, -radius), e);

vec2 upperRight = clampToRect(c + vec2(radius, radius+1.0), e);

vec2 diagonal = upperRight - lowerLeft;

float usedArea = abs(diagonal.x * diagonal.y);

float originalArea = (2.0*radius+1.0) * (2.0*radius+1.0);

vec4 ul = sample(image, samplerTransform(image, vec2(lowerLeft.x, upperRight.y)));

vec4 ur = sample(image, samplerTransform(image, upperRight));

vec4 ll = sample(image, samplerTransform(image, lowerLeft));

vec4 lr = sample(image, samplerTransform(image, vec2(upperRight.x, lowerLeft.y)));

return ( ul + lr - ur - ll ) * usedArea / originalArea; }

Page 279: Live Photo Editing and RAW Processing with Core Image

// CIKernel box blur from integral image

kernel vec4 boxBlur(sampler image, float radius, vec4 e) {

vec2 c = destCoord();

vec2 lowerLeft = clampToRect(c + vec2(-radius-1.0, -radius), e);

vec2 upperRight = clampToRect(c + vec2(radius, radius+1.0), e);

vec2 diagonal = upperRight - lowerLeft;

float usedArea = abs(diagonal.x * diagonal.y);

float originalArea = (2.0*radius+1.0) * (2.0*radius+1.0);

vec4 ul = sample(image, samplerTransform(image, vec2(lowerLeft.x, upperRight.y)));

vec4 ur = sample(image, samplerTransform(image, upperRight));

vec4 ll = sample(image, samplerTransform(image, lowerLeft));

vec4 lr = sample(image, samplerTransform(image, vec2(upperRight.x, lowerLeft.y)));

return ( ul + lr - ur - ll ) * usedArea / originalArea; }

Page 280: Live Photo Editing and RAW Processing with Core Image

// CIKernel box blur from integral image

kernel vec4 boxBlur(sampler image, float radius, vec4 e) {

vec2 c = destCoord();

vec2 lowerLeft = clampToRect(c + vec2(-radius-1.0, -radius), e);

vec2 upperRight = clampToRect(c + vec2(radius, radius+1.0), e);

vec2 diagonal = upperRight - lowerLeft;

float usedArea = abs(diagonal.x * diagonal.y);

float originalArea = (2.0*radius+1.0) * (2.0*radius+1.0);

vec4 ul = sample(image, samplerTransform(image, vec2(lowerLeft.x, upperRight.y)));

vec4 ur = sample(image, samplerTransform(image, upperRight));

vec4 ll = sample(image, samplerTransform(image, lowerLeft));

vec4 lr = sample(image, samplerTransform(image, vec2(upperRight.x, lowerLeft.y)));

return ( ul + lr - ur - ll ) * usedArea / originalArea; }

Page 281: Live Photo Editing and RAW Processing with Core Image

// CIKernel box blur from integral image

kernel vec4 boxBlur(sampler image, float radius, vec4 e) {

vec2 c = destCoord();

vec2 lowerLeft = clampToRect(c + vec2(-radius-1.0, -radius), e);

vec2 upperRight = clampToRect(c + vec2(radius, radius+1.0), e);

vec2 diagonal = upperRight - lowerLeft;

float usedArea = abs(diagonal.x * diagonal.y);

float originalArea = (2.0*radius+1.0) * (2.0*radius+1.0);

vec4 ul = sample(image, samplerTransform(image, vec2(lowerLeft.x, upperRight.y)));

vec4 ur = sample(image, samplerTransform(image, upperRight));

vec4 ll = sample(image, samplerTransform(image, lowerLeft));

vec4 lr = sample(image, samplerTransform(image, vec2(upperRight.x, lowerLeft.y)));

return ( ul + lr - ur - ll ) * usedArea / originalArea; }

Page 282: Live Photo Editing and RAW Processing with Core Image

// CIKernel box blur from integral image

kernel vec4 boxBlur(sampler image, float radius, vec4 e) {

vec2 c = destCoord();

vec2 lowerLeft = clampToRect(c + vec2(-radius-1.0, -radius), e);

vec2 upperRight = clampToRect(c + vec2(radius, radius+1.0), e);

vec2 diagonal = upperRight - lowerLeft;

float usedArea = abs(diagonal.x * diagonal.y);

float originalArea = (2.0*radius+1.0) * (2.0*radius+1.0);

vec4 ul = sample(image, samplerTransform(image, vec2(lowerLeft.x, upperRight.y)));

vec4 ur = sample(image, samplerTransform(image, upperRight));

vec4 ll = sample(image, samplerTransform(image, lowerLeft));

vec4 lr = sample(image, samplerTransform(image, vec2(upperRight.x, lowerLeft.y)));

return ( ul + lr - ur - ll ) * usedArea / originalArea; }

Page 283: Live Photo Editing and RAW Processing with Core Image

// CIKernel variable box blur from integral image and mask

kernel vec4 variableBoxBlur (sampler integralImage, sampler maskImage, float radius, vec4 e) __attribute__((outputFormat(kCIFormatRGBAf)))

{

vec4 v = unpremultiply ( sample ( maskImage, samplerCoord ( maskImage ) ) );

radius *= v.r;

return boxBlur (integralImage, radius, e);

}

Page 284: Live Photo Editing and RAW Processing with Core Image

// CIKernel variable box blur from integral image and mask

kernel vec4 variableBoxBlur (sampler integralImage, sampler maskImage, float radius, vec4 e) __attribute__((outputFormat(kCIFormatRGBAf)))

{

vec4 v = unpremultiply ( sample ( maskImage, samplerCoord ( maskImage ) ) );

radius *= v.r;

return boxBlur (integralImage, radius, e);

}

Page 285: Live Photo Editing and RAW Processing with Core Image

// CIKernel variable box blur from integral image and mask

kernel vec4 variableBoxBlur (sampler integralImage, sampler maskImage, float radius, vec4 e) __attribute__((outputFormat(kCIFormatRGBAf)))

{

vec4 v = unpremultiply ( sample ( maskImage, samplerCoord ( maskImage ) ) );

radius *= v.r;

return boxBlur (integralImage, radius, e);

}

Page 286: Live Photo Editing and RAW Processing with Core Image

// CIKernel variable box blur from integral image and mask

kernel vec4 variableBoxBlur (sampler integralImage, sampler maskImage, float radius, vec4 e) __attribute__((outputFormat(kCIFormatRGBAf)))

{

vec4 v = unpremultiply ( sample ( maskImage, samplerCoord ( maskImage ) ) );

radius *= v.r;

return boxBlur (integralImage, radius, e);

}

Page 287: Live Photo Editing and RAW Processing with Core Image

// CIKernel variable box blur from integral image and mask

kernel vec4 variableBoxBlur (sampler integralImage, sampler maskImage, float radius, vec4 e) __attribute__((outputFormat(kCIFormatRGBAf)))

{

vec4 v = unpremultiply ( sample ( maskImage, samplerCoord ( maskImage ) ) );

radius *= v.r;

return boxBlur (integralImage, radius, e);

}

Page 288: Live Photo Editing and RAW Processing with Core Image

// Create a mask image to control size of blur effect (0..1) —> (0..radius)

let maskImage =

CIFilter(name: "CIRadialGradient",

withInputParameters: [

"inputCenter": centerOfEffect,

"inputRadius0": innerRadius,

"inputRadius1": outerRadius,

"inputColor0": CIColor.black(),

"inputColor1": CIColor.white()

])?.outputImage

Page 289: Live Photo Editing and RAW Processing with Core Image
Page 290: Live Photo Editing and RAW Processing with Core Image

1176 x 660

Page 291: Live Photo Editing and RAW Processing with Core Image

1176 x 660

Page 292: Live Photo Editing and RAW Processing with Core Image

Tips and tricksUsing CIImageProcessor

If your processor:• Wants data in a color space other than the context working space,

- Call CIImage.byColorMatchingWorkingSpace(to: CGColorSpace) on the processor input

• Returns data in a color space other than the context working space,- Call CIImage.byColorMatchingColorSpace(toWorking: CGColorSpace)

on the processor output

Page 293: Live Photo Editing and RAW Processing with Core Image

Tips and tricksUsing CIImageProcessor

If your processor:• Wants data in a color space other than the context working space,

- Call CIImage.byColorMatchingWorkingSpace(to: CGColorSpace) on the processor input

• Returns data in a color space other than the context working space,- Call CIImage.byColorMatchingColorSpace(toWorking: CGColorSpace)

on the processor output

You can see how your processor fits into a full-render graph by running with the CI_PRINT_TREE environment variable

Page 294: Live Photo Editing and RAW Processing with Core Image

// Example log with CI_PRINT_TREE=1

initial graph render_to_display (metal context 1 frame 1) extent=[0 0 1532 1032] =

clamptoalpha roi=[0 0 1532 1032]

colormatch workingspace-to-"Color LCD" roi=[0 0 1532 1032]

affine [1 0 0 1 16 16] roi=[0 0 1532 1032]

kernel variableBoxBlur(iImage,rImage,scale=16,origExtent) roi=[-16 -16 1532 1032]

processor integralImage 0x12345678 roi=[-1 -1 1502 1002]

clamp [0 0 1500 1000] roi=[-1 -1 1502 1002] opaque

affine [1 0 0 -1 0 1000] roi=[0 0 1500 1000] opaque

colormatch "sRGB IEC61966-2.1"-to-workingspace roi=[0 0 1500 1000] opaque

IOSurface BGRA8 alpha_one roi=[0 0 1500 1000] opaque

colorkernel _radialGradient(params,c0,c1) roi=[0 0 1500 1000]

Page 295: Live Photo Editing and RAW Processing with Core Image

// Example log with CI_PRINT_TREE=1

initial graph render_to_display (metal context 1 frame 1) extent=[0 0 1532 1032] =

clamptoalpha roi=[0 0 1532 1032]

colormatch workingspace-to-"Color LCD" roi=[0 0 1532 1032]

affine [1 0 0 1 16 16] roi=[0 0 1532 1032]

kernel variableBoxBlur(iImage,rImage,scale=16,origExtent) roi=[-16 -16 1532 1032]

processor integralImage 0x12345678 roi=[-1 -1 1502 1002]

clamp [0 0 1500 1000] roi=[-1 -1 1502 1002] opaque

affine [1 0 0 -1 0 1000] roi=[0 0 1500 1000] opaque

colormatch "sRGB IEC61966-2.1"-to-workingspace roi=[0 0 1500 1000] opaque

IOSurface BGRA8 alpha_one roi=[0 0 1500 1000] opaque

colorkernel _radialGradient(params,c0,c1) roi=[0 0 1500 1000]

Page 296: Live Photo Editing and RAW Processing with Core Image

// Example log with CI_PRINT_TREE=1

initial graph render_to_display (metal context 1 frame 1) extent=[0 0 1532 1032] =

clamptoalpha roi=[0 0 1532 1032]

colormatch workingspace-to-"Color LCD" roi=[0 0 1532 1032]

affine [1 0 0 1 16 16] roi=[0 0 1532 1032]

kernel variableBoxBlur(iImage,rImage,scale=16,origExtent) roi=[-16 -16 1532 1032]

processor integralImage 0x12345678 roi=[-1 -1 1502 1002]

clamp [0 0 1500 1000] roi=[-1 -1 1502 1002] opaque

affine [1 0 0 -1 0 1000] roi=[0 0 1500 1000] opaque

colormatch "sRGB IEC61966-2.1"-to-workingspace roi=[0 0 1500 1000] opaque

IOSurface BGRA8 alpha_one roi=[0 0 1500 1000] opaque

colorkernel _radialGradient(params,c0,c1) roi=[0 0 1500 1000]

Page 297: Live Photo Editing and RAW Processing with Core Image

// Example log with CI_PRINT_TREE=1

initial graph render_to_display (metal context 1 frame 1) extent=[0 0 1532 1032] =

clamptoalpha roi=[0 0 1532 1032]

colormatch workingspace-to-"Color LCD" roi=[0 0 1532 1032]

affine [1 0 0 1 16 16] roi=[0 0 1532 1032]

kernel variableBoxBlur(iImage,rImage,scale=16,origExtent) roi=[-16 -16 1532 1032]

processor integralImage 0x12345678 roi=[-1 -1 1502 1002]

clamp [0 0 1500 1000] roi=[-1 -1 1502 1002] opaque

affine [1 0 0 -1 0 1000] roi=[0 0 1500 1000] opaque

colormatch "sRGB IEC61966-2.1"-to-workingspace roi=[0 0 1500 1000] opaque

IOSurface BGRA8 alpha_one roi=[0 0 1500 1000] opaque

colorkernel _radialGradient(params,c0,c1) roi=[0 0 1500 1000]

Page 298: Live Photo Editing and RAW Processing with Core Image

// Example log with CI_PRINT_TREE=1

initial graph render_to_display (metal context 1 frame 1) extent=[0 0 1532 1032] =

clamptoalpha roi=[0 0 1532 1032]

colormatch workingspace-to-"Color LCD" roi=[0 0 1532 1032]

affine [1 0 0 1 16 16] roi=[0 0 1532 1032]

kernel variableBoxBlur(iImage,rImage,scale=16,origExtent) roi=[-16 -16 1532 1032]

processor integralImage 0x12345678 roi=[-1 -1 1502 1002]

clamp [0 0 1500 1000] roi=[-1 -1 1502 1002] opaque

affine [1 0 0 -1 0 1000] roi=[0 0 1500 1000] opaque

colormatch "sRGB IEC61966-2.1"-to-workingspace roi=[0 0 1500 1000] opaque

IOSurface BGRA8 alpha_one roi=[0 0 1500 1000] opaque

colorkernel _radialGradient(params,c0,c1) roi=[0 0 1500 1000]

Page 299: Live Photo Editing and RAW Processing with Core Image

// Example log with CI_PRINT_TREE=1

initial graph render_to_display (metal context 1 frame 1) extent=[0 0 1532 1032] =

clamptoalpha roi=[0 0 1532 1032]

colormatch workingspace-to-"Color LCD" roi=[0 0 1532 1032]

affine [1 0 0 1 16 16] roi=[0 0 1532 1032]

kernel variableBoxBlur(iImage,rImage,scale=16,origExtent) roi=[-16 -16 1532 1032]

processor integralImage 0x12345678 roi=[-1 -1 1502 1002]

clamp [0 0 1500 1000] roi=[-1 -1 1502 1002] opaque

affine [1 0 0 -1 0 1000] roi=[0 0 1500 1000] opaque

colormatch "sRGB IEC61966-2.1"-to-workingspace roi=[0 0 1500 1000] opaque

IOSurface BGRA8 alpha_one roi=[0 0 1500 1000] opaque

colorkernel _radialGradient(params,c0,c1) roi=[0 0 1500 1000]

Page 300: Live Photo Editing and RAW Processing with Core Image

// Example log with CI_PRINT_TREE=8

programs graph render_to_display (metal context 1 frame 1 tile 1) roi=[0 0 1532 1032] =

program affine(clamp_to_alpha(premultiply(linear_to_srgb(

unpremultiply(color_matrix_3x3(variableBoxBlur(0,1))))))) rois=[0 0 1532 1032]

program RGBAf processor integralImage 0x12345678 () rois=[-1 -1 1502 1002]

program clamp(affine(srgb_to_linear())) rois=[-1 -1 1502 1002]

IOSurface BGRA8 1500x1000 alpha_one edge_clamp rois=[0 0 1500 1000]

program _radialGradient() rois=[0 0 1500 1000]

Page 301: Live Photo Editing and RAW Processing with Core Image

// Example log with CI_PRINT_TREE=8

programs graph render_to_display (metal context 1 frame 1 tile 1) roi=[0 0 1532 1032] =

program affine(clamp_to_alpha(premultiply(linear_to_srgb(

unpremultiply(color_matrix_3x3(variableBoxBlur(0,1))))))) rois=[0 0 1532 1032]

program RGBAf processor integralImage 0x12345678 () rois=[-1 -1 1502 1002]

program clamp(affine(srgb_to_linear())) rois=[-1 -1 1502 1002]

IOSurface BGRA8 1500x1000 alpha_one edge_clamp rois=[0 0 1500 1000]

program _radialGradient() rois=[0 0 1500 1000]

Page 302: Live Photo Editing and RAW Processing with Core Image

// Example log with CI_PRINT_TREE=8

programs graph render_to_display (metal context 1 frame 1 tile 1) roi=[0 0 1532 1032] =

program affine(clamp_to_alpha(premultiply(linear_to_srgb(

unpremultiply(color_matrix_3x3(variableBoxBlur(0,1))))))) rois=[0 0 1532 1032]

program RGBAf processor integralImage 0x12345678 () rois=[-1 -1 1502 1002]

program clamp(affine(srgb_to_linear())) rois=[-1 -1 1502 1002]

IOSurface BGRA8 1500x1000 alpha_one edge_clamp rois=[0 0 1500 1000]

program _radialGradient() rois=[0 0 1500 1000]

Page 303: Live Photo Editing and RAW Processing with Core Image

// Example log with CI_PRINT_TREE=“8 graphviz”

{46} program affine [1 0 0 -1 0 414] clamp_to_alpha premultiply linear_to_srgb unpremultiply kernel variableBoxBlur rois=[0 0 736 414] extent=[0 0 736 414]

{45} program colorkernel _radialGradient rois=[0 0 736 414] extent=[infinite]

{44} program RGBAf processor IntegralImage: 0x12345678 rois=[0 0 736 414] extent=[0 0 736 414]

{42} program affine [1 0 0 -1 0 414] rois=[0 0 736 414] extent=[infinite][0 0 736 414]

{32} IOSurface 0x170000d60 BGRA 735x414 edge_clamp rois=[0 0 736 414] extent=[infinite][0 0 736 414]

programs graph render_to_surface

(metal context 2 frame 1 tile 1) rois=[0 0 736 414]

Page 304: Live Photo Editing and RAW Processing with Core Image

// Example log with CI_PRINT_TREE=“8 graphviz”

{46} program affine [1 0 0 -1 0 414] clamp_to_alpha premultiply linear_to_srgb unpremultiply kernel variableBoxBlur rois=[0 0 736 414] extent=[0 0 736 414]

{45} program colorkernel _radialGradient rois=[0 0 736 414] extent=[infinite]

{44} program RGBAf processor IntegralImage: 0x12345678 rois=[0 0 736 414] extent=[0 0 736 414]

{42} program affine [1 0 0 -1 0 414] rois=[0 0 736 414] extent=[infinite][0 0 736 414]

{32} IOSurface 0x170000d60 BGRA 735x414 edge_clamp rois=[0 0 736 414] extent=[infinite][0 0 736 414]

programs graph render_to_surface

(metal context 2 frame 1 tile 1) rois=[0 0 736 414]

Page 305: Live Photo Editing and RAW Processing with Core Image

The Core Image Book Club Recommends

Page 306: Live Photo Editing and RAW Processing with Core Image

What You Learned Today

How to adjust RAW images on iOSHow to edit Live Photos How to use CIImageProcessor

Page 307: Live Photo Editing and RAW Processing with Core Image

More Information

https://developer.apple.com/wwdc16/505

Page 308: Live Photo Editing and RAW Processing with Core Image

Related Sessions

Advances in iOS Photography Pacific Heights Tuesday 11:00AM

Working with Wide Color Mission Thursday 1:40PM

Page 309: Live Photo Editing and RAW Processing with Core Image

Labs

Live Photo and Core Image Lab Graphics, Games, and Media Lab C Thursday 1:30PM

Live Photo and Core Image Lab Graphics, Games, and Media Lab D Friday 9:00AM

Color Lab Graphics, Games, and Media Lab C Friday 4:00PM

Page 310: Live Photo Editing and RAW Processing with Core Image