A new Post-Processing Pipeline

Wolfgang EngelConfetti

09/26/2013

Confetti

• Think-Tank for game and movie related industries• Middleware Provider

– Aura – Dynamic Global Illumination System– PixelPuzzle – PostFX pipeline– Ephemeris – Dynamic Skydome system

-> License comes with full source-code• Services:

– Hardware vendors– many game developers (“Engine Tuner”)

• Provides software solutions for games, movies and tools for GPU manufacturers

• http://www.conffx.com

Confetti

Tomb-Raider – TressFX / Hair Rendering

Confetti

Confetti

Confetti

Confetti

Confetti

Confetti

Dynamic Global Illumination system / Skydome System with volumetric Clouds

Agenda Post-Processing

• History • Call for a new Post-Processing Pipeline• Yxy Color Space• Dynamic Local Gamma• Depth of Field

History

• The first HDR rendering pipeline appeared in a DirectX SDK in 2004

• From there on we called a collection of image space effects at the end of the rendering pipeline Post-Processing Pipeline

History

• The idea was to re-use resources == render targets and data in the Post-Processing Pipeline to apply effects like – Tone-mapping + HDR rendering == dynamic contrast

operator– Camera effects like Depth of Field, Motion Blur, lens flare– Color filters like contrast, saturation, color additions and

multiplications• One of the first coverages of a collection of effects in a

Post-Processing Pipelines happened at GDC 2007 [Engel2007]

History

• Since then – Numerous new tone mapping operators were

introduced [Day2012]– New more advanced Depth of Field algorithms

with shaped bokeh were introduced• … but nothing changed fundamentally

Call for a new Post-Processing Pipeline

• RGB is not a color space for PostFX-> we should use a color space that uses luminance

• Global tone mapping operators didn’t work out well in practice, only on paper-> your artists probably limit the luminance values and therefore the tone mapping operator because the textures “blow out”

Call for a new Post-Processing Pipeline

• A fixed global gamma adjustment at the end of the pipeline is a waste of cycles – because it does a “similar thing” as the tone

mapper … why not just make it dynamic and part of the tone mapper

– … we can also make it more local and not just globally adjusting gamma

Call for a new Post-Processing Pipeline

• We can also add more stages to the Post-Processing Pipeline

• Adding Screen-space– Ambient occlusion by occupying the fourth channel of the

PostFX render targets or the luminance channel– Skin– Reflections by utilizing the “same” blur kernels

• Any new Post-Processing Pipeline needs to be written in compute-> substantial bandwidth savings and speed increases

Simplified Pipeline Stages from 2007

Depth of FieldTone MappingColor Filters

Gamma Control

Frame-Buffer

16x16 RTLuminance

4x4 RTLuminance

1x1 RTLuminance

64x64 RTLuminance

1x1 RTAdapt Lum.

Measure Luminance

Adapt Luminance

Primary Render Target

BrightPass Filter

Z-Buffer

Downsample½ Size

Downsample¼ Size

Gauss Filter

Gauss Filter II

Yxy Color Space

• Instead of running the Post-Processing Pipeline in RGB we can run it in Yxy

• Y holds then the luminance

Yxy Color Space

• To apply any tone mapping operation, we have to convert RGB into a space that let’s us easily separate luminance-> tone mapping is applied to luminance*

* … don’t apply it to a RGB value … an artist will notice

Yxy Color Space

• If we use Bloom, we have to apply tone mapping twice in our pipeline-> once in the Bright-Pass filter and once in the final pass-> that means we convert from RGB to luminance and from luminance to RGB twice

Yxy Color Space

• If we run the whole pipeline in a color space that holds luminance in a dedicated channel, we only covert once into this color space and at the end of the pipeline back

Yxy Color Space

• For Bloom we can run the bright-pass filter in one channel-> modern scalar GPU hardware -> speed-up

Yxy Color Space

• To apply SSAO … just “mix” it into your Y channel

• … and re-use a blur kernel you use anyway to blur it

Yxy Color Space

• Why Yxy color space?

*based on Greg Wards LogLuv model

High-Dynamic Range Rendering

Color Space

# of cycles (encoding)

Bilinear Filtering

Blur Filter Alpha Blending

RGB - Yes Yes Yes

HSV ~34 Yes No No

CIE Yxy ~19 Yes Yes No

L16uv* ~19 Yes Yes No

RGBE ~13 No No No

Dynamic Local Gamma

• From linear gamma to sRGB

float3 Color; Color = (Color <= 0.00304) ? Color * 12.92 : (1.055 * pow(Color, 1.0/2.4) - 0.055);

Dynamic Local Gamma

• It seems everyone is using a global Gamma setting that is the same for every pixel on screen

• Propose to change Gamma per-pixel-> in fact Gamma correction is considered different depending on brightness-> we just didn’t implement it this way …

Dynamic Local Gamma

• The human eye’s visual gamma changes the perceived luminance for various adaptation conditions [Bartleson 1967] [Kwon 2011]

• If the eye’s adaptation level is low, the exponent for Gamma increases

Dynamic Local Gamma

Changes in relative brightness contrast as a function of adaptation of relative luminance and adaptation luminance levels according to the result [Bartleson 1967]

Dynamic Local Gamma

• Local Gamma varies with luminance [Kwon 2011]*

* … build your own equation, depending on the type of game

Dynamic Local Gamma

• is changed based on the luminance value of the current pixel -> that means each pixels luminance value might be gamma corrected with a different exponent-> with the equation above, the exponent == gamma value is in the range of 0.421 to 0.465

Dynamic Local Gamma

Applied Gamma Curve per-pixel based on luminance of pixel• Eye’s adaptation == low -> blue curve• Eye’s adaptation value == high -> green curve

Dynamic Local Gamma

• works with any tone mapping operator• Example: [Reinhard]

• Artistically desirable to burn out bright areas• Source art not always HDR

• Leaves 0..1

– White = 6.0

•What is the visual difference?• Dynamic light & shadow information is considered for

gamma-> new information introduced in the pipeline is used to modify the gamma correction-> changes “naturally” depending on how bright or dark a scene is

• Light looks better / shadows look better

Dynamic Local Gamma

Depth of Field

• When using cameras, this refers to the distance in a scene that appears in focus

• To add artistic effect and/or focus attention• Physically correct parameters and calculations

needed for a proper looking depth of field– Tradeoff: parameters are not artist friendly

• Need a fast and efficient way to generate a high quality Depth of Field effect

Depth of Field

• Circle of Confusion(CoC)• When using a lens to produce an image, the size of

the resulting “spot” produced by a point in the scene

• The Depth of Field is the region where the CoC is less than the resolution of the human eye (or of the display medium)

Depth of Field

• Circle of Confusion(CoC)– Affected by:

• F-stop - ratio of focal length to aperture size• Focal length – distance from lens to image in focus• Focus distance – distance to plane in focus

Depth of Field

• Calculating Circle of Confusion(CoC)[Potmesil1981]

• CoC is negative for far field, positive for near field• Convert CoC from meters to pixel units to find the

effective CoC on screen

Depth of Field

• Basic depth of field effect:– Calculate CoC for each pixel– Use CoC to generate separate results for near field

and far field• Flat, shaped kernel useful for producing Bokeh effect

– Combine far field and focus field based on CoC– Combine with the near field based on CoC and

near field coverage

Depth of Field

Red = max CoC (near field CoC)Green = min CoC (far field CoC)

Depth of Field

Far field result in Yxy

Depth of Field

Near field result in Yxy

Depth of Field

References• [Bartleson 1967] C. J. Bartleson and E. J. Breneman, “Brightness function: Effects of

adaptation,” J. Opt. Soc. Am., vol. 57, pp. 953-957, 1967. • [Day2012] Mike Day, “An efficient and user-friendly tone mapping operator”,

http://www.insomniacgames.com/mike-day-an-efficient-and-user-friendly-tone-mapping-operator/

• [Engel2007] Wolfgang Engel, “Post-Processing Pipeline”, GDC 2007 http://www.coretechniques.info/index_2007.html

• [Kwon 2011] Hyuk-Ju Kwon, Sung-Hak Lee, Seok-Min Chae, Kyu-Ik Sohng, “Tone Mapping Algorithm for Luminance Separated HDR Rendering Based on Visual Brightness Function”, online at http://world-comp.org/p2012/IPC3874.pdf

• [Reinhard] Erik Reinhard, Michael Stark, Peter Shirley, James Ferwerda, "Photographic Tone Reproduction for Digital Images", http://www.cs.utah.edu/~reinhard/cdrom/