Geometry shader-based bump mapping setup

Mark KilgardNVIDIA Corporation

February 5, 2007Updated October 18, 2007

Overview

• GPU bump-mapping traditionally involves complicated “hard to get right” per-primitive setup math to compute how object-space quantities map into texture space– Existing art requires CPU work

• This invention uses a geometry shader to accomplish this more efficiently– Artist authors traditional textured 3D model– Authors height field with same texture parameterization as the

model’s decal texture– Vertex shader can animate & skin model– Geometry shader constructs object-to-texture space mapping

• Object-space light and view vectors can now be transformed to texture space

– Pixel shader does efficient texture-space bump mapping

Geometry Shader in CgTRIANGLE voidmd2bump_geometry(AttribArray<float4> position : POSITION, AttribArray<float2> texCoord : TEXCOORD0, AttribArray<float3> objPosition : TEXCOORD1, AttribArray<float3> objNormal : TEXCOORD2, AttribArray<float3> objView : TEXCOORD3, AttribArray<float3> objLight : TEXCOORD4){ float3 dXYZdU = objPosition[1] - objPosition[0]; float dSdU = texCoord[1].s - texCoord[0].s; float3 dXYZdV = objPosition[2] - objPosition[0]; float dSdV = texCoord[2].s - texCoord[0].s; float3 tangent = normalize(dSdV * dXYZdU - dSdU * dXYZdV); for (int i=0; i<3; i++) { float3 normal = objNormal[i], binormal = cross(tangent,normal); float3x3 basis = float3x3(tangent, binormal, normal); float3 surfaceLightVector : TEXCOORD1 = mul(basis, objLight[i]); float3 surfaceViewVector : TEXCOORD2 = mul(basis, objView[i]);

emitVertex(position[i], texCoord[i], surfaceLightVector, surfaceViewVector); }} warning: not tolerant of mirroring (see next version)

Canonical TriangleA B

C

A = ( u=0, v=0, x0, y0, z0, s0, t0 )B = ( u=1, v=0, x1, y1, z1, s1, t1 )C = ( u=0, v=1, x2, y2, z2, s2, t2 )

∂xyz / ∂u = (x1,y1,z1) - (x0,y0,z0)∂xyz / ∂v = (x2,y2,z2) - (x0,y0,z0)∂s / ∂u = s1 – s0∂t / ∂v = t2 – t0

v

u

vs

us

us

vxyz

vs

uxyz

vs

vxyz

us

uxyz

s

xyz

quotient rule for

partial derivatives:

us

vxyz

vs

uxyz

es

xyznormaliznormalizetangent

Forming anObject-to-Texture Space Basis

z

y

x

z

y

x

obj

obj

obj

tex

tex

tex

NzBzTz

NyByTy

NxBxTx

us

vxyz

vs

uxyz

TzTyTx normalize),,(tangent

normalvertex -per),,(normal NzNyNx

),,(),,(binormal NzNyNxTzTyTx assumes left-handedtexture space

provided by vertex shader

Transform Object-Space Vectors for Lighting to Texture-Space

z

y

x

z

y

x

L

L

L

L

L

L

NzBzTz

NyByTy

NxBxTx

z

y

x

z

y

x

V

V

V

V

V

V

NzBzTz

NyByTy

NxBxTx

object-spacetexture-space

light vector

view vector

Issue: Texture Mirroring

• Texture space is ad-hoc– Mapping defined by artists during model construction– Artists often “re-use” regions of texture space– Includes mirroring regions of the texture

• Example: bilateral symmetry of face or humanoid

• Mathematical consequence of mirroring– Signed area of triangle in texture space is negative– Requires reversal of the sense of gradients– Geometry shader can compensate

Compensating for Texture Mirroing

AtCtAsCs

AtBtAsBs

space in texture area signed

If signed area in texture space “less than” zero, compute basis with negated tangent:

z

y

x

z

y

x

obj

obj

obj

tex

tex

tex

NzBzTz

NyByTy

NxBxTx

2x2 determinant

Example of Mirroring

artist’s original decal derived height fieldRGB normal map

generated from height field

Visualization of3D model’s signed area

in texture spacegreen=front-facingred=back-facing

only half of faceappears in decal

Consequence on Lighting from Accounting for Mirroring

Bad: mirrored skull buckle and beltlighting wrong on left side (indents in)

Correct lighting

Mirroring-tolerantGeometry Shader in Cg

TRIANGLE voidmd2bump_geometry(AttribArray<float4> position : POSITION, AttribArray<float2> texCoord : TEXCOORD0, AttribArray<float3> objPosition : TEXCOORD1, AttribArray<float3> objNormal : TEXCOORD2, AttribArray<float3> objView : TEXCOORD3, AttribArray<float3> objLight : TEXCOORD4){ float3 dXYZdU = objPosition[1] - objPosition[0]; float dSdU = texCoord[1].s - texCoord[0].s; float3 dXYZdV = objPosition[2] - objPosition[0]; float dSdV = texCoord[2].s - texCoord[0].s; float3 tangent = normalize(dSdV * dXYZdU - dSdU * dXYZdV); float area = determinant(float2x2(dSTdV, dSTdU)); float3 orientedTangent = area >= 0 ? tangent : -tangent; for (int i=0; i<3; i++) { float3 normal = objNormal[i], binormal = cross(tangent,normal); float3x3 basis = float3x3(orientedTangent, binormal, normal); float3 surfaceLightVector : TEXCOORD1 = mul(basis, objLight[i]); float3 surfaceViewVector : TEXCOORD2 = mul(basis, objView[i]);

emitVertex(position[i], texCoord[i], surfaceLightVector, surfaceViewVector); }}

additional &changed codefor mirroring

Discarding Triangles Significantly Back Facing w.r.t. the Light

TRIANGLE voidmd2bump_geometry(AttribArray<float4> position : POSITION, AttribArray<float2> texCoord : TEXCOORD0, AttribArray<float3> objPosition : TEXCOORD1, AttribArray<float3> objNormal : TEXCOORD2, AttribArray<float3> objView : TEXCOORD3, AttribArray<float3> objLight : TEXCOORD4){ float3 dXYZdU = objPosition[1] - objPosition[0]; float dSdU = texCoord[1].s - texCoord[0].s; float3 dXYZdV = objPosition[2] - objPosition[0]; float dSdV = texCoord[2].s - texCoord[0].s; float3 tangent = normalize(dSdV * dXYZdU - dSdU * dXYZdV); float maxLightZ, maxLightThreshold = -0.3; for (int i=0; i<3; i++) { float3 normal = objNormal[i], binormal = cross(tangent,normal); float3x3 basis = float3x3(tangent, binormal, normal); float3 surfaceLightVector : TEXCOORD1 = mul(basis, objLight[i]); maxLightZ = i==0 ? normalize(surfaceLightVector).z : max(maxLightZ, normalize(surfaceLightVector).z); float3 surfaceViewVector : TEXCOORD2 = mul(basis, objView[i]);

if (i < 2 || maxLightZ > maxLightThreshold) emitVertex(position[i], texCoord[i], surfaceLightVector, surfaceViewVector); }}

Graphics pipeline dataflowwith geometry shader-based setup

application

Vertex shader

Primitive assembly

Geometry shader

Rasterizer

Fragment shader

Raster operations

framebuffer

•Animation and skinning of object-space vertices•Output texture coordinates•Output object-space light & view vectors•Output clip-space position

•Compute object-to-texture space basis for triangle•Transform object-space vectors to texture space•Output triangle

•Access perturbed normal from normal map texture•Compute lighting in texture space•Output resulting color

Sans Per-vertex Normals

us

vxyz

vs

uxyz

e

s

xyz

(Tx,Ty,Tz)

normaliz

normalize

tangent

ut

vxyz

vt

uxyz

e

t

xyz

BzByBx

normaliz

normalize

),,(tangent2

tangenttangent2),,(normal NzNyNx

Geometry Shader in Cg Sans Normals

TRIANGLE voidmd2bump_geometry_sans_normal( AttribArray<float4> position : POSITION, AttribArray<float2> texCoord : TEXCOORD0, AttribArray<float3> objPosition : TEXCOORD1, // IGNORE per-vertex normal! AttribArray<float3> objView : TEXCOORD3, AttribArray<float3> objLight : TEXCOORD4){ float3 dXYZdU = objPosition[1] - objPosition[0]; float2 dSTdU = texCoord[1] - texCoord[0]; float3 dXYZdV = objPosition[2] - objPosition[0]; float2 dSTdV = texCoord[2] - texCoord[0]; float3 tangent = normalize(dSTdV.s * dXYZdU - dSTdU.s * dXYZdV); float3 tangent2 = normalize(dSTdV.t * dXYZdU - dSTdU.t * dXYZdV); float area = determinant(float2x2(dSTdV, dSTdU)); tangent = area >= 0 ? tangent : -tangent; float3 normal = cross(tangent2,tangent); tangent2 = area >= 0 ? tangent2 : -tangent2; for (int i=0; i<3; i++) { float3x3 basis = float3x3(tangent, tangent2, normal); float3 surfaceLightVector : TEXCOORD1 = mul(basis, objLight[i]); float3 surfaceViewVector : TEXCOORD2 = mul(basis, objView[i]); emitVertex(position[i], texCoord[i], surfaceLightVector, surfaceViewVector); }}

Geometry shader setup with and without per-vertex normals

Setup without per-vertex normals, relyingon normalized gradients only; faceted look

Setup with per-vertex normals;smoother lighting appearance

Visualizing Tangent, Normal, and Bi-normals for Discontinuities

• Using per-vertex normals

Visualize normalsReasonably smooth

Visualize tangentsFlat per-triangle

Visualize tangentsSemi-smooth, semi-flat

Issue: Lighting Discontinuities

• Problem: Geometry shader computes single tangent vector for entire triangle– See tangent visualization (prior slide)– Still each vertex of triangle has its own unique

normal so basis has some variation of triangle

• This per-triangle “tangent flatness” can lead to lighting discontinuities

Lighting Discontinuities

• Example of unsightly discontinuities when applying geometry shader bump mapping setup to a sphere:

Dealing with Lighting Discontinuities

• Final shading when bump mapping can hide these discontinuities somewhat– Imperfect solution

• Perturb the normals within the normal map to compensate for the discontinuities– How?– I have some (unproven, uninvestigated) ideas

More Information and Examples• Get the Cg Toolkit

– Cg 3.1 is available now– http://developer.nvidia.com/cg-toolkit-download

• Or http://http.developer.nvidia.com/Cg/cg_3_1_0010.html– See the “Download” section of this page

• Find source code once installed in– examples/OpenGL/advanced/gs_md2render

gs_md2bumpGeometryshaderbump mapexample

gs_md2shadowGeometry

shaderbump map +

shadow volumesexample

Supplemental Slides

Key frameblended3D model

Decalskin

Bumpskin

Glossskin

GPURendering

Authored 3D Model Inputs

Animation via Key Framesor Vertex Skinning

GPURendering

Frame A

Frame B

Other possible key frames