3d gfx in mpeg-4 tutorial cgi’04 (crete, gr, 2004-06-16) module 2: shapes patrick gioiafrance...

3D GFX in MPEG-4 tutorial

CGI’04 (Crete, GR, 2004-06-16)

Module 2: Shapes

Patrick Gioia France Télécom R&D, FRFrancisco Morán Universidad Politécnica de Madrid, ES+ +Mikaël Bourges-Sévenier Mindego, USMahnjin Han Samsung Advanced Institute of Technology, KRGauthier Lafruit Interuniversity Micro-Electronics Centre, BEAlain Mignot SGDL Systems, CAAlexandru Salomie Vrije Universiteit Brussel, BEMichael Steliaros Superscape, UK

3D GFX in MPEG-4 tutorial @ CGI 2004 Module 2: Shapes (2)

Surfaces (2,5D)• Primitives

Polygons for arbitrary meshes LODs (Levels Of Detail) for progressive 3D mesh coding

Patches for smooth surfaces SSs (Subdivision Surfaces)

Inherently hierarchical 3D surface coding (> progressive 3D mesh coding) “Plain” SSs for mesh smoothing WSSs (Wavelet SSs) for surface approximation

MeshGrid Easily hierarchical 3D surface coding

• Advanced adaptive techniques CGD (Computational Graceful Degradation) View-dependent 3D surface coding (WSSs and MeshGrid)

Backchannel

Solids (3D)







Backchannel

Solids (3D)

3D object modelling paradigms


Approximation of target surface• Method: tesselation with planar facets• Quality: first order (linear) no smoothness (C0 continuity)

Mesh definition: IFS (Indexed Face Set)• Connectivity: list of faces {…, Pn = {in0, in1, in2, …}, …}

arbitrary topology (non-manifold, open, higher genus, etc.)

• Geometry: list of vertices {…, Vk = (xk, yk, zk), …}

Mesh coding• Connectivity (lossless): triangle strips, triangle+vertex trees, etc.

• Geometry (lossy): coordinate quantisation + prediction from conn.

Polygons


Polygons: LODs (Levels Of Detail)

LOD concept• 1976: Clark introduced the idea• Main interest: rendering efficiency

Taxonomy of LOD extraction techniques• Static vs. dynamic• Global vs. local• Progressive vs. hierarchical LODs

Successful simplification techniques• 1996: Hoppe’s edge collapses• 1997: Garland’s quadrics qslim

Progressive 3D mesh coding• 1996: Hoppe’s PM (Progressive Mesh)• 1998: IBM’s PFS (Progressive Forest Split)


2381627957529519Triangles: 69451 vs.

Polygons: mesh simplification example


MPEG-4 (1999): IFSs• Based on VRML97• Arbitrary topology meshes• “Properties” (normals, colours and textures)

MPEG-4 Amd.1 (2000): 3DMC (3D Mesh Coding)• 40-50:1 compression of IFSs by IBM’s TS (Topological Surgery)• Incremental transmission and rendering• Progressive coding by IBM’s PFS• Error resilience by SAIT

Polygons in MPEG-4 (pre-AFX)









Backchannel

Solids (3D)


Approximation of target surface• Method: tesselation with predefined curved patches• Quality: higher order (polynomic/rational) Cn continuity

Mesh definition• Connectivity: regular grid of quads. or triangles

planar topology

• Geometry: list of control points {…, Pk = (xk, yk, zk), …}

Mesh coding• Connectivity (lossless): implicit

• Geometry (lossy): coordinate quantisation + prediction from conn.

Patches


Tensor product of cubic Bézier curves

Patches: example (bicubic Bézier’s)

Compression

(3547 polygons; 1215 vertices)

vs.

(86 patches; 212 control points)

Single patch (4x4 control points)

vs.

two patches (4x7 control points)


MPEG-4 Part 16 (2003): NURBS• Based on VRML97 Amd., originally proposed by blaxxun• Support for NURBS curves and patches

Specific nodes for Bézier’s curves and patches (for increased efficiency)

• Support for free-form deformations

Patches in MPEG-4 (AFX)









Backchannel

Solids (3D)


SSs: definition

SS = limit of recursive refinement of base control mesh

NB: refinement affects bothconnectivity (of abstract graph) and geometry (of 3D mapping)

z

yx

Geometry smoothing achieved with stencils particular to each scheme

1/81/8

½

½

1/161/16

1/161/16

1/16 1/169/16 9/16

Border/sharp vs. interior edge

stencils of “butterfly” scheme

SS inherently define hierarchically nested LODs


Approximation of target surface• Method: tesselation with curved patches• Quality: higher order Cn continuity

Mesh definition• Connectivity: list of triangles/quads., e.g., {…, Tn = {in0, in1, in2}, …}

arbitrary (manifold) topology

• Geometry: list of control points {…, Pk = (xk, yk, zk), …}

Mesh coding• Connectivity (lossless): as for polygonal (manifold) mesh

• Geometry (lossy): as for polygonal (manifold) mesh

SSs (Subdivision Surfaces)


SSs: advantages over polygons and patches

Polygons+ Are the simplest approach (linear approximation)

+ Can resolve fine details and handle arbitrary topologies

– Lead to unstructured, huge meshes

Patches+ Are a more powerful approach (higher order approximation)

+ Are convenient for coarse and smooth models

– Need cumbersome trimming and stitching mechanisms

SSs+ Connect and unify the two extremes above

++ Provide multi-resolution handles for hierarchical coding/editing









Backchannel

Solids (3D)


“Plain” SSs: classic schemes

Dyn++’s “butterfly” (1990): triangular, primal, interpolating, C1

Loop’s (1987): triangular, primal, approximating, C2

Catmull-Clark’s (1978): quadrilateral, primal, approximating, C2


“Plain” SSs: extended Loop scheme (1/3)

Motivation: real-time performance on low spec. terminals• High-speed rendering means triangles due to commodity HW &

optimised SW, but…• designers like working with quads.

Problems with Catmull-Clark’s scheme• Rendering in real-time requires triangles for visibility culling and

HW-based or optimised SW renderers• The best that can be done is triangulation prior to rendering

This defeats the whole “memory efficiency” argument, since both mesh sets (quad.-based and triangulated) must be maintained

More importantly, the triangulation is out of the designer’s control

For consistent real-time rendering on lightweight devices• A guaranteed triangulation is essential• Topology must be maintained in native format



Loop subdivision is more “client friendly”…• Triangle-based input mesh + triangle meshes at all levels, already in

“native format” needed for rendering

… but it is not very “designer friendly” extension of Loop’s scheme through edge visibility tagging

• Still uses Loop’s scheme mostly renderer friendly

• Allows flexibility of quads, but with exact triangulation control• Addresses Loop material boundary problem at extraordinary vertices

Such vertices are common due to the way designers tend to work This problem occurs with colour, texture or any material discontinuity



(b) (c)(a)

(d) (e) (f)

Extended Loop subdivision stencils• Identical to Loop if no invisible edges C2

• Local subdivision matrix is not constant in general but…• recently proven to converge also in the regular quad. case

=

2π2

cos4

1

8

3

8

51

kk,

(a) interior smooth vertex(b) interior dart vertex(c) boundary/crease vertex(d) visible edge(e) invisible edge(f) boundary/crease edge


Piecewise smooth surfaces• Smooth (interior) vs. sharp/boundary edges and vertices• Concave vs. convex corner vertices

NYU’s edge/vertex tagging• Complete set of schemes (C-C, Loop) in a unified implementation• Sound normal control (flat-spot C2)

“Plain” SSs: normal control (1/2)


“Plain” SSs: normal control (2/2)









Backchannel

Solids (3D)


Wavelet-based MRA (Multi-Resolution Analysis)• 1st generation wavelets: classic nD signals• 2nd generation wavelets: Lounsbery’s extension to 2-manifolds

Arbitrary vs. subdivision (semi-regular) connectivity meshes Base mesh extraction and remeshing techniques needed

Applications of MRA• High compression

1993: Shapiro’s “zerotrees” (hierarchical organisation of wavelet coefficients) 1996: Said’s SPIHT (Set Partitioning In Hierarchical Trees) 1997: Kolarov’s coding of scalar functions defined on 2-manifolds 1999: Morán’s and Khodakovsky’s hierarchical coding of surfaces

• Editing/animation 1994: Finkelstein’s multiresolution curves 1997: Zorin’s interactive multiresolution mesh editing

WSSs (Wavelet/detailed SSs)


Price (requirements)• Base mesh extraction• Subdivision scheme = predictor• Details (3D vectors) = prediction errors remeshing

Prize (advantages)• Predictive coding immediate (if smooth target mesh)• Zerotree/SPIHT-based coding not quite...

WSSs: subdivision = prediction mechanism

Target surfaceBase mesh

Subdivide Add details


WSSs: detail hierarchy and quantisation

Local normal frame {n, t1, t2} tied to edge e

• Normal deserves more bits• Tangential components can be treated worse• Normal can be approximated• Tied to coarser LOD for multiresolution editing

TBTA

t1 = en

t2 = n × e

Edge-based detail hierarchy• Based on CalTech’s PGC (Progressive Geometry Compression)


Mesh sizes bit/vertex

• Raw (valid for up to 217 131 k triangles) 144• Basic coding (triangle strips + geometry quantisation) 50• Single resolution coding (Touma’s) 15-20• Progressive coding (Rossignac’s) 22-28

Approximation distances/errors• Point to {vertex, plane, surface}• Surface to surface (Lp)

L1: average L2: RMS ... L: maximum (Hausdorff’s)

• Most common: L2 relative to bounding box diagonal

WSSs: of mesh and mess sizes

P

P

PS

S

dpp

dps

dpv

V


Venus (V=50002; T=100000)

32232017

14

11

8 bitplanes read

0

2

4

6

8

10

12

14

16

0 2 4 6 8 10 12 14

Rate (bit/vertex)

L2 r

eco

nst

ruct

ion

err

or

(x 1

0-4

)

t=390; s=4; T'=99840

23 bitplanes read17 bitplanes read11 bitplanes read8 bitplanes read0 bitplanes read

WSSs: rate vs. distorsion curves

Original mesh


MPEG-4 Part 16 (2003): “plain” + wavelet SSs• “Plain” SSs for mesh smoothing

Considered schemes: Catmull-Clark, [extended] Loop, butterfly No details are added but… normal control achievable through edge/vertex tagging of initial control mesh

• Wavelet/detailed SSs for surface approximation Possibly tagged base mesh Details are added after each subdivision step, which are… wavelet-transformed according to one of several possible schemes Most suited for multi-resolution editing/animation Most suited for view-dependent transmission

SSs in MPEG-4 (AFX)









Backchannel

Solids (3D)


MeshGrid: components

CW (Connectivity-Wireframe)

RG (Reference-Grid)


MeshGrid: hierarchical RG

RG is hierarchical

Object topology defines Arbitrarily shaped Non-uniformly distributed

, ,U WVS S S

RS = 3 sets of surfaces , ,U WVS S S

RG = WU VS SS

Hierarchical animation is possible Applies Dyn’s 4-point interpolating

scheme for curves


MeshGrid: hierarchical CW

, ,VUU

WWV

CW C C CS S S

CW is hierarchical


MeshGrid: CW features

• Regular 4-connectivity

• Triangulation on the fly using connectivity rules: Triangles, Quadrilaterals, Pentagons, Hexagons, Heptagons

• Compact storage using a 3D extension of Freeman chain code


MeshGrid: relationship between CW and RG

1

1 2

, with [0,1)G V

offset offsetG G

��

��

Vertex offset is a relative value:

G1

V

G2

1 1 2G V G G offset��

Update vertex position whengrid is deformed or animated:

G1

G2

V


MeshGrid: examples


MPEG-4 Part 16 (2003): MeshGrid• Combines a surface (CW) with a volumetric (RG) description

Multi-scalable: resolution (CW), global shape (RG), refinement (offsets) Versatile animation possibilities: vertex-level, grid-level, offset-level

• Surface defined as a non-uniform connectivity net with higher density

High curvature areas Locations that are deformed during the animation

• Allows view-dependent partitioning in ROIs (Regions Of Interest)

MeshGrid in MPEG-4 (AFX)









Backchannel

Solids (3D)


CGD (Computational Graceful Degradation)

Hints on rendered scene complexity from server to client• What is “rendered scene complexity”?

Viewpoint: distance perceived size of objects Mesh complexity: number of vertices/triangles Rendering mode & texture complexity — coming soon to this same theater…

• What kind of “hints”? Necessary processing power = f (#covered pixels, #triangles, rendering mode)

• What can the client do about…? Viewpoint: nothing much… Mesh complexity: interpret or request coarser LOD Rendering mode & texture complexity: ignore user’s crave for luxury…


View-dependent WSSs

but adaptive transmission is better!Progressive/hierarchical transmission is good…

Client must be able to• read bitstream parts independently

bitstream must have markers• ask for specific parts OR transmit back

viewing parameters and received dataviewed area

cached area

Use of cache in fly-over app.


00100101110 (discarded)10110111 (new)1001010110011000

View-dependent WSSs

00100101110101101100101000

Coefficients of interest (cached)

Downstream: tree t, bitplane b, 001001…

Server

Bitstreamwith

markers

Backchannel: request or cache info.

Client


View-dependent MeshGrid

Encoding(decompose model in LODs)

Partition each resolution level in one or several ROIs according to complexity

Encode surface locally in each ROI

Downstream: ROI r, bitplane b, 001001…

Server

Bitstreamwith

markers

Backchannel: request or cache info.

Client

View-dependent decoding• Downstream

Receive and decode ROIs

• Backchannel Request ROIs or send cache info.


View-dependent MeshGrid: example


MPEG-4 Amd.1 (2000): CGD

MPEG-4 Part 16 (2003): View-dep. WSSs and MeshGrid• Reduction of transmitted data volume for any 3D modelling paradigm• Backchannel

Use depends on application type Server-driven: client sends status and server decides Client-driven: client sends requests and server replies (if possibe)

Upstream data depends on parameter/command type General parameters (field of view, frame rate): low frequency updates Navigational parameters (position, orientation): high frequency updates Paradigm-specific parameters (WSSs, MeshGrid, moreToCome?)

Advanced adaptive techniques in MPEG-4









Backchannel

Solids (3D)


Solids

• Solid primitives

• Solid models: the “arithmetic of forms”


Solids: solid primitives

Solid primitive

Closed polygonal shape

Algebraic shape

Implicit Quadric

Predefined Quadric

Sphere

Cylinder

Cone

Box

closed Indexed face set


Solids: algebraic shapeImplicit equation:

Quadrics:

Quartics:

Simplest solid primitive:

for


Solids: Implicit nodeImplicit { # % NDT = SFGeometryNode

exposedField SFVec3f bboxSize 2.0 2.0 2.0

exposedField MFFloat c [ ] #10 coeffs for quadrics,

# 35 coeffs for quartics

exposedField SFBool solid TRUE

exposedField SFBool dual FALSE

exposedField MFInt32 densities [] # ( 1, 2)

}

The cylinder and its dual

Animation by interpolation of coefficients values

Various quartics (4th degree equations)


Solids: Quadric node

Quadric { #%NDT=SFGeometryNode

field SFVec3f bboxSize 2.0 2.0 2.0

exposedField SFVec4f P0 -1 0 0 1

exposedField SFVec4f P1 1 0 0 1





exposedField SFBool solid FALSE

exposedField SFBool dual FALSE

exposedField MFInt32 densities[] ( 1, 2)

}

10 geometric control points


Solids: building blocks

Predefined quadrics

Solid modelling by juxtaposition of predefined quadrics

Exact geometry, solid and holes by building blocksCanadian Space Agency Model of SMT (Special Purpose Dexterous Manipulator - Task Verification Facility - Manipulator Test bed )


Solids: arithmetic of forms

F = Q1 * Q2 * Q3F = Q1 * Q2 * Q3

F1

*0 1 2

F0

0 0 0 0

1 0 1 2

2 0 2 4

Multiplication of two forms

A cube = multiplication of 3 degenerated quadrics

Examples of solid operations


Solids: arithmetic operators

)()()(10

PPP ddd r

)(*)()(10

PPP ddd r

Op Description Rules for each P(X1,X2,X3,X4) Syntax

+ Arithmetic addition of the density of two forms

F0+F1

* Arithmetic multiplication of the density of two forms

F0*F1

- The positive difference of two forms F0 and F1.

Returns the difference between the densities if this result is nonnegative, and 0 if the result is negative.

F0-F1

pow Exponentiation of forms

Raises one form to the power of another form. The density of a point with respect to F1 serves as exponent to the density of the same point with respect to F0.

pow( F0, F1)

Gcd Greatest common divisor

Returns the gcd of the densities of two forms. gcd (F0, F1)

lcm Least common multiple

Returns the lcm of the densities of two forms. lcm (F0, F1)

% Integral remainder The Integral remainder operator calculates the integer remainder when the density of a point with respect to F0 is divided by the density of that point with respect to F1

F0 % F1

abs Absolute difference Returns the result of the subtraction between the densities of two forms when this result is nonnegative. Otherwise, the return is the result opposed value.

Abs (F0 - F1)

icurt

Integral cube root Returns the integral cube root of the density of the form. icurt F0

isqrt

Integral square root Returns the integral square root of the density of the form.

isqrt F0

max Maximum This operator is equivalent to the Kleene union for n-ary logic

max(F0 , F1)

min Minimum This operator is equivalent to the Kleene intersection for n-ary logic.

min(F0 ,F1)


Solids: ternary logic operatorsOp

Description Rules for each P(X1,X2,X3,X4) Syntax

| Kleene union of two forms.

F1

| 0 1 2

F0

0 0 1 2

1 1 1 2

2 2 2 2

F0 | F1

& Kleene intersection

F1

& 0 1 2

F0

0 0 0 0

1 0 1 1

2 0 1 2

F0 & F1

>>

Kleene implication

F1

>> 0 1 2

F0

0 2 2 2

1 1 1 2

2 0 1 2

F0 >> F1

<<

Reciprocal Kleene implication

F1

<< 0 1 2

F0

0 2 1 0

1 1 1 1

2 0 1 2

F0 << F1

! Ternary dual of the volume

!

F0

0 2

1 1

2 0

! (F0)

Dual of Reciprocal Kleene implication


Solids: density filters

Filter

Description Rules Syntax

== Equality filter The density of the volume that checks the equality test, and 0 otherwise.

F0 == F1

>= greater than or equal filter

The density of the second volume if the density of the first one is greater than or equal to the density of the second volume, and 0 otherwise.

F0 >= F1

> Greater than filter

The density of the second volume if the density of the first one is greater than the density of the second volume, and 0 otherwise.

F0 > F1

<= Less than or equal filter

The density of the second volume if the density of the first one is less than or equal to the density of the second volume, and 0 otherwise.

F0 <= F1

< Less than filter The density of the second volume if the density of the first one is less than the density of the second volume, and 0 otherwise.

F0 < F1

even

Even filter The volume density if the density is even, and 0 otherwise

even F0

odd Odd filter The result is the volume density if the density is odd, and 0 otherwise.

odd F0

!= Difference filter The result is the second volume density if the densities are different, and 0 otherwise.

F0 != F1


Solids: SolidRep nodeand arithmetic expression

DEF Egg_script Script { # Solid Tree definition

field SFNode Egg USE solid_node

#primitives

field SFNode White # container

field SFNode Yolk # matter inside

# code

url "vrmlscript:

function initialize(){

Egg.solidTree = ( White + Yolk );

}

"

}

Density filtering:

Different densities of the egg cut by a box

SolidRep { #%NDT=SFGeometryNode

exposedField SFVec3f bboxSize 2.0 2.0 2.0

exposedField SF3DNode solidTree NULL

exposedField MFInt32 densityList []

}


Solids: applications

Virtual models

Architecture

Mechanics

Biotechnology


Solids: main benefits• Exact geometry

• Embedded topology

• Compactness

21 Kb 37 Kb 1.1 Mb407 Kb


MPEG-4 Part 16 (2003): Implicit, Quadrics, SolidRep• Algebraic surfaces & solids

Algebraic surfaces by coefficients Quadrics by geometric control points

• Arithmetic of forms Arithmetic, logic (ternary) and filter operators Integration of algebraic solids and polygonal primitives Arithmetic expressions through language Independent of the rendering mode (polygonal, pixel-based, ray-tracing, etc.)

Solids in MPEG-4 (AFX) ???


Conclusions

MPEG-4 has lots of interoperating 3D modelling tools…• Surfaces: polygons, patches, SSs, WSSs, MeshGrid• Solids: building blocks and powerful language

… usable with lots of flexibility…• Progressive (or, even better, hierarchical) multi-resolution content• View-dependent content, for possibly client-driven scenarios

… in lots of potential applications!• Generic 3D objects, from cartoonish to highly-detailed• Virtual character animation• Interactive simulations and high-resolution virtual reality• Accurate CAD/CAM/AEC models and scientific data visualization• Terrain fly-over with adaptive (view-dependent) refinements• You name it!

3d gfx in mpeg-4 tutorial cgi’04 (crete, gr, 2004-06-16) module 2: shapes patrick gioiafrance...

Documents

d mesh codingpatches

d mesh coding1996

d mesh coding40

d surface coding progressive

d mesh codingplain sss

d surface coding wsss

mesh smoothingwsss wavelet

uk3d gfx