- i -

Rich Visualizations From Discrete Primitives

By

BRETT EUGENE WILSON B.S. (University of California, Berkeley) 2000

M.S. (University of California, Davis) 2003

DISSERTATION

Submitted in partial satisfaction of the requirements for the degree of

DOCTOR OF PHILOSOPHY

in

Computer Science

in the

OFFICE OF GRADUATE STUDIES

of the

UNIVERSITY OF CALIFORNIA

DAVIS

Approved:

Committee in Charge

2005

- ii -

Abstract

Visualization is becoming an increasingly important part of science and the arts. Over the past twenty years,

computer graphics algorithms and hardware have matured to the point where any imaginable image can be

created. Now, the challenge is to create clear, visually rich images which faithfully represent the data while

clearly illustrating the details of interest.

Non-photorealistic rendering is one important method for creating rich images because viewers are

more accustomed to abstraction, allowing less important and distracting details to be removed. The pen-

and-ink non-photorealistic rendering style, consisting of discrete black primitives, is one such technique,

but is particularly prone to over-detailing in areas with complex geometry. A two-dimensional rendering

and image processing pass is added after the initial three-dimensional processing of a scene. It incorporates

segmentation and other image-processing methods to identify and abstract away areas that might be con-

fusingly complex. Stippling is a related style often used for abstract images, and the dots that make up the

image must be carefully placed to avoid undesirable patterns. While previous algorithms can be pro-

hibitively slow, a pre-generated data structure is presented which provides high-quality, variable-density

blue noise dot distributions at interactive rates. It supports both two and three dimensional rendering and

provides temporal coherence, a feature lacking in previous high-quality stippling algorithms.

In the field of scientific visualization, the purpose is to identify areas of interest. Therefore, as many

details as possible must be represented accurately, even when the data is far larger than can be handled by a

visualization system. A hybrid point/volumetric representation is presented which optimizes storage for

either point-based or volumetric data sets. The dual-representation increases efficiency by exploiting the

advantages of each: the point-based portion is used to represent small local details, while the volumetric

portion is used to represent continuous, slowly-varying portions of the data. Ultimately, richer non-

photorealistic and scientific visualizations will create more effective methods of exploration and commun-

ication for scientific and artistic advancement.

- iii -

Contents

Abstract······································································································································································· ii Contents ····································································································································································iii 1 Introduction·······························································································································································

1.1 Discretization in art and science ····················································································································· 1.2 Rich visualization ·············································································································································· 1.3 Outline ································································································································································

2 Rich illustrative rendering········································································································································ 2.1 Introduction to illustrative rendering·············································································································

2.1.1 Geometry-based methods····················································································································· 2.1.1.1 Pen-and-ink styles···················································································································· 2.1.1.2 Geometry-based stippling······································································································· 2.1.1.3 Painting and cartoon styles···································································································

2.1.2 Image-based methods·························································································································· 2.1.2.1 Pen-and-ink styles·················································································································· 2.1.2.2 Image-space stippling ············································································································ 2.1.2.3 Watercolor and painting styles ·····························································································

2.1.3 Volumetric methods ···························································································································· 2.1.4 Silhouette and edge processing methods ·························································································· 2.1.5 Hybrid methods····································································································································

2.2 Point-based illustrative rendering················································································································· 2.2.1 Introduction··········································································································································

2.2.1.1 Sample hierarchies ················································································································· 2.2.2 Dynamic Voronoi hierarchies·············································································································

2.2.2.1 Two-dimensional dynamic Voronoi hierarchies································································ 2.2.2.2 Three-dimensional dynamic Voronoi hierarchies ····························································· 2.2.2.3 Sampling fine details ··············································································································

2.2.3 Rendering dynamic Voronoi hierarchies ·························································································· 2.2.3.1 Ambiguous point locations··································································································· 2.2.3.2 Edge position adjustments···································································································· 2.2.3.3 Matching two-dimensional input ························································································

2.2.4 Performance ·········································································································································· 2.2.5 Future work in stippling ······················································································································ 2.2.6 Review of dynamic Voronoi hierarchies ··························································································· 34

2.3 Perception-based illustrative rendering ······································································································· 36 2.3.1 Image segmentation ······························································································································38 2.3.2 The hybrid pipeline······························································································································

2.3.2.1 The three-dimensional processing stage ·············································································40 2.3.2.2 The two-dimensional processing stage················································································ 2.3.2.3 The final rendering stage······································································································· 2.3.2.4 Performance····························································································································

2.3.3 Differences from other hybrid work ··································································································

- iv -

2.3.4 Rendering styles···································································································································· 2.3.4.1 The pen-and-ink rendering style ························································································· 2.3.4.2 Analysis of traditional pen-and-ink illustrations······························································· 2.3.4.3 Requirements for a renderer·································································································

2.3.5 Complex silhouette rendering ············································································································ 2.3.5.1 Generating complexity maps································································································ 48 2.3.5.2 Separating regions·················································································································· 2.3.5.3 Rendering lightly colored complex regions ········································································

2.3.6 Complex region hatching···················································································································· 2.3.6.1 Segmentation ·························································································································· 2.3.6.2 Stroke generation ···················································································································

2.3.7 Examples················································································································································ 2.3.7.1 Tree models····························································································································· 2.3.7.2 Primate lung CT scan ············································································································

2.3.8 Review of hybrid illustrative rendering····························································································· 2.3.8.1 The two-dimensional processing stage················································································ 2.3.8.2 The final rendering stage······································································································· 2.3.8.3 Other future directions··········································································································

3 Hybrid rendering for scientific visualization······································································································· 66 3.1 Introduction····················································································································································· 66

3.1.1 Direct volume rendering ····················································································································· 66 3.1.2 Point-based volume rendering ···········································································································

3.2 Hybrid point/volume visualization for volume rendering········································································· 3.2.1 Hybrid data generation························································································································ 68

3.2.1.1 Point selection ························································································································ 69 3.2.1.2 Optimized representations···································································································· 69 3.2.1.3 Storage ····································································································································· 70

3.2.2 Rendering hybrid data ························································································································· 70 3.2.2.1 Sources of error ······················································································································

3.2.3 Results ···················································································································································· 3.2.3.1 The argon bubble data set ····································································································· 3.2.3.2 The chest data set ··················································································································· 3.2.3.3 The Furby data set ··················································································································

3.2.4 Review of hybrid visualization for volume rendering ····································································· 8 3.3 Hybrid rendering for particle visualization ·································································································

3.3.1 Particle beam visualization ················································································································· 3.3.2 Data representation······························································································································ 84 3.3.3 User interaction ···································································································································· 3.3.4 Rendering ·············································································································································· 3.3.5 Results ···················································································································································· 3.3.6 Review of hybrid visualization for particle rendering·····································································

3.4 Conclusions on hybrid visualization ············································································································ Conclusion ······························································································································································· References································································································································································· 96

1

1 Introduction

Visualization and rendering are becoming more important parts of today’s scientific and artistic landscape

as researchers from all fields increasingly use computer graphics as part of the everyday scientific workflow.

Complex computer simulations and sophisticated measurements allow research efforts to be more accur-

ately directed and experiments to be more precisely analyzed, but produce enormous streams of data. Simple

algorithmic analysis of this data is sufficient for some purposes, such as verifying that certain values are as

expected. However, the scientific process depends on communication with others, and the features of

interest are often not known or are difficult to parameterize. It is these tasks of communication and

exploration in which visualization is critical.

Until recently, “visualization” meant forming a mental image, such as imagining the final print of a

photograph as it is being taken to aid in composition and exposure. Ansel Adams observed, with respect to

one of his early and more famous photographs of Half Dome,

This photograph represents my first conscious visualization; in my mind’s eye I saw (with reasonable completeness) the final image as made with the red filter. … Over the years I became increasingly aware of the importance of visualization. The ability to anticipate — to see in the mind’s eye, so to speak — the final print while viewing the subject makes it possible to apply the numerous controls of the craft in precise ways that contribute to achieving the desired result. [Adams 1983, pp. 4–5]

In the field of computer graphics, visualization has come to refer to the process of producing images from

data to yield insight, that is, a “visualization” in the mind of the viewer of the intangible numbers in the

computer. The result of this visualization provides guidance for what to do next: just as visualization helped

Adams control his camera to achieve the desired results, scientific visualization helps scientists control their

experiments to achieve the desired scientific discoveries.

2

Computers are also increasingly used to generate artistic output, where they are used in two primary

capacities. First, computers are used by artists to generate effects that would be difficult or impossible using

traditional media. One example is photographic manipulation, where photographs are combined and

manipulated in the computer to achieve an impossible or otherwise novel image. The use of photographs

allows a level of realism that can not be duplicated with artistic media, while the computer allows the images

to be manipulated almost as easily as a painting. Another example where a computer can be useful is

animation with an artistic style: the artist carefully defines the look of the subjects of the animation and the

computer adapts this specification to new frames. This frees artists from having to draw every frame

individually, while allowing substantial artistic freedom.

Second, computers are used to generate artistic renderings automatically or semi-automatically. While

traditional rendering has the goal of making images as photorealistic as possible, these “non-photorealistic”

rendering methods attempt to duplicate a specific artistic style and are designed to replace all or part of an

artist’s efforts. They are usually used to more efficiently generate a desired image where the efforts of a

professional artist would be too expensive or time-consuming. Simple examples of automatic non-

photorealistic rendering are certain filters in Photoshop. There are also a wide range of more complex

methods available to artists and engineers that produce higher quality output but require more complex and

time-consuming processing.

1.1 Discretization in art and science

Most types of traditional artistic styles approximate continuous functions (the desired image) with discrete

components. In some styles, this effect is minimal. For example, in watercolor and oil painting, each brush

stroke is discrete, but can be any of a continuous range of colors, and can also interact with underlying layers

of paint to produce blending effects. This effectively eliminates, to the extent the artist desires. the discrete

nature of the media.

3

Other artistic methods, however, do not have the flexibility of continuous color. Pen-and-ink is one

example of such a media. Pens produce sharp, well-defined lines that are easily distinguishable from other

strokes, and, because the ink dries quickly, do not usually blend with previous strokes. Changing inks in

most technical pens is difficult, requiring meticulous cleaning of the pen, so the palette of colors is usually

very limited. This limitation, as well as the nature of the artistic characteristics of the most common styles,

mean the majority of most pen-and-ink drawings use only black ink. And, while some instruments such as

quills can give variable-width lines, the more commonly used technical pens are strictly limited in the line

width or dot size that they produce.

Many types of scientific simulations and measurements produce discrete output. In some cases these

discrete measurements are samples of continuous fields and are often rendered as such through

interpolation. Examples include scientific simulations and three-dimensional scans for medical or

mechanical purposes where values of interest are measured on a grid in space. In contrast, other applications

are not as appropriate for interpolation. One example is particle accelerator simulations. In this application,

the density of particles does have some meaning and can be interpolated, the greatest meaning of the data

involves the movement of the particles as the simulation.

1.2 Rich visualization

When computer graphics was relatively new, visualization was difficult and merely producing useful images

from data was sufficient. Now that computer graphics has matured and become more efficient, scientists and

the public expect images from the data, and producing the richest possible images is the biggest challenge.

Richness in visualization might mean different things in different situations, but at the lowest level, refers to

getting the most out of a single image or animation. For example, it might refer to representing the most

possible detail out of some measurements, achieving high frame rates for interactive purposes, illustrating

areas of interest with the least possible distraction, or even making the most interesting possible image for

4

distribution to a curious public. The goals and approaches necessary for rich rendering vary considerably

based on desired style and application, of which four are addressed by this research.

Stipple drawings approximate continuous functions, usually a reference photograph or rendering,

through the use of small dots. This style has been historically popular for illustrations in printed material

because the two-toned nature reproduces well using only black ink, yet is capable of implying the full range

of grayscale value. It is used on computers today for its artistic value and for its clear differentiation from

other rendering styles, making it useful for multi-modality rendering. However, the placement of these dots

is extremely critical in giving the illusion of a continuous function. Any recognizable patterns or higher level

(low frequency) structure will be interpreted by the viewer as detail in the image rather than a side-effect of

the representation. Accurate depictions of photographs or models, therefore, require that the dots be evenly

spaced, yet devoid of any recognizable patterns. Previous research has produced algorithms capable of

generating extremely detailed stipple renderings with highly optimized distributions. However, such

optimization comes at the cost of performance. Section two presents an algorithm that quickly produces

stipple illustrations with minimal artifacts, allowing interactive uses such as three-dimensional visualization

in addition to the more traditional static black-and-white images.

Hatching is another artistic illustration style that uses discrete black-and-white primitives to

approximate continuous functions. This style uses short pen lines to represent grayscale value and surface

texture, meaning that the style is usually concentrates more on edges than does stipple drawing. The result is

that typical hatched illustrations have greater detail, but with this greater detail comes the danger of over-

detailing. In places where many edges are present, drawing them all can ruin the grayscale value and create

visual confusion, leaving areas of particular interest difficult to differentiate. Much of traditional computer

graphics avoids this problem, due primarily to the fact that most computer-generated scenes are so simple

and edges are not emphasized. As computers get faster, the complexity of the scenes that can be rendered

increases, becoming particularly problematic for hatching and related binary illustration styles. The research

presented in the second part of section two provides a generalized framework for addressing these

5

complexity problems by incorporating a measure of the viewer’s visual response to the image into the

rendering pipeline. This measurement can be used to guide the application of detail or abstraction to

different areas of an image, giving simpler and more clear illustrations.

While the use of discrete points as a rendering primitive may have started as a response to the need for

dual-tone reproduction, they have also been used in three-dimensional visualization. In this case, the goal is

the same: to produce an approximation of a continuous function with a set of discrete primitives. The

reasons for using points in these applications are flexibility and efficiency rather than artistic merit or ease of

reproduction. Points can also be rendered more efficiently without hardware acceleration than more

complex primitives such as polygons or volumes. With today’s hardware acceleration, however, points are

relatively inefficient to render because the hardware is optimized for displaying texture-mapped polygons

for games. A middle ground can be reached by combining the flexibility and efficient storage of point-based

representations with the fast rendering performance of volume rendering. The flexibility that the addition of

points offers a reduction in data size with a less-than-proportional reduction in quality, allowing larger data

to be more efficiently visualized. This addition of discrete point-based rendering for volume rendering

applications is discussed in section three.

Hybrid point and volumetric representations can also be used to more efficiently represent and render

data that was originally particle-based. Pure particle data renders slowly on most PCs, but some data is

inherently point-based, such as particle accelerator simulations in which each point represents a subatomic

particle. The slow rendering times, combined with the large size of each time step, means that interactive

visualization is impractical. Replacing key portions of the point data with volumetric representations is

discussed in the second part of section three, and results in faster rendering times and lower storage

requirements. In this case, continuous volume rendering is used to give the illusion of a discrete point-based

rendering rather than the opposite as is most commonly the case.

In these four application areas of visualization, this research addresses the use of discrete, visible

primitives in generating rich visualizations. In some areas, such as volume rendering, the addition of

6

discrete points enhances a type of rendering that is usually rendered using continuous volumes. In other

areas, the inherent discrete nature of the data or the rendering style is enhanced or supplemented to provide

more or clearer information or better interactivity. In all cases, this research leverages the strengths and

weaknesses of the human visual system to generate visualizations with a better subjective appearance, or

provide better performance without degradation. Improved visualization and rendering allows increasingly

more efficient exploration and more effective communication.

1.3 Outline

This dissertation presents contributions to visualization and rendering in two primary areas: non-photo-

realistic rendering and scientific visualization. In section 2.2, a new data structure called a dynamic Voronoi

hierarchy is introduced for the generation of non-photorealistic stipple illustrations. It allows interactive

computation of temporally coherent dot distributions with few low-frequency components. In section 2.3, a

hybrid pipeline is introduced to aid in the generation of a non-photorealistic renderings of scenes

containing complex geometry. This technique uses an intermediate two-dimensional rendering and image

processing stage that extracts information about how a viewer might perceive the final rendering. The

pipeline is demonstrated using the pen-and-ink rendering style, one that is particularly prone to over-

detailing.

In section 3.2, a method for converting volume data into a hybrid point-and-volume representation for

visualizing large datasets is presented. This technique uses points to represent localized areas of high detail

and a volume to represent those areas of low detail. Because each method is used in the way it is most

efficient, data size is reduced, and large volumes can be viewed with minimal quality degradation. A related

approach is presented in section 3.3, using the same hybrid representation for particle beam simulations.

These simulations are particularly hard to visualize because the original data will not fit on a typical

visualization system. The point-based portion of the hybrid data is used for areas of low density and fine

detail, while the volumetric data replaces more homogenous, dense areas. All of these methods aim to visual

richness and clarity of images for exploration and communication.

7

2 Rich illustrative rendering

2.1 Introduction to illustrative rendering

Hand-drawn non-photorealistic renderings have both artistic and practical importance. Artistic drawings

are appropriate for a variety of uses. Sometimes, all that is needed is an attractive image, and non-photo-

realistic renderings can fill this roll well. For example, Schumann et al. [1996] found that many architects

preferred stylized renderings over traditional “exact” CAD drawings for some purposes, especially for the

initial design phase. The architects felt that the sketch-style renderings used in the study conveyed a sense of

uncertainty, allowing clients to focus on the large-scale design rather than on the details which may not have

been decided yet. Other techniques are particularly well-suited to certain representations. For example,

black-and-white line and stipple illustrations ideal for many printed reproductions.

From a practical standpoint, artistic techniques can also aid in the understanding of a scene or object.

For example, silhouette edges and contour lines can convey shape that is not easily visible in a realistically

shaded object. Gooch et al. [1998] demonstrate the use of silhouette edges and artistically inspired lighting

to generate technical illustrations that are more clear than using traditional lighting and rendering models.

Lum and Ma [2002] use similar methods to improve clarity in the context of interactive volume rendering,.

However, silhouette edges, an important component of many such approaches, break down when there are

many small objects very close together. Silhouette edges and contour lines become so closely spaced that

they are an impediment to clarity rather than an aid. In these cases, high-level scene knowledge can assist in

simplifying complex areas to produce both meaningful and attractive results.

8

There have been two primary approaches for rendering in non-photorealistic styles: geometry-based

methods and image-based methods. Geometry-based methods deal with the primitives directly, and render

them using a specific texture or style. Visibility and the screen size of objects is often used to make

adjustments for level-of-detail or to scale strokes appropriately, but typically no other image-space

parameters are used.

Image-based methods operate only on images, which can be photographs or drawings in addition to

computer-generated renderings. They can emulate a wider range of styles than geometry-based methods

because they can take a higher-level view of the scene, but important details can be lost in the process. Less

commonly used are hybrid methods, such as the one presented in section 2.3, which incorporate informa-

tion from both geometry and images to form the final rendering.

2.1.1 Geometry-based methods

2.1.1.1 Pen-and-ink styles

One of the most influential works on geometry-based non-photorealistic rendering is by Winkenbach and

Salesin [1994], who use pre-generated, resolution-dependent prioritized stroke textures on simple planar

geometry to emulate pen-and-ink drawings. The user also has the option of specifying important portions of

the drawing so that unimportant areas can be left out. They extend the technique in [Winkenbach 1996] for

more complex parametric surfaces. The extensions include controlled-density hatching for constant line

density on surfaces that change area, and planar maps of the scene which identify regions that correspond to

a single visible object or shadow.

Instead of using predefined textures and parametric equations for selecting stroke directions, many

researchers have worked on choosing good stroke directions for arbitrary surfaces. Girshick, et al. [2000]

argue that principal curvature direction is the most useful for portraying the shape of the surface, but the

results lack the artistic look of some other techniques. Rössl and Kobbelt [2000] describe an interactive

system in which the user edits a segmentation of a two-dimensional direction field. Strokes are generated

along this direction field to cover a three-dimensional model. Deussen et al. [1999] compute surface lines by

9

intersecting the objects with sets of planes. By drawing different portions of the image, a style similar to

copper-plate engraving is produced. Ostromoukhov [1999] achieves a similar effect by defining curved

meshes over portions of the model to guide the generation of engraving lines. He also discusses this method

with the addition of some halftoning techniques in [Ostromoukhov 2000]. Hertzmann and Zorin [2000]

grow hatches out from silhouette edges along a direction field. The direction field is created by choosing the

principal curvature direction for areas where it is well-defined, and optimizing the intermediate areas. They

also provide high contrast for undercuts to highlight areas where the surface occludes itself.

While the aforementioned pen-and-ink techniques deal primarily with ease-of-use or image quality,

there has also been research on interactive techniques to emulate pen-and-ink rendering. Praun et al. [2001]

pre-compute a set of textures called tonal art maps for the model using a natural parameterization. For

arbitrary surfaces, lapped textures [Praun 2000] are used for generating a texturing of an object given an

example texture. The technique uses hardware multi-texturing to blend between differing resolutions of the

maps to create temporally coherent, interactive shading. Freudenberg et al. [2001] implement a similar mip-

mapped hatching technique geared for game engines and interactive walkthroughs.

2.1.1.2 Geometry-based stippling

Stippling is an illustration style consisting of dots made with a technical pen. These images usually consist of

one dot size and ink color (almost always black), and the artist creates the desired image by generating the

correct density of points much like dithering algorithms. Geometry-based stippling algorithms attach a set

of points to the object surface. Because the points move with the surface, these methods usually have good

temporal coherence, and the movement of the points provides important visual cues about the third

dimension. Meruvia et al. [2003] generate a hierarchy of points on the surface using mesh subdivision. This

hierarchy encodes which points to draw for a given density, allowing density to be locally adjusted at

rendering time. Lu et al. [2002 (b)] use a similar method but generate samples using a Poisson disk sequence

over the interior of each face of the mesh.

10

Pre-generated hierarchies offer fast, temporally coherent rendering and a selectable level-of-detail, so

they have also been used for a wide range of other rendering styles, including hatching [Winkenbach 1994,

Rössl 2000]. The limitation of geometry-based approaches is that the stipple distribution can not be as even

as in image-space approaches. These methods trade off precise distributions for runtime variability. When a

point is removed, the local density decreases dramatically, resulting in a visible hole in the distribution as in

Figure 1(b).

2.1.1.3 Painting and cartoon styles

Much work has gone into generating temporally coherent animations of a specific rendering style. Meier

[1996] attaches a set of seed points to the scene geometry which are used as the source for brush strokes

along the surface. Strokes are rendered in image-space and Z-buffered, resulting in an abstract painting style

that exhibits frame-to-frame coherence of brush strokes. Kalnins et al. [2002] attempt a more detailed

rendering style consisting of silhouette edges and hatch shading. The user specifies some example strokes at

one or more viewpoints and the computer adapts the strokes for novel orientations so that silhouette,

shadow, and highlight strokes “stick” to the appropriate location on the surface.

There has also been some work on interactive painting styles. A straightforward approach is to use pre-

generated textures that emulate a given style, such as Majumder and Gopi’s [2002] charcoal rendering

method. Lum and Ma [2001] combine different textures to build up light and texture for emulating

Figure 1 If a slightly lower density than in (a) is required, removing one point leaves a noticeable hole (b). Re-optimizing that point’s immediate neighbors, as in the dynamic Voronoi hierarchies discussed in section2.2 allows the change to be spread across a wider region (c).

(a) (b) (c)

11

watercolor rendering in graphics hardware. Textures are not required for all techniques, however. Claes et al.

[2001] present an interactive rendering technique using a simply colored cartoon model in hardware. Special

care is taken to generate sharp and continuous transitions between light and shadow areas.

2.1.2 Image-based methods

2.1.2.1 Pen-and-ink styles

Many have proposed interactive systems to aid in the manual generation of pen-and-ink renderings. While

they often require significant time be spent by a person to generate an image, these methods have the

advantage that they can capture the artistic vision of a person while still being much faster than manual

drawing. Salisbury et al. [1994] allow the user to “paint” with a set of textures from which the computer

generates individual strokes. Salisbury et al. [1997] propose a more efficient, higher-level system in which the

user edits a direction field and specifies a set of example strokes. The computer applies the example strokes

based on the direction field to generate the output strokes.

2.1.2.2 Image-space stippling

Most stippling algorithms work in two-dimensions, just like their artistically generated counterparts. This

process of picking point locations is similar to sampling algorithms for antialiasing, a field with a very long

history in computer graphics. Yellot [1983], in his study of the retinal cells of primates, proposed that the

best form of sampling would have minimal low frequency components and no strong frequency

components that would cause aliasing. In signal processing, blue noise is noise with power proportional to

the corresponding frequency, but in computer graphics the term “blue noise” is often used to refer to any

signal meeting Yellot’s criteria [Mitchell 1987].

McCool and Fiume [1992] use a dart throwing technique [Mitchell 1985] to generate blue noise, in

which sampling positions are generated uniformly (the white noise distribution), but are accepted or

rejected based on a minimal distance criteria from previously selected points. Hierarchical multiresolution

distributions are generated by progressively reducing the minimum distance as the points are selected. The

12

resulting distribution is then refined using Lloyd’s method [1982]. In this algorithm, the Voronoi diagram of

the point distribution is calculated and each point is moved to the center of its Voronoi cell. This process is

repeated until the distribution converges to a desired precision, as measured by the maximum movement of

all points during the previous iteration. Theoretical analysis of Lloyd’s method and centroidal Voronoi

diagrams can be found in [Du 1999].

Hiller et al. [2001] refine this technique to produce sampling tiles. In this work, an initial point

distribution is generated randomly and optimized using Lloyd’s algorithm. More dense resolutions are

created by inserting a new set of points and optimizing them with Lloyd’s algorithm while keeping the

previously existing points fixed. Special consideration is given to making the distributions tile seamlessly,

allowing samplings of varying distributions to be generated by using a variety of different tiles.

Distributions with minimal low-frequency components are ideal for stippling, because points are

spaced at regular intervals without any higher-level pattern. In contrast, white noise, which contains all

frequencies, often contains visible clumps of points and empty space that the eye interprets as detail.

Deussen et al. [2000 (a)] produce blue-noise-based stipple drawings by first distributing a given number of

samples over an image. These points are relaxed using Lloyd’s algorithm to give the desired distribution, but

user input is required to achieve some effects such as sharp edges. Secord [2002] extends this method to

eliminate the need for manual input by weighting the centroids of the Voronoi cells by the grayscale value of

the reference image. Hiller et al. [2003] expand on this technique, allowing more complex primitives than

just points to be distributed.

Because Voronoi-based relaxation works on the plane, these stippling techniques work only from two-

dimensional input. Furthermore, they typically require significant processing time and the points do not

move coherently if animation is desired. Secord et al. [2002] use a different image-based approach based on

two-dimensional distribution functions that provides temporal coherence and quick generation times. The

trade-off is that the resulting distributions are not as evenly distributed as those generated with Voronoi-

based relaxation.

13

All two-dimensional approaches are limited by the amount of three-dimensional information they can

provide. Generally, three-dimensional information is not available (as in the form of a depth buffer, for

example), and those techniques that provide temporal coherence do so by moving points in the screen plane.

The subtle movement of the surface in three dimensions which provides depth cues is lost.

2.1.2.3 Watercolor and painting styles

Watercolor and impressionistic painting styles are popular techniques to emulate using image-based

methods due to their indistinct nature: it does not matter that many of the small details are lost since this is

an accepted part of the technique. In one of the earliest investigations of painting styles, Strassman [1986]

discusses a model for the brush, paint, and paper for manually creating simple paintings in the traditional

Japanese bokkotsu style of sumi-e. More recently, Curtis et al. [1997] present a relatively complete model for

computer-generated watercolor rendering that includes models for the brush, pigments, and paper, and

includes both interactive and automatic components. Durand et al. [2001] describe an interactive technique

for manually generating artistic renderings in a variety of styles. The user draws strokes on the image and

the system interactively generates an output image based on models of artistic materials and the source

image, giving the user a high degree of control without the need to worry about small details.

Hertzmann et al. [2001 (a)] stylize images given two source images: an original image and a stylized

version. This method can then duplicate the style for other original images. It can handle a wide range of

rendering styles and image-processing effects, but requires both an example input and output which are

seldom available. Hertzmann [2001 (b)] also generates painterly renderings from source images using a

relaxation function to optimize brush strokes. Different rendering styles are generated by varying the

weights for the energy function parameters.

2.1.3 Volumetric methods

Non-photorealistic rendering has most often been applied to surfaces and images, but some volumetric

methods have also been developed. Nagy et al. [2002] combine stroke rendering and cartoon shading to

14

augment volume visualization (in this sense, it could also be categorized as hybrid) and to make embedded

surfaces more clear. Likewise, Treavette and Chen [2000] present a system capable of handling three-

dimensional, two-dimensional, and two+-dimensional (multiple two-dimensional renderings) data. An

enhanced version is presented in [Treavett 2001] which describes a hybrid system that also handles volume

data by applying different stylistic filters in modeling space, rendering space, and lastly, image space.

Dong et al. [2001] detect fine structures such as muscle fiber orientations, blood vessels, and cracks in

medical volume data. These details are then enhanced in the volumetric rendering to make the rendering

more clear. Dong et al. [2003] also describe a method for direct stroke-based rendering of volume data.

Strokes along the surface of the volume in question are drawn, along with many interior strokes aligned

according to an extracted direction field. This technique allows more detail to show through than a surface

rendering alone, but because the strokes are not optimized in screen space, the resulting drawing is not as

refined as some image-space hatching techniques. Interrante [1997] generates a volumetric texture from the

source data. A set of points are uniformly distributed throughout the volume, and are turned into a series of

three-dimensional strokes by following the principle curvature direction for a short distance. Isosurfaces are

displayed by rendering the corresponding set of strokes from the volume.

Ebert and Rheingans [2000] take a different approach to volume illustration. Their model uses a

physics-based illumination model along with a variety of feature enhancement techniques, depth cues, and

tone shading to make the details and structure of the volume more clear.

2.1.4 Silhouette and edge processing methods

One important part of many non-photorealistic rendering styles is the extraction and processing of

silhouette edges. Because brute-force extraction of silhouette edges for polygonal models is straightforward

most work has focused on faster or higher-quality results. Raskar and Cohen [1999] present a fast and

simple method for rendering silhouette edges in hardware. This method produces only bitmapped output, so

the lines can not be drawn in any special styles. Buchanan and Sousa’s [2000] edge buffer achieves

interactive rates without hardware acceleration.

15

Extracted edges are often not directly suitable for high-quality renderings because they represent too

much detail of the underlying geometry. For example, silhouette edges extracted from a triangle mesh will

consist of a series of straight lines, and the discrete nature of the data means that there are often small areas

of false silhouette that will also be detected. Freeman et al.’s [1999] method processes lines such as extracted

silhouette edges and produces an artistically styled version based on a set of example strokes. Northrup and

Markosian [2000] process the extracted edges in the two-dimensional projection. The result is a set of long,

smooth lines that fixes many of the small artifacts and discontinuities of lines extracted from triangle

meshes. Hertzmann and Zorin [2000], in addition to surface stroke generation (discussed above), also

address high-quality, topologically correct silhouette and cusp detection for smooth surfaces and meshes.

Claes et al. [2001] supplement their interactive cartoon-style shading algorithm with high-quality, real-

time silhouette edges. These edges are identified by traversing a pre-generated four-dimensional octree.

Edges are represented in the octree as a pair of points, one for each surface adjacent to the edge, with the

coordinates of the points being the four values from the plane equation of the surface. Silhouette edges are

detected when the dot product of these points with a four-dimensional representation of the camera view

vector. Interrante et al. [1995] generate ridge and valley lines to portray important features of the surface, but

focus on minimizing occlusion of surfaces or volumes that may lie behind them.

2.1.5 Hybrid methods

One interpretation of “hybrid” non-photorealistic rendering is the use of more than one rendering style.

Many of the previously discussed interactive drawing systems either implement, or can easily be enhanced

to implement, additional drawing styles, such as [Nagy 2002] and [Treavett 2001]. Also of note is the inter-

active system of Halper et al. [2002] for image generation. It allows the user to specify, using a simple sketch

interface, the rendering styles used and the pipeline in which they are applied.

The hybrid rendering in this work, however, focuses more on hybrid image-generation pipelines, that

is, using multiple processing methods to achieve a given result. The graphics community has long

recognized the need to combine information that typically is available only to different rendering methods

16

or to disjoint steps of a single pipeline. The simplest example is probably the use of the screen size of an

object to control its rendering attributes. For example, small objects are drawn with fewer strokes to

maintain a consistent stroke spacing on the final image. In the case of real-time graphics, small objects can

be drawn with a faster, lower precision technique [Chamberlain 1996].

A common hybrid approach is to use a two-dimensional rendering algorithm, but to supply it with

several input variables in the form of multiple renderings generated from the original three-dimensional

data, sometimes referred to as two+-dimensional processing. Pixar’s RenderMan renderer is able to generate

multiple output variables which Pixar demonstrates can be used in a two-dimensional post-process [PRMan

App. Note № 24] to produce images rendered in different styles, including pen-and-ink and cartoon styles.

This image-generation pipeline, therefore, consists of two distinct phases, a three-dimensional rendering

phase, and a two-dimensional post-processing stage. In Hamel and Strothotte’s rendering technique [1999], a

set of renderings representing different variables, which they call base images, are also used. This hybrid

approach also consists of an initial three-dimensional rendering stage followed by a secondary two-

dimensional stage. However, this system generates pen strokes directly from the input geometry in the first

stage. These strokes are then modified in the second stage according to the base images using a template that

maps base image parameters to certain changes in line style. The system is able to extract templates from

manually generated renderings and apply them to novel models, in effect transferring the rendering style.

Deussen and Strothotte [2000 (b)] use a somewhat different hybrid approach. This work, specifically

targeted at tree and plant rendering, uses two-dimensional depth renderings to extract important features in

the foliage. In the two-dimensional rendering stage, each leaf is rendered as a disk or, for more detailed

drawings, as a leaf-shaped primitive. Strokes are generated where there are large depth differences. By

varying the size of the primitive and the depth threshold required for stroke generation, a range of rendering

styles can be emulated. Level-of-detail is handled automatically if the primitive size and depth threshold is

constant for all objects in the scene, an effect that can be exaggerated by increasing primitive sizes for

smaller objects.

17

2.2 Point-based illustrative rendering

2.2.1 Introduction

Sampling is one of the fundamental concepts in computer graphics. The stipple rendering style is

particularly sensitive to the distribution of samples. This style is often used because of its aesthetic value and

because its dual-tone nature makes it well-suited for many types of printing. Its usefulness also extends into

interactive visualization. The dot pattern is minimalist, mostly transparent, and clearly differentiable from

other rendering styles, making it ideal for multi-modality rendering.

As discussed above, when using stippling to represent a model or a feature in a data set, it is very

important that the points be spaced evenly across the surface. Small low-frequency changes in density are

easily discerned by the human eye, and the perception of the shape of the surface can be lost. An ideal

stipple rendering has even sample spacing and minimal low-frequency components. An ideal rendering

would also have point density which varies smoothly according to the underlying surface and it would be

temporally coherent: not only moving points in a pleasing manner, but moving them so as to enhance the

perception of the surface.

2.2.1.1 Sample hierarchies

One simple application for a variable density sampling approach is merely to produce samplings of the plane.

Such distributions are used to reduce aliasing, for example, in ray tracing programs, as well as for evenly

distributing arbitrary objects such as plants as in [Deussen 1998]. The minimal spacing properties of the blue

noise make it very desirable for many such applications because of the attenuation of low frequencies that

can be visible as changes in density. Unfortunately, the highest-quality method for generating this type of

distribution is Lloyd’s algorithm, which is very slow.

To generate a hierarchy of samples from top to bottom, a set of points is first generated over the plane.

One point is removed, and the local density of points at that location is recorded. Likewise, the hierarchy

can be built from the bottom up by selecting a very-low-density initial distribution and repeatedly inserting

18

points at low-density regions. Hierarchies such as these, like McCool and Fiume [1992] and Hiller et al.

[2001] are static, and, as discussed above, changes in density can be uneven. Hiller’s method offers some

flexibility in this respect. By generating fewer levels of density through adding and optimizing many points

at once, the distribution can be improved, but there will be fewer levels of distribution to choose from. This

trade-off is perfect for certain applications such as antialiasing where the precise sample density is less

important. However, for other applications such as stippling, which must represent a precise density over a

local area, such a trade-off is not worthwhile.

2.2.2 Dynamic Voronoi hierarchies

Dynamic Voronoi hierarchies are a method for quickly generating approximated blue noise sampling

distributions with a variable density. Applied to three-dimensional stippling, the algorithm is an geometry-

based approach in which the distribution problem common to static hierarchical methods is reduced; the

samples are no longer fixed to a single position, but instead move slightly on the surface of the object. This

allows the density change of removing a point to be spread across several neighboring points, making the

change less visible. This difference is illustrated in Figure 1(c) on page 10, where it is extremely difficult to tell

a point was removed.

(a) Step 1: The Delaunay neighbors (open circles) are identified for

the point to be removed (center).

Figure 2 The steps in removing one point. The point removal/optimization process is repeated until allpoints are removed.

(b)Step 2: The center point is

removed and the Delaunay neighbors are re-optimized.

(c) Step 3: The new position and densities of the neighbors are

stored in the hierarchy.

19

Compared to most previous image-space Voronoi stippling, the goal of a global minimum-energy

solution is relaxed in exchange for more local control of stippling density. The result is a slightly less optimal

distribution, but with temporally coherent animation and variable density at interactive rendering rates.

And, using a three-dimensional hierarchy, points move with the surface of the object, giving important

visual hints about the shape of the surface.

2.2.2.1 Two-dimensional dynamic Voronoi hierarchies

Dynamic Voronoi hierarchies reduce the sudden change in point density by distributing it to the

surrounding points. When a point is removed, that point’s immediate neighbors (those points with Voronoi

cells adjacent to that of the removed point) are repositioned using Lloyd’s algorithm, as shown in Figure 2 on

page 18. Because the remaining points in the distribution are kept fixed during this process, each point

typically moves a short distance toward the location of the removed point. The hierarchy can also be built in

reverse, inserting a point, and optimizing it and its neighbors.

The removal of points and the optimization of that point’s neighbors is repeated until the lowest

desired density is reached. As points are moved, their locations are stored, along with the index of the point

whose removal or addition caused the movement. The result is a list associated with each point containing,

in order, the indices of each neighbor that is removed, and the new position the point takes as a result of the

removal. Using these lists, the movement of points can be “played back” at rendering time to achieve smooth

transitions as point density is adjusted. Figure 3 on page 20 illustrates the difference between static (a) and

dynamic hierarchies (b). The dynamic approach reduces the difference between the most dense and the least

dense points. Figure 4 on page 21 shows three arbitrary density levels extracted from the same hierarchy,

along with the corresponding Fourier transforms. The Fourier transforms exhibit the characteristic blue

noise distribution, with few low frequency components (the dark center) and no spikes in frequency. The

inner ring represents the spacing between each point and its closest neighbor, and larger rings represent

successive “layers” of neighbors.

20

Figure 3 A static hierarchy (a) can achieve reasonable results, but has an overall unevenness to it becausepoints are not spaced evenly. A dynamic Voronoi hierarchy (b) can even out the point spacing to give amuch smoother appearance, but still suffers from some artifacts around edges where points are drawn out-of-order (two such cases are circled). Selecting the point position by averaging all desirable positions (c) canremove most of these artifacts.

(a) Static hierarchy

(b) Dynamic hierarchy

(c) Dynamic hierarchy with averaging

21

Figure 4 One hierarchy can represent a range of point densities over a region with blue noise characteristics.These images were extracted from the same 5,000 point hierarchy. The Fourier transforms are shown to theright, with the spacing between the “rings” corresponding to the point spacing in the image.

(a) 50 density

(b) 25 density

(c) 20 density

22

2.2.2.2 Three-dimensional dynamic Voronoi hierarchies

Dynamic two-dimensional samplings are useful for generating static images (see section 2.2.3.3 “Matching

two-dimensional input” on page 29), but rendering point-sampled surfaces in three dimensions is also very

important, especially in the visualization field. The two-dimensional algorithm can be extended to arbitrary

surfaces to give good samplings at any density. This gives the ability to preserve point density regardless of

scale or angle of viewing incidence. It can also be used to give the illusion of lighting or other effects on the

surface.

As noted above, ideal distributions have few low frequencies, but this is measured in the screen plane

rather than on the surface of the object. The screen space sampling is approximated with an geometry-based

sampling, which provides good results in many cases. When the surface of the object has a large angle to the

screen plane, this approximation no longer holds (see section 2.2.3.2 “Edge position adjustments” on page

26).

The first step in building the hierarchy is to generate an initial sampling of the surface. The application

uses triangle meshes for simplicity, but the algorithm generalizes to any other surface representation

including point-based surfaces and implicit isosurfaces in a volume. Meruvia, et al. [2003] use the mesh

vertices as the generators for the initial distribution, whereas I, like Lu [2002 (b)], distribute initial samples

over the interior of the triangles in the mesh. The points are inserted in each triangle with a white noise

distribution and are then globally optimized using Lloyd’s algorithm. This gives a more even global

distribution than Lu’s per-triangle Poisson disk distribution, but requires much more time to generate.

Using this initial distribution, higher- or lower-density representations can be computed by adding or

removing points and optimizing the neighbors of those points. The application always generates the initial

distribution at the highest density and uses decimation to generate lower levels. This requires more work in

the initial optimization stage because there are more points, but simplifies the later handling of details that

are small compared to the current sampling rate (see section 2.2.3.2 “Sampling fine details” on page 26).

23

Figure 5 The teapot mesh with 52,000 total points. The view and scale can be changed interactively atruntime while maintaining an even point distribution. The points move coherently with the surface as the camera is moved, giving important three-dimensional cues.

24

At each iteration, the most dense point is removed from the hierarchy and its last Voronoi cell’s area

stored. To compute the new positions of the point’s neighbors, a planar projection is generated tangent to

the surface at the removed point as in [Alexa 2001]. The points are projected onto this plane so two-

dimensional Voronoi diagrams can be computed. Lloyd’s algorithm is then used to optimize the removed

point’s neighbors, keeping the rest of the points fixed. Because there are so few points being moved and their

movement is constrained, the process typically converges to within acceptable error in less than three

iterations. The points are then projected back onto the surface and their new locations are stored in the

hierarchy. This method of local planar Voronoi projections is also what is used to optimize the initial

distribution using Lloyd’s algorithm,

2.2.2.3 Sampling fine details

Because the Voronoi diagrams are computed over local planar projections of the surface, the point and its

immediate neighbors must be projected onto a plane with little distortion. If there are small details or

corners, or the sampling rate is very low, these projections will not be accurate.

Figure 6 The hip joint data set with 14,000 samples. Only 5,000 are needed to illustrate this image, the restare not drawn because they are on the back, or the density of the region does not require many points.

25

From a sampling perspective, the Voronoi hierarchy represents a multiresolution sampling of the

surface. The ability to locally project samples onto a plane with a given error threshold is related to how well

the surface is sampled. If the underlying surface has higher frequencies than the requested sampling rate can

represent, there are no point positions that can accurately represent that detail.

One approach would be to simplify the object to the maximum representable detail for a given level of

sampling. This would be ideal for scaling the object and corresponds to level-of-detail adjustments that are

very common in computer graphics. However, one major goal is also to use the varying density to correct

for the viewing angle and for lighting. Surface simplification would not be appropriate in these cases, since

the object would change shape if the light changes intensity or a highlight moves.

From an artistic perspective, the goal is to represent some feature using fewer primitives than would

ordinarily be required. In these cases, an artist would be likely to abandon an accurate representation

altogether. Instead, most information would be abstracted away and only key details would be drawn. A

system implementing this approach would not generate very low sampling rates at all, but would switch to a

different approach such as Sousa and Prusinkiewicz’s [Sousa 2003] system for suggestive contours.

The implementation uses a simple rule to address this problem. If a point’s Voronoi cell can not be

accurately represented by projecting it and its neighbors onto a plane, that point is “locked” and no longer

considered for movement. Once points are locked, they behave as in a static hierarchy with no local

optimization steps. This does not produce visibly inferior distributions because of the nature of the surface

in these cases. If a point’s Voronoi cell can not be accurately represented on its own tangent plane, that area

can not be accurately projected onto the screen either, and the measurement of surface area and calculation

of centroids has no meaning.

2.2.3 Rendering dynamic Voronoi hierarchies

Once the hierarchy has been created, rendering is very efficient. First, the desired points are selected. For

stippled surface rendering, this means adjusting for lighting, scale, and angle of view. Points are then

selected based on their “death” area, that is, the surface area of the largest Voronoi cell out of all of the point’s

26

positions. If desired distribution requires larger Voronoi cells (it is less dense) than a given point allows, it is

not drawn.

The surface normal for lighting information is computed once for each point at its first position in the

hierarchy. To properly compute densities based on each possible location of each point would require a

much more complex optimization process, and because points do not usually move very much, one test is

usually sufficient. Because all lighting information is computed at each point, the original mesh is not

required for rendering except for masking the depth buffer for hidden surface removal.

In a second pass, the ideal position of each point is selected according to which of its neighbors are

being drawn. If the point density is set globally, point selection will occur in the same order as the

decimation process and each point has a uniquely optimized position.

2.2.3.1 Ambiguous point locations

When the density of the point distribution is not scaled uniformly, each point’s list of positions may become

inconsistent. In this case, points are selected in a different order than they were removed during the hier-

archy generation process, and the hierarchy encodes more than one desirable position for a point. This

problem occurs most frequently on the edge of sharp highlights. If only one of these positions is chosen,

some artifacts can appear, as shown in Figure 3(b) on page 20.

If no one position in the hierarchy can be chosen for a point, all desired positions are averaged. This is

not guaranteed to produce good results, but works well in practice, as seen in Figure 3(c). Conceptually, the

averaging operation works because each point typically remains in a localized area on the surface. If a point’s

neighbors are “pulling” it in opposing directions, the best place to be relative to those neighbors is the center.

2.2.3.2 Edge position adjustments

The ideal stipple distribution is blue noise in screen space, but the distribution and optimization are

performed in object space. As the surface approaches a perpendicular angle to the view plane, object space is

no longer a reasonable approximation for the screen plane. The error introduced is proportional to the sin of

27

the angle of the surface from the screen plane. This means, that for a typical convex object, the majority of

the surface will have little error. However, some artifacts are visible around the edges, and certain objects can

exhibit this problem more significantly.

These artifacts are a result of the compression of the surface in one direction. This turns the relatively

isotropic blue noise distribution into a highly anisotropic distribution. Conceptually, the Voronoi relaxation

tends to align points in a semi-regular hexagonal grid. When viewed at an angle, these points can line up,

producing visible patterns. This is effect is illustrated with an 80 degree projection in Figure 7 (a) and (b) on

page 28.

To reduce the visibility of this effect, low density attenuation can be traded for decreased anisotropy by

adding a small amount of white noise. Each point is pre-assigned a random noise value between –1 and +1

to ensure temporal coherence. This noise value is scaled proportional to the amount of distortion and the

local point density when displacing each point perpendicular to the direction of distortion. The jitter vector

is J = sinθ · d /2 · r · E × N , where θ = cos E · N is the angle between the surface normal and the screen

plane, d is the average distance to neighboring points in object space, r is the randomized jitter distance for

the point, E is the normalized vector to the eye, and N is the surface normal.

It is important not to add too much white noise or low frequency artifacts will become visible. Figure 7

shows this effect for varying amounts of noise. Without noise, as seen in (b), the horizontal compression

produces horizontal artifacts and a distractingly anisotropic distribution. More noise results in less

anisotropy but increased low frequencies which can become distracting as in (e). Moderate amounts of noise

such as (c) or (d) are probably the best compromise between low frequency attenuation and anisotropy.

28

Figure 7 A uniform planar distribution (a) rotated 80 degrees from the screen plane around the vertical axis (b–e). Artifacts are created because optimization is done on the untransformed plane, but the rotationaround vertical compresses the points horizontally. The horizontal distance between points in (b–e) is about 17 of the original. Adding in random vertical displacements when surfaces approach perpendicularreduces some of the artifacts, trading better anisotropy for less low density attenuation. Relatively low noise values as in (c) produce the most pleasing compromise. The Fourier transforms of each distribution are seen on the bottom row. As the noise is increased, the distribution becomes more isotropic but low-frequencies are more significant.

(a) (e)(b) (c) (d)

29

2.2.3.3 Matching two-dimensional input

Dynamic Voronoi hierarchies can also be used to generate two dimensional output that matches a reference

image. One approach would be to generate a weighted hierarchy for one specific image. The reference image

could be used to compute weighted centroids for the optimization steps and the initial distribution. This

would allow the image to be rendered at any resolution at high quality. However, because of the long

preprocessing time, it is not practical for most applications needing stippled images. More useful is the

ability to quickly match any input image using a single pre-generated hierarchy, even if quality is not as high.

This can be done using an evenly distributed hierarchy computed on the plane, and then selecting the

proper point density out of the hierarchy to match each area of the image. The process is the same as for

generating three-dimensional images: visibility is computed at each point’s initial position based on the

reference image, and the selected points are optimized relative to each other in a second pass.

Figures 8, 9, and 10 were all generated from the same 50,000 point hierarchy computed on a square.

Because a single hierarchy can be re-used for any image, preprocessing time is not significant. The run times

given in these figures are for computation of the point positions and does not include the time to render a

point at that position. These images were all generated by writing points directly to an encapsulated

PostScript file. Generating this file takes about twice as long as selecting the positions of the points.

Figure 8 Gradient image consisting of 4,957 points. Point positions were calculated in 0.06 seconds on a1.33 GHz Athlon. The source hierarchy is an evenly sampled square consisting of 50,000 points.

30

Figure 9 Stippled output consisting of 12,392 points based on a photograph. The source hierarchy is anevenly sampled square consisting of 50,000 points. Around sharp transitions such as the two marked, (a)has a slight blurring effect while with the addition of weighting (b), the points follow the edge more closely.The gap marked with the arrow is in the original photograph and is not an artifact.

(a)Image calculated using dynamic Voronoi hierarchies with “regular” point selection.

Point selection takes 0.07 seconds on a 1.33 GHz Athlon.

(b)Point positions from (a) modified using a weighted average of four nearby positions.

Point selection takes 0.13 seconds on a 1.33 GHz Athlon.

31

Performing only one sample of the reference image per point is very fast, but can miss some small

details that happen never to be underneath a point’s initial position. A further problem is that the optimiza-

tion step can cause points to move some distance away from their initial position. These effects are visible as

boundaries that are smeared slightly and as gaps in small details such as the railing in Figure 9 or the thin

spike at the top of the lighthouse.

There are many possibilities for improvement if additional processing time is available. Because high-

quality but slow stippling algorithms exist for two-dimensional input, this research has focused on

generating the highest possible quality at interactive rates. My approach is to add a third pass after the point

positions are optimized relative to each other. This pass adds back in more dependence on the reference

image by sampling it at several positions around each point and moving the point to the weighed centroid of

its samples. This process can be seen as a very rough approximation of one iteration of weighted Voronoi

relaxation. The result of this optimization is visible on the lower image of Figure 9. The weighted

optimization successfully sharpens the boundaries of high-contrast edges as compared to the un-optimized

Figure 10 Stippled output consisting of 8,334 points from a volume rendering (with silhouette enhance-ment) of a skull. The source hierarchy is an evenly sampled square consisting of 50,000 points. Theintermediate rendering was blurred before stippling because the rendering contains many high frequencydetails not representable by the comparatively low resolution stippling. These details would appear in the stippled result as artifacts. The blurring results in less distinct edges, but a much more coherent overallimage. Point positions were calculated in 0.06 seconds on a 1.33 GHz Athlon from the blurred rendering.

32

image on the left. This image was computed with only four linearly interpolated samples per point.

Additional optimization is possible if more precision is required, such as adding more samples to give a

better approximation to true centroid calculation. At the extreme, the output can be used as the initial

distribution for full weighted Voronoi iteration. Because the initial distribution would already be very good,

fewer iterations should be required to achieve a high-quality result.

2.2.4 Performance

Like all algorithms based on Voronoi relaxation, the hierarchy generation process is very slow, even with the

use of hardware accelerated Voronoi diagram computation [Hoff 1999]: reading the frame buffer and

traversing the resulting bitmap are both time-intensive operations. For an even plane of 5,000 points, one

relaxation step takes 15 seconds on a 1.33 GHz Athlon. Removal and the accompanying optimization takes

about 0.12 seconds for a single point, or about 10 minutes for all 5,000 points. With 15 initial relaxation steps,

the plane takes about 15 minutes total to generate. The teapot example has many more points and requires

extra time for projection to and from the tangent planes for Voronoi optimization. The 52,000 point

example in Figure 5 on page 23 takes about 30 minutes for one full Voronoi relaxation step, and point

removals take about ⅓ second. The full generation time with five initial relaxation steps is about seven hours.

Once the hierarchy has been created, rendering the final image is much more efficient because the

hierarchy contains all the optimization information. The 52,000 point teapot example can be rendered at 10

frames per second on a 1.33 GHz Athlon. This includes the time to compute optimal point density at each

sample, lighting, and for traversing the lists of point movements. Because most computation is done in

software, performance does not depend on the video card to a significant degree.

Due to the significant preprocessing cost, the applications of the dynamic hierarchy to surface

rendering are limited to those cases where a given hierarchy can be re-used or when sample distribution is

extremely important, as in stippling. If the three-dimensional point motion is not necessary, as in the

example of stippling based on a reference image, a single two-dimensional hierarchy can be re-used for all

33

possible input. As a result, rendering is extremely efficient: the example images in Figures 8, 9, and 10 can all

be generated at 10–15 frames per second using a single pre-generated 50,000 point hierarchy.

In an optimized Voronoi distribution, each point has six neighbors, so when a point is removed, its six

neighbors move to fill the hole. The result is that each point averages six different positions in its lifetime. At

each three-dimensional position, the index of the point whose removal (or addition) caused the movement

is also stored. Therefore, with 32-bit numbers, each of the (on average) six positions of each point requires 16

bytes of storage. With the addition of the threshold density at which each point is removed (4 bytes) the

surface normal (12 bytes), and bookkeeping information for the position lists (8 bytes) each point requires a

total of 120 bytes of storage. This is 4.5 times more storage than a static hierarchy (28 bytes to store position,

normal, and threshold density).

Figure 11 The paths that points take as the density is changed. The longer paths belong to the points that aredrawn at very sparse densities. The very short paths belong to the points that will be removed first if the density is decreased.

34

2.2.5 Future work in stippling

The stipple rendering style has promise in visualization mostly when combining it with other styles for

multi-modality rendering. This technique may be useful for displaying static surfaces, such as simulation

boundaries, in a minimal style clearly differentiable from the simulation itself. Because many of these types

of structures are static, one three-dimensional hierarchy can be precomputed and re-used for many

visualizations or frames of a simulation. For applications requiring more flexibility, such as frequently

changing isosurfaces, the three-dimensional hierarchies are not practical due to the high generation time. In

these applications, a two-dimensional stippling style can be used, which gives interactive updating at the

expense of a loss of three-dimensional depth cues as the object is moved.

Animation and soft transitions of point additions and deletions is another area of stippling requiring

further work. Some previous work has used soft transitions as a way to deal with problems of distribution in

a static hierarchy: because points fade in and out, the change in density is less noticeable. This work

concentrates on the harder problem of high-quality binary stippling, which is more true to the original

artistic style, preserves the advantage of stippling when printed, and offers greater flexibility in rendering:

because changes in color and opacity of points are not required to give good distributions, these variables

can be used to illustrate other dimensions of the input data. However, softer transitions are useful if the

extra dimensionality is not required and the output will only be displayed on a computer screen. In this case,

point additions and deletions would be faded in and out, and the neighbors of those points would be moved

gradually to their new positions. This approach is difficult because the position of a point depends on its

neighbors, and these neighbors appear and disappear in an unpredictable order. It is likely that the only

approach would be for the rendering to lag several frames behind the actual distribution so that softer

transitions can be made.

2.2.6 Review of dynamic Voronoi hierarchies

Dynamic Voronoi hierarchies are a flexible method for computing multiresolution samplings of surfaces

with blue noise characteristics. By allowing the samples to move slightly on the surface, the density can be

35

adjusted while maintaining a relatively even distribution. And, because the hierarchy is precomputed,

interactive rendering times are possible.

The precomputation step uses hardware-accelerated Voronoi diagram computation, but it still takes

significant resources in time and computation power. Moreover, because reading back and processing the

frame buffer takes significant time, the hardware accelerated approach may not have provided the best

performance. The use of these discrete intermediate results also introduces numerical instability, making

implementation more difficult, especially in the case of three-dimensional hierarchies that may have

discontinuities. More study is necessary to determine the best implementation approach for Voronoi

diagram computation.

Although useful for any application needing fast blue-noise-like distributions, including sampling for

antialiasing, dynamic Voronoi hierarchies are used here to generate stippled illustrations. This drawing style

consists of distinctly visible dots so it is particularly sensitive to the quality of point distributions. Using a

pre-generated hierarchy computed on a plane, the algorithm can generate stippled output to match a

reference image at much higher qualities than were previously possible at interactive rates. This pre-

computed hierarchy also means that pre-computation time is not a factor in image generation speed.

Dynamic Voronoi hierarchies can also be used for three-dimensional rendering, providing important

depth cues as the points move with the surface. These images are aesthetically attractive, and are also of

practical value because the style is distinct from most other forms of computer-generated rendering, making

it visually separate from other pars of an image. Three-dimensional applications are more limited than the

two-dimensional case because the hierarchies can not be re-used for different surfaces. This means that

some cases where multi-modality rendering would be desired, such as embedded isosurfaces, are not

appropriate for this technique because the surface can not be interactively changed. In some of these cases, a

two-dimensional approach would work, although depth cues would be lost. In other cases, such as

simulation boundaries, the surface is known in advance and the three-dimensional dynamic Voronoi

hierarchy is a good rendering choice.

36

2.3 Perception-based illustrative rendering

The stippling style discussed above is primarily concerned with the optimal placement of dots as defined by

numerical measurements. In contrast, many other techniques are focused more on incorporating higher-

level artistic concepts to produce images that are æsthetically pleasing. They can also provide concrete

benefits over traditional methods: because the representation is not constrained to be realistic, a variety of

enhancements can be added that make the image more informative.

Much research has gone into developing precise and artistically pleasing rendering methods in a

variety of mediums, from pen-and-ink [Winkenbach 1994], to charcoal [Majumder 2002], to watercolor

[Curtis 1997]. Typical applications of these algorithms are simple, easily understandable geometry of single

objects such as sculptures. Many real-world scenes, however, include large amounts of complex overlapping

geometry that can be much more difficult to understand when viewing an image. Although these scenes

consist of increasing numbers of polygons, points, and textures, this research is focused more on perceived

complexity, that is, how hard it is to understand a complex area rather than how hard it is to process the

geometry. Perceived complexity is especially problematic for non-photorealistic techniques because artists

treat complex regions very differently from smooth surfaces with few details.

Human artists keep in mind multiple levels of abstraction when producing artwork. At the highest

level is the overarching artistic æsthetic which we can not hope to duplicate by computer. At the lowest level

of abstraction are the individual objects and geometry that make up the scene, including edges, surfaces, and

texture. In between these extremes exists knowledge about the visual interaction of objects. Using domain

knowledge and intuition about how a viewer will perceive the scene, the artist picks which details to

represent exactly, and which details to represent at a higher level of abstraction. The result is an image where

objects and features of interest are very clearly defined, and additional details are represented at a minimum

level. Figure 12 illustrates two examples of the same scene, one with a more abstract representation, and one

with potentially ambiguous boundaries highlighted.

37

The bulk of computer graphics deals with the lowest level of abstraction: that of the individual

primitives. If multiple objects are considered for a given calculation, it is almost always to compute the

interaction of light between them rather than how they will be perceived when combined in the final image.

While this approach is well-suited for rendering photorealistic scenes because it mimics reality, it is poor for

artistic renderings where much of the information exists only in the realm of suggestion. In these cases, a

higher-level view must be taken to generate high-quality output.

The pen-and-ink style of rendering is particularly prone to overwhelming detail. While it has the

advantage that very small details and surface properties can be very accurately and simply represented, it is

difficult to represent large amounts of detail in small areas. The binary nature of the medium means that if

too many lines are drawn close together, the result will be an incomprehensible black mass. Even if lines are

drawn far enough apart to be well-resolved, many small lines in a small area can be distracting and convey a

limited amount of information.

The goal is to deal with highly complex geometrical models in a way that gives clear, meaningful, and

artistically believable renderings. This is done by integrating a two-dimensional image processing step into

the three-dimensional rendering pipeline, providing more information to make choices about handling

(a) Scene where different

geometry has adjacent and similar looking areas.

(b)Abstract representation where similar overlapping regions are merged into larger meta-objects

in screen space.

(c) More precise representation where potentially ambiguous boundaries are highlighted.

Figure 12 A hand-drawn example of how an artist may incorporate image-space information in a rendering.

38

complex geometry and artistic style.

The hybrid image generation pipeline narrows the gap between the multilevel view taken by human

artists and the more limited local view typically taken by computer algorithms. This pipeline incorporates

both the low-level geometrical information of the scene and view-dependent, high-level information that

reflects how the scene will be perceived by a viewer. The additional information will allow the final

rendering to more closely emulate a human-generated drawing.

2.3.1 Image segmentation

The hybrid rendering pipeline in this work includes image segmentation. Therefore, it is important to

discuss some of the segmentation strategies that can be used, although they are not strictly non-

photorealistic rendering techniques and I do not attempt to improve on existing segmentation techniques.

Possibly the simplest segmentation technique is thresholding, in which one or more threshold values are

chosen and pixels are assigned to regions based on value alone. There have been many proposed

enhancements to make thresholding produce better results [Sahoo 1988], for example, by incorporating

spatial information [Mardia 1988].

Another common class of segmentation algorithms is region growing. In these methods, certain seed

points are selected in each region, and the regions are expanded outward to incorporate material of similar

character [Adams 1994]. Gradient information [Sato 2000] and extracted edges [Pavlidis 1990] can also be

used to help define the boundaries of segments. Watershed algorithms [Meyer 1990, Vincent 1991] treat the

image as a heightfield and simulate rising water that isolates different regions. Enhancements to this

algorithm use edges [Haris 1998] and gradient information [Gauch 1999] to improve the segmentation and

build up a logical hierarchy of regions. There are also a many methods that fall under the category of

“relaxation,” in which a segmentation is iteratively optimized [Rosenfeld 1976, Rosenfeld 1981]. Some are

also interactive, for example, [Hansen 1997]. Other techniques include snakes [Kass 1987], balloons [Cohen

1991], and region competition [Zhu 1996].

39

Normalized-cut image segmentation [Shi 1997] is an advanced image segmentation technique that can

achieve more meaningful results than simple methods like region growing. This method repeatedly divides

the image in two (or occasionally more) portions, maximizing both the difference between identified

regions and their internal self-similarity. As a result, it is not as sensitive to noise, small fluctuations, or

indistinct boundaries as region-growing and thresholding techniques. Since it is a global optimization, it is

also not subject to selections of seed points. Unfortunately, implementation is complex and subject to

numerical instability, and execution time can be prohibitive for some applications, especially for larger

images. The ability to use any metric for measuring the similarity between pixels makes this algorithm easy

to enhance. Malik et al. [2001] use this ability to segment based on the texture of regions by convolving the

image with a variety of kernels to extract texture frequency, direction, and magnitude. This method is able to

handle images with strong but regular textures that would confuse most other algorithms.

2.3.2 The hybrid pipeline

One of the most useful properties of artistic rendering styles is the ability to simplify an image while

maintaining the perception that it is complete. Simpler images are easier to understand and make the details

important to the artist generating the image more clear to the person viewing it. The ability to simplify the

image comes from both domain knowledge of what needs to be represented and what can be omitted, and

the multi-level understanding the artist has of the scene. Certain low-level details may be omitted or

simplified if they are unimportant in a higher-level context. For example, small objects are left out or

simplified, or a silhouette line that might otherwise be drawn can be omitted if the objects it divides already

have sufficient contrast.

A multi-level approach is also necessary for the high-quality implementation of many rendering styles.

A pen-and-ink artist may group objects of similar shading and material properties and treat the group as

one meta-object having a single stroke style. Shadows are often represented as long lines crossing object

boundaries. Painters often build up layers, first ignoring image boundaries and filling in large areas of

similar color. Smaller details are then filled in on top.

40

Traditional computer graphics techniques, which deal with only the low-level primitives, are unable to

simplify based on a higher-level understanding of the scene. A hybrid pipeline, incorporating a large

amount of two-dimensional scene information, can address many of these shortcomings. This work is a

hybrid pipeline for the automatic or semi-automatic generation of non-photorealistic renderings. The

approach attempts to be style-independent: the pipeline should be useful for a wide range of rendering styles.

The pipeline is divided into three distinct stages. First, the three-dimensional processing stage works from

the source geometry to generate a series of renderings. Next, these renderings are processed in the two-

dimensional stage to extract information describing the scene. Last, the final rendering stage generates the

output image based on the information from the previous stages. See Figure 13 for a chart describing this

process.

2.3.2.1 The three-dimensional processing stage

The three-dimensional processing stage reads the source data and generates a series of two-dimensional

renderings for use in the two-dimensional processing stage. This work deals primarily with polygonal input

data, although volumetric data could be used directly, as a source for an isosurface, or in combination with

some other type of data such as a previously extracted isosurface. The type of renderings that must be

generated depend on the needs of the following two stages. Some possibilities are:

□ Grayscale and/or color rendering. This is the standard rendering used to select stroke densities or paint

colors when generating the final output.

□ Depth rendering. Two objects can be adjacent in the image and have identical color, but they may be at

very different depths. The depth information can be used to determine which edges need more or less

emphasis.

□ Edge rendering. The approximate density of visible edges over a given area can give hints about

complexity.

41

□ Normal buffer. Surface normals can give important scene information. Sharp changes in normals may

not always correspond to changes in depth or lighting, but they may be important to give the proper

impression of an object.

The silhouette is also calculated during the three-dimensional processing stage. This is done both to generate

an edge rendering for the two-dimensional processing stage, and for the final rendering stage to use directly,

if needed. Many non-photorealistic rendering techniques rely on high-quality silhouette edges. In these

cases, it is best to use software silhouette edge extraction for the its precise output and line-based output.

Unfortunately, this is a slow operation for large models since all edges must be checked, and high-

performance techniques rely on pre-computation or temporal coherence, neither of which is helpful for a

single computation. If such precision is not required for the final rendering, a much faster hardware-

accelerated technique could be used to generate the edge rendering, such as [Raskar 1999].

2.3.2.2 The two-dimensional processing stage

The goal of the two-dimensional rendering stage is to extract high-level information from the scene as it

would be perceived by the viewer. These image-processing operations are therefore based on the view-

dependent renderings generated in the three-dimensional processing stage.

□ Value-based region growing. One of the simplest segmentation techniques is to pick a point and grow a

region outward until pixels differ from the seed point by more than a pre-determined value. This

method is easy to implement and can be used when speed or ease-of-implementation are important.

Unfortunately, it can be very sensitive to noise and the selection of the seed point.

□ Gradient-based region growing. For some types of renderings, it is desirable to make regions of

contiguous smooth gradient. Gradient-based region growing expands regions until a given gradient

magnitude is reached. This style is appropriate for depth images that consist of large areas of smooth

gradient.

□ Gaussian blur. Gaussian blur can be useful as a pre-processing step before another technique such as

segmentation. For example, by generating a rendering of all edges in a scene (silhouette edges and/or

42

creases) and blurring it, one obtains a map of the approximate edge complexity of various regions.

Segmenting or thresholding the resulting image will result in regions of different complexities. Such

information could be used by the final rendering phase to select rendering strategies.

□ Normalized-cut image segmentation. Normalized cut [Shi 1997] is a more advanced image

segmentation technique that can achieve more meaningful results than simple methods like region

growing.

□ Texture-based segmentation. Some image segmentation techniques can segment images into regions of

similar texture [Malik 2001]. This type of segmentation is particularly useful when used in conjunction

with texture extraction (see below).

□ Other segmentation strategies. The field of image segmentation has a long history and a great variety of

algorithms, many of which may be well-suited for certain tasks in the pipeline. Some of these

techniques are discussed in the previous work section.

□ Texture extraction. Some regions, especially those computed by texture- or edge-complexity-based

Originalgeometry

Intermediaterendering

Imageprocessing

Final NPRrendering

Depth modality

Grayscale modality

... other modalities ...

Figure 13 The rendering pipeline, with an intermediate rendering stage and image processing. These extra steps allow view-dependent image information to be integrated into the final rendering. In addition to the segmentation results, all geometry and the bi-directional mapping between the geometry and the segmen-tation are available to the final renderer.

43

segmentation should be rendered differently according to their texture. Applying a texture extraction

step such as that used by [Malik 2001] for segmenting, could give important information about the

overall visual “feel” of a region. If the texture of the region is strongly in one or two principal directions,

a rendering style should be chosen to reflect that. If there is little texture coherence, a scribbling style

rendering may be more appropriate.

2.3.2.3 The final rendering stage

The final rendering stage is where the data from the two-dimensional and three-dimensional stages is

brought together to form the final image. It implements one or more non-photorealistic rendering styles that

utilize the previously computed three-dimensional and two-dimensional information. More information on

specific rendering styles, their requirements, and implementation is available in section 2.3.4: “Rendering

Styles.” The final rendering stage can also implement any photorealistic technique desired.

2.3.2.4 Performance

The pipeline is not intended to be interactive. Some of the operations, especially the segmenting and stroke

generation operations, are inherently expensive, and, since there are many such steps, interactivity could not

be achieved even if each step was interactive on its own. Rather, the aim is to generate the higher-quality

output, that is, output that is both informative and artistically pleasing.

The three-dimensional processing phase is the fastest and most flexible because it can take advantage

of hardware acceleration, and there are many performance/quality tradeoffs that can be made depending on

the needs of the output. The grayscale, depth buffer, normal, and edge rendering operations, for example,

can all take advantage of modern graphics cards’ rasterization capabilities for interactive or near-interactive

speeds. If higher-quality output is desired for computing visibility or shadows, software methods can be

used at the expense of speed. Unfortunately, one of the slowest operations of modern graphics hardware is

reading the frame buffer from the graphics card into main memory, an operation that is required for each

rendering style.

44

Silhouette edge calculation can take several seconds, especially for large models, since all edges must

be traversed. But exact silhouette edges may not be necessary for a given final rendering style, so a fast,

potentially interactive hardware accelerated method like [Raskar 1999] could be used. If pre-processing is

acceptable, a method such as [Buchanan 2000] can be used to achieve interactive rates.

The two-dimensional processing stage is very difficult to run quickly. Even simple, fast techniques such

as region growing can take many seconds to complete since there are many individual region-growing steps

that must be run to segment a single image. More complex techniques such as relaxation [Hansen 1997] or

normalized cut [Shi 1997] can require many minutes or even hours to complete. One compromise between

quality and speed is to use a very simple segmentation technique on the whole image to obtain a coarse,

first-level segmentation. A more complex algorithm could be used to refine the segmentation. Shi and Malik

showed that the run time of normalized cut is typically about O(n3/2), with n being the number of pixels of

the area being segmented, so using smaller input sets can result in a substantial improvement in speed.

2.3.3 Differences from other hybrid work

In some ways, the hybrid method is similar to several previous works, but there are also important

differences. One key paper is Deussen and Strothotte’s Computer-Generated Pen-and-Ink Illustration of Trees

[2000 (b)], discussed on page 16. The hybrid method is a generalization of Deussen’s technique, which

focuses on techniques specifically for dealing with the challenge of realistic tree rendering. Deussen’s

algorithm must be given special sets of data for the branches and the leaves, which it is hard-coded to treat

differently for realistic results. This limits it to rendering plants in a specific range of styles, which, while

attractive, does not nearly encompass the full range of hand drawn plants.

The goal of the hybrid method is to achieve similar results for a wider range of scenes with little or no

inherent scene knowledge. The addition of image processing, especially segmentation, for multiple

rendering modalities is key, as it helps the program differentiate areas that should be rendered with different

strategies. It incorporates both more three-dimensional information, and a wider range of two-dimensional

45

image processing techniques in an attempt to automatically make decisions that were a result of specific

domain knowledge in Deussen’s work.

Some of these shortcomings were addressed in Hamel and Strothotte’s Capturing and Re-Using

Rendition Styles for Non-Photorealistic Rendering [1999] (discussed on page 16). This technique does use

multiple three-dimensional rendering styles in order to learn more about the scene for the final rendering

stage. However, these two-dimensional base renderings are used only to generate input values to a mapping

function. No image segmentation or other two-dimensional operations are performed to gain a higher-level

understanding of the scene.

2.3.4 Rendering styles

Before attempting to replicate any specific rendering style, we must first determine what makes it unique,

and which types of information will be necessary in order to emulate it as best as possible. The discussion

and implementation in this research focus primarily on pen-and-ink rendering, but the technique itself is

not limited to any specific rendering style, and the potential for watercolor rendering is also analyzed.

2.3.4.1 The pen-and-ink rendering style

The pen-and-ink rendering style has been used extensively for a wide range of applications. The discrete,

black-and-white nature of pen-and-ink illustrations, like the copper-plate and wood engravings before them,

makes them easy for printers to incorporate into black-and-white text. The fine lines allow for a great deal of

detail to be represented precisely, making them ideal for technical illustrations. These limitations are less

significant for today’s increasingly electronic documents. But, the style is still useful for artistic reasons and

because abstraction can be presented in a natural way [Schumann 1996] that is not as distracting as missing

detail from an otherwise realistic image.

46

2.3.4.2 Analysis of traditional pen-and-ink illustrations

Silhouette edges are one of the key features of many pen-and-ink drawings because the medium is

inherently line-based and because the limited dynamic range makes drawing edges one of the most effective

ways to add detail to an illustration.

Very long lines are seldom drawn because they are difficult to control and can result in uneven texture.

As a result, artists usually divide constant or smoothly varying regions into small blocks, each which is filled

in using a series of short strokes. To convey a very smooth effect, these blocks will have aligned stroke

directions and will be arranged in a very regular pattern. Less-regular patterns can be represented using

random block patterns or stroke directions; a variety of weave patterns are also possible. Long lines and lines

that are curved are difficult to control accurately using a pen, and so are seldom used, even when the under-

lying surface is curved. This observation contradicts much of the computer-generated pen-and-ink drawing

literature, which, although generally producing high-quality results, has focused a great deal of effort in

drawing contour lines [Hertzmann 2000, Girshick 2000]. Instead, an artist often chooses one stroke

direction that is appropriate for the general curvature of the surface in question, and the three-dimensional

effect is completed by careful shading or overlapping of different stroke directions. Curved lines are typically

only found on silhouette edges or in scribbles representing very uneven or complex regions.

Finally, pen-and-ink illustrations usually leave out a lot of detail. Since all strokes look similar, complex

regions can quickly become overwhelming if all details are drawn. As a result, most successful illustrations

represent important details using full detail, but and leave the rest to suggestion. Although potentially

confusing, psychological studies [Cavanagh 1989] have shown that the human brain is very good at filling in

details based on simple suggestions such as shadows and outlines.

2.3.4.3 Requirements for a renderer

There is a variety of two-dimensional and three-dimensional data that can help in duplicating the attributes

of a hand-drawn pen-and-ink illustration. Furthermore, the renderer should have at its disposal:

47

□ Silhouette edges: One of the easiest and well-studied areas is silhouette detection, which must be done

in three-dimensional space for accurate results. However, an artistic silhouette rendering leaves plenty

of room for interpretation because many silhouette edges may be omitted if they are not necessary.

□ Curvature: Stroke directions are often based on principle curvature, which should be available on a

per-pixel basis and as dominant directions for different logical regions.

□ Scene rendering: A rendered version of the scene can be used to determine if there is sufficient contrast

in an area where a silhouette edge may not be necessary. The grayscale value would also be used as a

source for stroke density.

□ Edge density map: A blurred rendering of edges indicates areas where there may be an excessive

amount of detail and a simplified representation should be used instead.

□ Segmentation: Segmentation information is very important for creating blocks of similar strokes.

2.3.5 Complex silhouette rendering

Much of the information in pen-and ink illustrations is contributed by the silhouette lines. However, the

portions of an image with very high geometric complexity have large numbers of silhouette lines that can

interfere with each other and make the result meaningless. There are two, sometimes conflicting goals in

silhouette rendering. First, as much texture and important silhouette lines should be preserved as possible.

Second, the approximate grayscale built up by the lines should accurately reflect the darkness or brightness

Figure 14 Renderings of a tree model showing the target grayscale image (a) and the extracted silhouetteedges (b).

(a) (b)

48

of a scene. I use a reference grayscale rendering to determine the density of the scene and to ensure that

grayscale value is consistent. Value must be consistent both within complex regions and in relation to non-

complex regions potentially rendered with other algorithms because it can give important visual clues about

shape which are lost if its use is inconsistent. Figure 14 shows the reference grayscale rendering and all

extracted silhouette edges for the tree model. Despite the simple square leaves, the silhouette rendering is far

too complex to look good. More realistic leaves would be even more confusing.

2.3.5.1 Generating complexity maps

One of the key advantages that the hybrid pipeline offers is the ability to identify complex regions that may

require special treatment. For this purpose, a two-dimensional complexity map is generated that measures

the approximate complexity across the image. Texture maps, hierarchical stroke textures, and procedural

shaders are all sources of complexity in a scene. This work focuses on geometric complexity, which is

measured by rendering all visible silhouette edges and blurring the result with a Gaussian kernel of a user-

specified radius. Complex regions are therefore defined to be those areas where the complexity map value is

above a given threshold, meaning that there are a large number of silhouette edges in the vicinity. Figure 15

shows a complexity map corresponding to Figure 14.

The size of the Gaussian kernel should reflect how close lines can be without becoming ambiguous or

confusingly complex. It depends on the output resolution, the output image size, expected viewing distance,

Figure 15 Complexity map generated from the silhouette edge rendering in Figure 14(b). The leaves and trunk can be differentiated by the value of the complexity map. The radius of the blur can be adjustedaccording to the final image size and expected viewing distance.

49

and artistic aesthetic. Small kernels should be used when a very detailed output image is required, as strokes

can be very close without being merged by the blur operation. Larger kernels are appropriate for images

designed to be viewed at a greater distance, reproduced at low resolution, or when less detailed, more

impressionistic output is desired.

2.3.5.2 Separating regions

Complex areas are separated into regions requiring different treatment during silhouette rendering. These

areas are identified using a combination of the target grayscale rendering and a rendering of the silhouette

edges that has been blurred. The blurred edge rendering represents the approximate grayscale value that the

viewer will perceive if all edges are drawn, and uses the same user-specified kernel as the complexity map.

This blurred silhouette edge rendering is identical to the complexity map in the current implementation, but

this is not generally the case. More general definitions of complexity are also possible that would contribute

to the complexity map, and it is possible that additional candidate strokes should be considered in the

silhouette rendering.

By comparing the target grayscale rendering and the blurred edge rendering, the image is separated

into three regions. First, there are those areas where the edge rendering approximates the correct grayscale

value of the target rendering. These areas need no special handling because drawing all silhouette edges

gives a close approximation of both grayscale value and preserves all details. Second, there are the very dark

areas of the target rendering where, even though the edge density is sufficient to consider the area complex,

it is not enough to build up the desired grayscale value. Last are the regions where the impression given by

rendering all visible silhouette edges is darker than the target rendering. The areas that require darkening are

straightforward to handle because darkness can be added using traditional crosshatching on top of the

silhouette rendering, a technique also used by artists. The most common areas, however, are those requiring

lightening since complex areas will usually have large numbers of silhouette edges.

50

2.3.5.3 Rendering lightly colored complex regions

Representing very complex regions using silhouette edge lines often produces very dark masses of lines, but

lighting and surface properties may dictate that the area should actually be a light color. For these areas,

silhouette lines should be removed so that the correct value is represented, while preserving the

characteristics of the geometry as much as possible. A depth-based method is used to identify important

silhouette lines, and add the goal that the selected lines should be arranged such that the desired grayscale is

approximated.

The importance of a silhouette line segment is measured by the difference in depth on each side of the

segment. Those line segments separating objects very far apart in Z are most likely to represent the edges of

large logical regions such as clumps of leaves or the outermost edges of a group of objects. Manually

generated tags or spatial partitioning could also be used to identify important boundaries.

The depth difference for each candidate silhouette line segment is measured at its midpoint. This

method is fast and simple, and is appropriate for the finely tessellated polygonal models used here. Mea-

suring this difference using the Z-buffer does not work well for small regions with large numbers of edges

since visible surfaces may not be well-resolved in the Z-buffer. If the model has large numbers of very long

line segments in the complex regions, depth differences may vary significantly over the length of the line. To

support these types of models, one could evaluate the depth difference at intervals along each line and break

them into smaller lines of relatively constant depth difference.

All candidate lines in a given light region are checked in order of importance to determine which

should be included in the final rendering. If the addition of a line decreases the root mean square error

between the blurred rendering of all previously selected lines and the target grayscale rendering, it is added

to the selection. Since lines are checked in order, the most important lines that can contribute the necessary

grayscale are always used.

In many cases, it is also desirable to ensure that the most important boundaries are drawn even if they

make an area darker than is indicated in the reference rendering. Therefore, important lines above a given

51

threshold are drawn even if they do not make a good grayscale contribution. This importance threshold is a

tunable parameter so that the user can select between renderings with high grayscale fidelity and low detail,

or renderings with high detail differentiation and low grayscale fidelity.

At first, I experimented with a scaling factor that rewards the grayscale contribution of lines propor-

tionally to their importance. Thus, a somewhat important line could have a small negative impact on

grayscale fidelity and still be accepted, and a very important line could have a larger negative impact on

grayscale fidelity and still be accepted. However, this measure did not produce the desired results because

important lines tend to appear in clumps around the outlines of groups of objects. Once several such lines

are drawn close together, the error becomes so high that no additional lines can be drawn in the area

regardless of importance. When the boundaries of the groups of objects are broken up, much of the

advantage of having the boundary regions are lost.

The final implementation uses a hard importance threshold. All lines above the given threshold are

always drawn, and then lines below the threshold are drawn to achieve the desired grayscale contribution.

Therefore, our silhouette edge rendering method in complex regions reduces to a method very similar to the

leaf-drawing algorithm of [Deussen 2000 (b)] if an all-white target image is used, meaning that all grayscale

contribution is ignored. An example of the important edge detection is shown in Figure 16.

Figure 16 Tree image displaying only the most important lines. Compare this to the full silhouette renderingin Figure 14(b).

52

2.3.6 Complex region hatching

As with silhouette edge rendering, hatching is well-suited to simple surfaces, but does not work well when

there are many overlapping surfaces in small areas. In these cases, image-based techniques are often used to

abstract away the complex geometry to give a simple, understandable, and artistically believable image.

Because of a high degree of complexity and the difficulty of controlling long pen strokes, artists often

draw small stroke groups with similar properties with the aim of approximating local geometric behavior

[Guptill 1997]. An artist also keeps in mind the exact geometry being represented and is careful to respect

the boundaries of important features when generating strokes, an advantage over two-dimensional image-

processing techniques which can abstract away many important but subtle details.

To approximate this balance of abstraction and detail preservation used by artists, a 2+-dimensional

image processing technique [Hamel 1999] is used. This technique combines multiple renderings of the scene

along with image segmentation to extract regions that can be hatched with consistent pen strokes. The

segmentation process extracts the high-level information about how the user will perceive each region. The

use of multiple modalities helps fulfill the often opposing desire to respect important boundaries even if

they are not visible. Rendering modalities currently in use are a grayscale rendering, a depth rendering and a

rendering of the angle of the surface normal projected onto the screen. Any additional rendering modalities

representing surface texture or other properties are also straightforward to incorporate.

2.3.6.1 Segmentation

Segmentation is used to divide the image into areas that can be drawn with a consistent pen stroke. Two

segmentation strategies are used depending on the modality of the input rendering. For the grayscale and

the normal angle images, a simple seeded region growing segmentation method is used. For the depth

rendering, a gradient-based region growing technique is used. This gradient method allows the grouping of

the large, smooth areas of the gradient rendering that often occur with surface going perpendicular to the

view plane. Figures 17 on page 53 and 19(c) on page 56 show the resulting segmentation for the tree models.

53

2.3.6.2 Stroke generation

Once all stroke regions have been generated by intersecting the segmentations, each is separately filled with

pen strokes. As with some artistic styles, no attempt is made to follow the surface exactly, but rather use

small groups of parallel lines to represent approximate surface properties. This approach is better suited to

very complex regions where many surfaces may be present and the exact representation is very complex. For

rendering more simple regions, a traditional surface rendering approach would probably give a better

impression of surface properties.

When generating strokes for each segmented region, the angle of the pen strokes is determined from

on the average angle of all pixels in the region in the normal angle buffer. Cross hatches are then drawn in

the region to build up the correct grayscale value first parallel to the average angle, then perpendicular, then

in each diagonal direction. An example is shown in Figure 20 on page .

2.3.7 Examples

The processing is divided into a three-stage pipeline as in Figure 13 on page 42. The first stage is the three-

dimensional processing stage which works from the source geometry. After the user interactively picks view

parameters, the system generates the set of reference renderings used by the two-dimensional processing

stage and extract the silhouette edges required for edge processing.

The second stage is the two-dimensional processing stage, which works only from the reference images

Figure 17 Segmentation output, combining the segmentation of the grayscale reference image, a depthrendering, and a rendering of the surface normal direction. Different colors are used only to show each region and have no inherent meaning.

54

and silhouette edges generated previously. Here, the reference renderings are segmented and blurred, and

the set of regions that will be used for hatch strokes are generated. These hatch regions are the intersection of

all the regions extracted from the different rendering modalities, including grayscale, depth, and angle-to-

screen. This means that hatches can cross geometry that is very similar, but does not cross important

boundaries. Last, the target grayscale approximation is generated by blurring the grayscale buffer, the

complexity map is generated by blurring a rendering of the silhouette edges, and those silhouette regions

that require lightening or darkening are extracted.

The last stage is where the actual strokes are generated. One stylistic option is a pure silhouette

rendering such as Figure 19(i) on page 56. For very complex regions, these types of renderings can capture

the approximate texture and grayscale value of a sample rendering by drawing a subset of the visible

silhouette edges. For less complex regions, a different rendering style would be required for the same effect.

Another option is a cross-hatched version of the model such as 19(f) on page 56. This example uses the

segmentation-based hatch regions to pick good sizes and positions for groups of cross-hatches. By

combining the two styles, a range of styles can be generated, as seen in Figure 20 on page 57.

55

2.3.7.1 Tree models

Both tree models, the deciduous tree in Figures 14–18, and the conifer tree in Figures 19–20, were modeled

using the xFrog package in Maya. These trees are good examples of geometric complexity: the only regions

that are non-complex are the trunks of the trees. The hybrid pipeline is able to produce good results using

these models. Figure 18 and 20(c) are renderings consisting of important silhouette edges and hatching to fill

in the interior of the tree. The segmentation-based hatch groups allows the texture of the area to be

maintained without representing every object and boundary. Both of these renderings are displayed large so

the hatch boundaries are more visible, but the idea reproduction size is probably smaller: a rendering this

size can communicate more detail than is contained in these images.

Figure 18 The deciduous tree. A small subset of the most of important silhouette edges were drawn (as inFigure 16), combined with cross-hatching based on the segmentation in Figure 17.

56

Figure 19 Example of handling complex regions to produce (f) hatches and (i) silhouette edge renderings.The hatched result is based on the combined segmentation and the blurred target. The silhouette edgerendering displays the most important lines to match the blurred target.

(b)Silhouette edge rendering

(c) Combined segmentation

(a) Grayscale rendering

(d) Blurred target

(e) Complexity map

(f) Hatch only rendering

(g) Regions to lighten

(h)Regions to darken

(i) Adjusted silhouette edges

57

Figure 20 (a) Drawing only very important silhouette lines with no grayscale matching gives a very simple,open result. Compare to Figure 19(i). (b) Adding cross-hatching without segmentation. The result is more correct grayscale, but the interior is abstract. Compare to Figure 19(f). (c) Using different combinations of silhouette rendering and hatching to represent the tree.

(a) (b)

(c)

58

2.3.7.2 Primate lung CT scan

The primate lung data set is an isosurface consisting of about 3.5 million triangles that was extracted from a

256 × 512 × 512 voxel CT scan of a primate. The isosurface value was selected to show the surface of the lung

tissue, and a clipping plane was used to expose the interior. This data is challenging due to the complex

nature of the lung airways. This complexity is already confusing in the simple grayscale rendering in

Figure 21(a) on page 59. When a complex rendering style such as pen-and-ink is introduced as in Figure 22

on page 60, the detail is even more difficult to understand.

The image processing step is able to extract those areas that are complex and those that are not. Notice

that with the geometric definition of complexity, the center of the lungs, while appearing complex, is actually

identified as simple. This is actually a positive result, since this complexity is a result of shading on a bumpy

surface, rather than actual structures as in the inside of the lungs. Because these regions are identified as

separate, using different rendering styles can further differentiate them.

The complexity map is used to identify regions where a simpler rendering style should be used. The

style used in Figure 23 on page 61 for complex regions is a simple blurred thresholding operation that

approximates India ink applied with a brush. This style does not contain the distracting high-frequency

detail of the hatch strokes, so it is less confusing when used in the complex areas. And, because it preserves

the binary nature of the hatching, the overall feel of the fully hatched illustration is preserved.

The process of identifying complex areas through blurring can have negative side effects when the

results of the segmentation are clearly visible to the user. One example is seen along the top of the lower

edge of the lungs in Figure 23. The blurring has caused part of the simple surface on the bottom of the lungs

to be identified as complex, producing a thick black line, when it probably be more desirable to have a sharp

edge at the surface boundary. One way to deal with this general problem is to use the complexity map to

influence the rendering in a more subtle way, making the boundaries between regions less distinct. A better

solution would be to use a method that better respects sharp boundaries in complexity, an goal requiring

further research.

59

Figure 21 Example showing a 3.5 million triangle primate lung isosurface. Notice that the apparently complex center area is identified as simple because it is actually a single smooth surface. It appears theconfusingly similar to the lung detail because of the surface shape and lighting.

(a) Original grayscale rendering

(b) Extracted simple regions (c) Extracted complex regions

60

Figure 22 A non-photorealistic rendering of the primate lung isosurface using only hatching. The line direction emulates an artist, which draws clumps of parallel lines over similar regions, regardless of the underlying geometry, a feature made possible by two-dimensional image segmentation. However, this diagram is overly complex, and much of the detail of the lungs is hard to understand.

61

Figure 23 A Hybrid rendering of the primate lung isosurface using hatching and a solid ink-based style for the very complex regions. This preserves the feel of the previous rendering, but with better clarity in the finely detailed regions.

62

2.3.8 Review of hybrid illustrative rendering

Computer generated images are becoming increasingly complex as they capture more of the tremendous

detail present in everyday scenes. Both traditional and non-photorealistic rendering algorithms must adapt

to handle this complexity. The contribution of the hybrid rendering pipeline is to provide a framework for

making decisions about abstraction and detail in an image.

This general pipeline incorporates two-dimensional image processing techniques and traditional three-

dimensional rendering to leverage the strengths of each. The intermediate rendering stage, combined with

image processing provides additional information to the final renderer about how the viewer will perceive

the scene. This information is combined with the original geometry to generate the final stylized rendering.

Areas that are overly complex are candidates for abstraction, while separate areas that image segmentation

identifies as appearing similar can be made more distinct.

2.3.8.1 The two-dimensional processing stage

The addition of the two-dimensional image processing is key to understanding how an image will be

perceived by the viewer. The fast and simple region growing techniques used in this work produce results

good enough for most applications. To understand why, it is important to remember that the standards for

judging a segmentation for this application is very different from those of traditional image segmentation. In

traditional image segmentation, the goal is often to segment the image exactly how a human would segment

the same image. Problems are caused by slightly ambiguous areas that look similar, but actually belong to

different objects. This is the opposite of the goal in this work, which is to identify those potentially confusing

areas.

However, this work has only touched on the full range of image processing techniques possible in the

two-dimensional processing stage. Histogram analysis and enhanced seed point selection should improve

the existing segmentation techniques. More advanced techniques such as normalized cut can also be used to

improve the segmentation, but at significant cost of speed and ease-of-implementation. More significant

gains should be realized from the use of texture-based segmentation, as this information can be very helpful

63

in selecting rendering styles and strategies for a given region. Related to texture-based segmentation is

texture extraction. Even a very simple texture extraction implementation may be useful in the rendering

stage, since it can help choose stroke directions and styles to achieve a higher level of abstraction.

2.3.8.2 The final rendering stage

The pipeline was incorporated with several different rendering methods. For silhouette edge renderings,

complex areas are identified and important edges in those areas are drawn to match the grayscale value of a

target rendering. For hatching, small blocks of the image with similar properties are extracted, allowing

abstraction of less-visible detail while preserving the most important boundaries.

Combinations of rendering styles are also possible. Hatching and silhouette edge renderings are a

natural fit, and mimics a common method of pen-and-ink illustration: silhouette edges are used for the

most important details, while hatching is used to build up value. A simple solid black rendering method is

also demonstrated that approximates ink applied with a brush. This style, by virtue of its simplicity, is a good

choice for simplifying very complex areas while representing most of the detail and maintaining a non-

photorealistic appearance.

The generalized hybrid 2d/3d pipeline provides a large amount of detail about the scene, and this

information can be difficult to use. This work focuses on some of the challenges associated with the pen-

and-ink rendering style on scenes with high geometric complexity, and much more work can be done in the

field. Incorporating a wider range of pen-and-ink illustration algorithms can improve quality in many cases.

For example, specialized curved-surface illustration algorithms such as [Hertzmann 2000] can be used in

areas with very low geometric complexity. Also, automatic methods such as machine learning and example-

based rendering such as [Hamel 1999] may be good ways to incorporate more information from the two-

dimensional stages in an intuitive way.

Many other rendering styles can be used with the hybrid pipeline. One good candidate for further

study is watercolor, especially if it is combined with another style such as pen-and-ink. Watercolor drawings

are typically characterized by large brush strokes and multiple layers of pigment. Because the brush is large,

64

watercolor illustrations tend to be less precise than pen-and-ink illustrations; artists focus more on building

up tone and value, and portray edges and details by suggestion.

A renderer that implements the watercolor rendering style can be more complex than a pen-and-ink

renderer because it must use complex models for the paper, brush, and pigment [Curtis 1997]. In other

respects, however, the technique is simpler because it is less detailed; precise edges, curvature, and texture

are usually not as important in a watercolor painting as they are in a pen-and-ink illustration. The

watercolor renderer would rely heavily on the standard color rendering and segmentation. Some styles

combine watercolor with another medium, for example, using watercolor to build up color and value, and

pen-and-ink to add important details.

2.3.8.3 Other future directions

One possible direction for future study would be to generate an interactive version of the procedure. As

many of the implemented techniques will not run in real-time on current hardware, simpler, faster

alternatives must be used in their place. Possible strategies include using fewer two-dimensional rendering

types, creative use of graphics hardware, and taking advantage of temporal coherence to re-use segmen-

tation information between frames.

A related area of further study would be the use of the hybrid technique in animations. Currently, the

pipeline does not handle temporal coherence, and the rendering may vary wildly if the viewpoint is moved

even a small amount. This is not acceptable for animations, where the rendering primitives should vary

smoothly between frames. Previous efforts [Meier 1996, Praun 2001] have tackled the problem in three

dimensions, attaching the strokes to the geometry and selecting which portions to draw based on the

current view parameters. As long as stroke selection and order varies smoothly with respect to view

direction and lighting, the resulting animation will also change smoothly with time. Unfortunately, the two-

dimensional processing steps, such as segmentation, can produce drastically different results for similar

frames. It would therefore be necessary to optimize segmentations across several frames, or to use a tracking

65

technique such as optical flow [Beauchemin 1995] to ensure that the segmentations vary smoothly as the

view and lighting change.

For scientific visualization purposes, some support for the direct handling of volume data may be

useful. One way to handle this case is presented in [Dong 2003]. A possible approach that fits better into the

pipeline would be to generate a series of two-dimensional reference renderings directly from the volume.

These reference images would be generated by averaging the parameters such as gradient using the volume

rendering integral just as color is in a standard volume renderer. Working from these two-dimensional

images, appropriate strokes could be generated.

Ultimately, user studies will be required to evaluate specific combinations of abstraction and rendering

style, an inherently subjective property. The ideal combination would provide clear impressions of a scene

with minimal extraneous information in an attractive style. One way around this problem is to add an

additional step to the pipeline that would allow interactive editing of rendering parameters. Currently, the

pipeline is designed primarily for automatic generation of images, but much better results can be achieved

with the addition of domain knowledge and artistic vision that only a human can provide. In this step, the

user would be able to edit and refine the segmentation, select rendering styles and parameters, and assign

them to the objects and regions of the scene. Ideally, this technique would allow results that approach the

quality of hand-drawn image while requiring only a fraction of the effort from a person.

66

3 Hybrid rendering for scientific visualization

3.1 Introduction

Visual richness in thee context of scientific visualization is often measured by interactivity and the ability to

clearly represent all details. In contrast to the previously discussed non-photorealistic rendering techniques,

the important areas of interest in scientific data are often not known. Therefore, it is important to preserve

all possible details and to provide a way that the data can be interactively explored and analyzed. Yet, the

ability to render high detail and to achieve interactive rendering speeds are often mutually exclusive. This

research uses a combination of volume- and point-based storage and rendering to achieve higher-quality

interaction without sacrificing significant quality.

3.1.1 Direct volume rendering

Direct volume rendering is a common technique for visualizing volume data. It is straightforward and

preserves all the features of the original data, including structures that can not be analytically defined. Most

volume rendering techniques use an absorption-plus-emission or scattering-plus-shading model for

rendering [Max 1995] and can be applied to a wide range of cell formats, including rectilinear grid,

tetrahedral cell [Shirley 1990, Williams 1998, Röttger 2000], irregular grid [Wilhelms 1996, Rüdiger 1997],

and AMR (adaptive mesh refinement) [Weber 2001] data.

Hardware-accelerated volume rendering is usually implemented using either two-dimensional [Cabral

1994] or three-dimensional [Wilson 1994] texture-mapping hardware and paletted textures for interactive

transfer function manipulation [Meissner 1999]. Dedicated volume-rendering hardware [Pfister 1999] has

67

also been used. However, even on the most advanced systems, the size of many data sets can far exceed

available video memory. There are several approaches to rendering large volumetric data that does not fit

within the video card’s memory. The simplest is texture paging, which is often implemented in the display

driver and is used automatically when memory is full. Unfortunately, the latency required for transferring

data over the bus to video memory inhibits interactive display rates. Past efforts have also used parallel

rendering techniques to increase the amount of available texture memory [Lum 2002, Kniss 2001]. While

this approach is feasible, the required amount of hardware is often costly, efficient implementation can be

complex, and network latency and compositing can become bottlenecks. Finally, multi-resolution

techniques [Weiler 2000, LaMar 1999] attempt to make better use of texture memory by using lower

sampling rates for areas of low interest. The disadvantages of this technique are the introduction of artifacts

and loss of detail.

3.1.2 Point-based volume rendering

The use of points as a display primitive was first introduced in [Levoy 1985] and was more recently presented

in [Grossman 1998]. Related to this work are the point-based rendering methods used to handle the

interactive display of large and complex polygonal data sets [Rusinkiewicz 2000, Wand 2001]. In the context

of volume rendering, splatting [Westover 1989] has become an increasingly popular approach. There has

been the development of faster [Laur 1991] and higher-quality [Swan 1997] splatting techniques, the use of

splatting for rendering irregular-grid data [Mao 1996, Meredith 2001], and a more universal approach in

reconstructing surface and volume data [Zwicker 2001].

3.2 Hybrid point/volume visualization for volume rendering

Volume rendering and point-based rendering each have their own unique strengths and weaknesses.

Volume rendering provides fast rendering times on commodity graphics cards and even coverage of slowly

varying regions, but is (usually) restricted to a uniform resolution. Point-based rendering provides more

flexibility about where detail is placed, but can be less efficient to render. A hybrid storage and rendering

68

approach using both points and volumes can help overcome the limitations of each. This approach converts

a large, high-resolution volume into a low-resolution version, using this volumetric representation for the

majority of the volume. Points are then added to make up the detail lost in the conversion. Since most data

sets have few sharp boundaries compared to slowly varying regions, many fewer points are necessary than

would be required for a full point-based representation.

Using hardware-accelerated texture-based volume rendering for the majority of the volume means that

rendering times are kept fast. Using points around high-frequency boundaries ensures that those areas have

sharp transitions that are faithful to the original data. The result is a much smaller data set that preserves

most of the interesting details in the original. The goal is not to replace high-accuracy volume rendering.

Rather, this approach allows more efficiently storage, transfer, and interactive previewing of reasonably

accurate data using a single low-cost PC and graphics card.

3.2.1 Hybrid data generation

Converting a high-resolution volume data set to a hybrid representation involves generating a low-

resolution volume and a corresponding set of points. The low-resolution volume data represents the larger,

more uniform areas of the original, and the points are inserted where the low-resolution data skips or de-

emphasizes important details. Thus, storage and processing techniques are allocated according to the detail

requirements of a given region of data. Because this technique is most useful for data that is larger than that

easily handled by a single computer, processing the full-resolution version can be slow. The preprocessing

program requires on the order of tens of minutes for typical data sets on a desktop PC. Large data can also

be preprocessed in pieces, either in parallel on multiple computers, or in serial on a single computer. The

pieces are then merged to produce the final hybrid representation.

Any standard subsampling technique can be used to generate the low-resolution volume data; the best

choice depends on the nature of the data and the application it is used for. Most images in this paper were

generated using tri-cubic interpolation, although other subsampling techniques are more appropriate in

certain cases (see section 3.2.1.2 “Optimized representations” on page 69).

69

3.2.1.1 Point selection

Points are generated at those locations where the volumetric data contains large amounts of error. These

locations are identified by sampling the difference between the low-resolution representation and the

original data at regular intervals. Typically, the sampling interval of the error grid is the same as the original

resolution of the data, meaning the low-resolution data is compared to each of the original data values.

Lower-resolution error grids can also be used. Linear interpolation is used on the low-resolution data to

simulate the value the video card will compute at each error grid location. Point insertion is done based on a

threshold error value: if the error value is above a user-defined level, a point is inserted at that location. If a

maximum output size is given, a predefined number of points can be inserted at the locations with the

highest error magnitude.

3.2.1.2 Optimized representations

One source of error results from the limitation that video cards can only add color on the screen. That is,

points can only contribute to the opacity of a given area, and never make it more transparent. On average, a

low-resolution representation will overestimate the original function as much as it underestimates, so the

addition of points can only make up 50 of the error in the best case.

If something is known at the time of preprocessing about the type of transfer function the user will

select, the error can be improved by selecting the low-resolution representation such that it never produces

positive error for the expected transfer function. One common type of transfer function is one that is never

decreasing; the user often wishes to see areas of higher value as always being more opaque. In this instance,

the low-resolution representation is selected so that it never overestimates the original function value (the

interpolation function is the minimum function). In practice, we found this method produces poor results.

It generates large areas where the original function is extremely underestimated, so it requires an impractical

number of points to produce a good image. As a compromise, the preprocessing program can take a

parameter that specifies the minimum amount the low-resolution representation can overestimate the

original, allowing the user to decide the extent to which this type of accuracy balances the number of points.

70

3.2.1.3 Storage

The low-resolution volumetric data and point data are calculated in a preprocessing step and stored for later

viewing. The sampling interval and error threshold are provided as parameters to the preprocessing

program. For points, the position, error value, and the value and normal of the original data at that location

are stored. The original function value is used to map the point to the corresponding color and opacity when

it is displayed. The error value indicates how much the opacity should be scaled so that, when combined

with the volumetric data, the correct opacity is presented on the screen. The preprocessor also computes

normals for each value of the volumetric data. All normals are stored as one-byte indices into a lookup table

containing normal vectors uniformly distributed in three-space.

If a 2563 grid is used for error calculation, each point can be stored with 8-bit coordinates, giving a

storage space of six bytes (one byte each for x, y, z, normal, error, and original function value). If a higher-

resolution grid is used, each point requires 16-bit coordinates, giving nine bytes of storage per point. An

optimized hierarchical representation such as that presented in [Rusinkiewicz 2000] could reduce the

storage required for coordinates to about 16 bits for each point, giving five bytes storage per point. We did

not implement this method because the data size is dominated by volume data and the corresponding

normals, and such optimization would only reduce the total data size by 5–10 for the examples presented

here.

A 5123 volume with one-byte normals can therefore be converted into a 2563 volume with enough room

for 26 million points in the same amount of file space (46 million using the five-byte hierarchical coordinate

representation). In practice, many fewer points are required to achieve good results, and even lower

resolution volumes are acceptable for some purposes. Therefore, the hybrid approach can give a significant

savings in disk and memory requirements over the non-hybrid method.

3.2.2 Rendering hybrid data

The renderer for hybrid data was designed for older video cards. Therefore, three-dimensional textures are

not used and the volumetric portion of the image is rendered using parallel, axis-aligned, texture-mapped

71

planes. Each texture is an indexed-color slice of the data, allowing fast updating of the transfer function by

manipulation of the palette. To prevent the volume from disappearing when the slices are viewed edge-on,

three sets of slices are maintained, one for each axis, and the set most parallel to the view plane is used. Since

each slice of the data is usually partially transparent, the video card’s Z-buffer will not correctly handle

ordering of objects. Therefore, special care must be taken to draw items back-to-front.

Each slice also has a corresponding normal texture map. This map contains the one-byte normal

indices computed during preprocessing, stored as an indexed texture. A palette maps these indices to the

specular (alpha channel) and diffuse (red, green, and blue channels) lighting values for the corresponding

normal vector. Multi-texturing and hardware register combiners are use to multiply each data value with its

diffuse lighting value, and then add the specular light value.

Slices of points are drawn interleaved with each pair of volumetric planes so ordering is maintained.

These points can be rendered using either an OpenGL point primitive, which is rendered as a square, or a

Gaussian splat. OpenGL points are sized such that adjacent points overlap slightly, a fixed value dependent

on the resolution of the grid used to compute error (Gaussian splats must overlap more since their opacity

falls off toward the edges). If adjacent points do not meet, distracting patterns will appear, and if they

overlap too much, the covered area will be more opaque than was intended during preprocessing. Since

OpenGL points are drawn with integer pixel sizes only, problems can arise when the correct point size is a

small non-integer. In these cases, the round-off error is a large fraction of the correct value, resulting in

points that are much too big or too small. To combat this problem, point sizes are always rounded up when

passed to OpenGL, and their opacity is adjusted downward proportional to the increase in area of the point.

For greater rendering efficiency when drawing OpenGL points, each slice of points between adjacent

pairs of planes are loaded into a display list. This triples the amount of memory required for point storage on

the video card, since each point is loaded into one display list for each axis, but yields greater rendering

efficiency: even if the video driver must page the display list data out of video memory, it is still faster than

loading the point data from scratch. Gaussian splats are rendered using a Gaussian texture mapped onto

72

forward-facing squares (billboards). Since rendering a forward-facing primitive is a view-dependent

operation, Gaussian splats can not be loaded into display lists, increasing rendering time.

No care is taken to render points within a slice back-to-front, and this can introduce artifacts [Levoy

1985]. Although a hardware-based solution is given in [Rusinkiewicz 2000], it increases rendering time, and

the goal of the hybrid method is to display a reasonably accurate representation of large volumes as fast as

possible. If a more accurate representation is desired, a more effective use of display resources would be to

increase volume resolution or point counts. Furthermore, artifacts are limited for two reasons. First, whereas

the previous work uses points to display primarily opaque surfaces, the points in this research are mostly

transparent and ordering is less of a factor in the combined result. Second, since the error sampling grid is

only a small factor (usually two to eight) greater than the volumetric slice resolution (the scale at which

ordering is correct), the number of points with incorrect ordering is limited to the same small factor.

The video memory requirements of the texture data could be reduced by using the volumetric texture

support of recent video cards, which would eliminate the data duplication caused by maintaining three sets

of planes for the volume. Unfortunately, most video cards do not support texture paging for portions of a

volumetric texture, preventing the use of textures larger than video memory. This means a maximum

volumetric texture size of 2563 on the nVidia GeForce4 used for this project. Nevertheless, volumetric

textures are still desirable because the parallel planes drawn on the screen do not have to be aligned on the

texture’s axis. The planes can be always be drawn parallel to the view plane while the texture coordinates are

transformed, and the video card will interpolate the texture in three dimensions, reducing rendering

artifacts [Wilson 1994]. However, this method greatly complicates correct ordering when combined with

point rendering, since arbitrary slices of points must be selected to interleave with each texture-mapped

plane. It also prevents the use of display lists for point slices, since the shape of each slice of points is view-

dependent. As a result, we chose to stay with axis-aligned rendering planes for better speed, ease of

implementation, and to be able to generate large (5123) hardware-accelerated reference renderings.

73

3.2.2.1 Sources of error

Although the hybrid method can make up considerable amounts of error present in lower-resolution

volume representations, it is not without its own sources of error. First, as was discussed above, points are

only additive and can not compensate for overestimation errors. Second, if the transfer function’s color map

is rapidly changing, or error is great, the volumetric data will map to a significantly different color than the

correct value represented by a point at that location. The rendering algorithm described above only gives

correct opacity for the combination of a point and volume, the color may be incorrect (more precisely, it will

be the average of the correct color and the color of the volume data).

Last, since the hybrid method is essentially a multi-resolution rendering method, it is subject to the

same sources of error as other multi-resolution methods. In areas displayed at higher resolution through the

use of points, the transfer function must be adjusted downward to give consistent opacity over the entire

image. In addition, since the points tend to exist on edges and corners, the amount of adjustment required is

highly view-dependent. We chose not to address the view-dependent multi-resolution problem in this

current work, and found that global point transparency can be manually adjusted to achieve acceptable

results in many cases.

3.2.3 Results

Hybrid renderings generated with different parameters were compared with standard volumetric

representations of the same data. All tests and timing measurements were done on a 1.0 GHz Pentium 3

Xeon running Linux with 1gb of system memory and an nVidia GeForce4 Ti 4600 graphics card with 128mb

of video memory.

3.2.3.1 The argon bubble data set

The “argon bubble” data set is one frame of a 640 × 256 × 256 time-varying simulation of an argon bubble. It

was padded and converted to 5123 using tri-cubic interpolation. This step is required for an accurate

74

reference, since most video cards require texture sizes to be powers of two. The hybrid representations and

lower-resolution volume representations were all generated from the 5123 version of the data.

The argon bubble contains some very fine detail in the turbulence, as well as many localized areas of

very high density surrounded by areas of low density. As can be seen in Figure 24, a 2563 volume rendering

misses some details, produces incorrect density values for some small features (for example, the small

protrusion at the top-center) and exhibits severe pixilation artifacts. Using this low-resolution volume, a

hybrid representation was generated using an error calculation grid of 5123 and an error threshold of ⁄,

producing 361k points. The size of the hybrid representation is a mere 37mb, compared to 33mb for the low-

resolution volume rendering alone, and 268mb for the original.

75

(c)

(d) (e)

(b) (a)

Figure 24 Close-up images comparing renderings of the argon bubble (c). The top row shows standard volume renderings of the original 5123 volume (a) and the lower-resolution 2563 volume (b). The bottom row shows the same 2563 volume with 361k points, rendered with square OpenGL points (d) and with Gaussiansplats (e). The middle image is the entire view at 5123 with the zoom area marked.

76

The hybrid renderings, even with this relatively low point count is able to improve the low-resolution

rendering in several respects. Pixilation artifacts in key areas are decreased, and shapes are more obvious. It

also corrects color in some areas, such as the small yellow feature in the top center, which appears as red in

the low-resolution version. Notice that an artifact of point rendering, particularly visible at this extreme

magnification, is that small details tend to get larger (this hybrid volume was not produced using the

optimized representation given above to reduce the effect).

The hybrid rendering drawn with OpenGL points in the lower-left of Figure 24 displays interactively

(10 fps), but their square shape is clearly visible when zoomed in. The Gaussian splats in the lower-right

image look much smoother, but take several seconds to render. One approach would be to use OpenGL

points for normal interaction, and use Gaussian splats when both the image is static and the zoom level is

high. Recently, a point sprite extension was introduced in OpenGL which will probably allow better

interaction with Gaussian splats.

3.2.3.2 The chest data set

The chest data set is a 5123 MRI scan of a male chest. At full resolution, the fine airway structure of the lungs

is clearly resolved. However, at lower resolutions, almost all of this detail is lost. The hybrid approach is able

to restore much of this detail. However, because of the large area covered by fine detail, a great number of

points are required to make up the difference. Therefore, a loose error threshold for point insertions is

required to achieve acceptable performance and storage requirements.

77

Figure 25 Comparison of rendering techniques of the chest MRI dataset. In all pictures, the front and back havebeen clipped by 20 to allow the detail of the lungs to be visible. Greater numbers of points in (d) and (f) fill out the fine structures of the lungs more fully, and provide more accurate value. For example, the color of the lungpathways in (e) shifts toward pink due to down-sampling. The addition of points in (f) gives more true color compared to the original values in (a).

(a) 5123 volume rendering. 0.35 s/frame, 268mb. (b) 1283 volume rendering. 0.01 s/frame, 4mb.

(c) 1283 volume rendering with 3979k points:5123 error sampling grid, ⁄ error threshold,

0.29 s/frame, 40mb.

(d) 1283 volume rendering with 10999k points:5123 error sampling grid, ⁄ error threshold,

0.75 s/frame, 103mb.

(e) 2563 volume rendering: 0.04 s/frame, 33mb. (f) 2563 volume rendering with 7058k points:5123 error sampling grid, ⁄ error threshold,

0.47 s/frame, 97mb.

78

Figure 25 shows renderings of several different representations of the chest data. The two hybrid

renderings using a volumetric resolution of 1283 in images (c) and (d) provide a large improvement over the

1283 volume rendering alone in image (b). The ⁄ error threshold in image (d) produces an improvement

over the ⁄ error threshold in image (c) by extending the very ends of the lung airways. The 2563 volume

rendering in image (e) is able to capture the larger airways, but notice that the color is slightly incorrect: the

airways are more pink than those in the original. This is because the very low density areas around the

airways smear out the higher values, resulting in density values that are too low. The hybrid representation

in (f) is able to correct much of this error and add additional detail to the fine structures.

Figure 26 shows the improvement that the addition of points offer for a 1283 volume representation.

These images were generated by subtracting the standard and hybrid representations, respectively, from a

5123 reference rendering, and inverting the result for better clarity in print. The hybrid rendering noticeably

reduces the magnitude of the darker regions (those with highest error) of the difference image. The

magnitude of the lightly colored, low-error regions were not affected, because the error value of those

regions was below the threshold set during point generation (in this case, ⁄).

79

Black–white

(b)Pixel difference between 5123 resolution rendering and the same

1283 volume with 11 million points. mean = 245, std. dev. = 12

Figure 26 Pixel difference showing the effect of the addition of points. Values were computed as 256 –| high_res – low_res | for each channel, with white (256) indicating an exact match and black indicating that all channels had opposite values ( black – white ). The hybrid rendering increases the mean (reduces the average error) and reduces the standard deviation (fewer pixels have low value/high error).

No error

(a)Pixel difference between 5123 resolution rendering and

a 1283 volume rendering. mean = 238, std. dev. = 23

80

3.2.3.3 The Furby data set

The hybrid rendering method is particularly well-suited for mechanical data. This type of data is often

characterized by high resolution, sharp edges, fine detail, and large differences in density. The Furby data set

is a 512 × 512 × 2048 CT scan of a Furby, a mechanical toy that interacts with people by moving its body and

playing recorded pieces of conversation. It consists of a fur-covered plastic shell (the fur is not resolved in

the limited dynamic range of the data) containing a variety of electronic and mechanical parts. The original

data was reconstructed from 361 X-ray views at a resolution of 1746 × 1869.

The Furby exhibits many of the properties of mechanical data: it is too big to visualize in real time on

commodity hardware (1gb with one-byte normals), density values are concentrated at the upper and lower

extremes, and there are many small, sharp features. In the left column of Figure 27 on page 81, the transfer

function was selected to provide a faint outline of the plastic body in red, and to highlight the high-density

metal parts in yellow. The right column shows a slightly different transfer function and view angle, with the

outline of the cloth exterior (and, unfortunately, the box used to hold the toy) in purple, and the metal parts

in yellow and green.

Few metal parts are visible in the volume rendering (parts (a) and (b) of Figure 27) because the down-

sampling blurred these fine details into the surrounding low-density area (air). These areas have incorrect

density values and so do not appear when the transfer function is selected to highlight metal parts. The

larger metal parts that do appear are poorly resolved. However, in the hybrid renderings, the printed circuit

board, with the electronics parts coming off the top, is clearly visible in the abdomen of the Furby. The

hybrid method also restores the sharp corners on the metal bars used to pivot the joints, resolves the heads

on the screws, and does a particularly good job on the wires leading to the sensors in the forehead.

81

Figure 27 Comparison of hybrid rendering for the Furby data set. The original resolution was 512 × 512 × 2048 (1gb with 1-byte normals), too big to display on commodity hardware. The hybrid technique in (c) and (d) is able to restore much of the detail lost by subsampling. Using more points and a finer grid in (e) and (f) gives sharper, smoother details. For example, the wires leading to the head are better separated and less blocky.

(c) and (d): 2563 volume rendering with 2.9 million points, 5123 error sampling grid,1/16 error threshold, 0.36 s/frame 59mb.

(a) and (b): 2563 volume rendering, 0.07 s/frame, 33mb.

(e) and (f): 2563 volume rendering with 4.7 million points, 10243 error sampling grid,⅛ error threshold, 0.58 s/frame, 71mb.

82

Raising the resolution of the error sampling grid makes a visible difference in the quality of rendering

generated, but requires the error threshold be raised to prevent too many points from being generated.

Figure 27 shows the difference between error grids of 5123 and 10243. Raising the error threshold for the 10243

error calculation grid above ⅛ (the bottom row of Figure 27) to ⁄ (not shown) makes almost no visible

difference in the quality of rendering. This is because both ⅛ and ⁄ are enough to capture enough error

from the very dense metal parts, and neither ⅛ nor ⁄ is small enough to capture error present in the low-

density plastic and cloth regions. Raising the error threshold even further to ⁄ starts to capture error for

the plastic regions, but using a sampling grid of 10243 produces over 20 million points: too many to render

in real time.

3.2.4 Review of hybrid visualization for volume rendering

While visualization systems can handle increasing amounts of data, there will always be data sets larger than

the available graphics subsystem memory. Reducing data size by subsampling can allow this data to be

rendered, but many details and sharp transitions can be lost in this process. The hybrid method presented

here adds points to the volume where error between the original and subsampled data is large. In typical

data sets, these regions of high error are localized to material boundaries, meaning relatively few points can

make up much of the error in the subsampled image.

The most significant limitation of the hybrid approach for volume rendering is that the point-based

rendering performance is much lower than that of the texture-based rendering used by most hardware-

accelerated volume rendering methods. This means that even if data size is significantly reduced using the

hybrid representation, rendering performance might be unacceptable if large numbers of points are used.

Because of this limitation, the hybrid rendering technique is most useful when the original volume is too

large to fit into graphics memory, yet high-quality images are desired.

83

3.3 Hybrid rendering for particle visualization

The technique discussed above allows large-scale volumetric data to be more efficiently stored on

commodity PCs with limited memory and hard drive space. The hybrid point/volume technique can

leverage the same advantages of point-based and volume rendering for large-scale particle-based data: it

combines the speed of texture-based, hardware-accelerated volume rendering and the detail of point-based

rendering. This allows very large scale particle simulations to be interactively visualized on commodity PCs

with little degradation in image fidelity.

3.3.1 Particle beam visualization

Particle accelerators are playing an increasingly important role in basic and applied sciences, such as high-

energy physics, nuclear physics, materials science, biological science, and fusion energy. The design of next-

generation accelerators requires high-resolution numerical modeling capabilities to reduce cost and

technological risks, and to improve accelerator efficiency, performance, and reliability. While the use of

massively parallel supercomputers allows scientists to routinely perform simulations with 100–1,000 million

particles [Qiang 2000 (parallel)], the resulting data typically requires terabytes of storage space and over-

whelms traditional data analysis and visualization tools. At the same time, the complexity and importance of

these simulations makes visualization even more necessary.

The goal of beam dynamics is to understand the beam’s evolution in phase space as it propagates along

the accelerator. Two codes currently produces the results, consisting of the spatial coordinates (x, y, z) and

the momenta (Px, Py, Pz) of each particle: one using Fortran 90 [Qiang 2000 (halo)], and the other using

the Pooma (Parallel Object-Oriented Methods and Applications) C++ framework [Humphrey 1998]. Of

particular interest is the beam halo, the very-low-density region of charge far from the beam core. The halo

is responsible for beam loss as stray particles strike the beam pipe.

In the past, researchers visualized simulated particle data by either viewing the particles directly, or by

converting the particles to volumetric data representing particle density [McCormick 1999]. Each of these

84

techniques has disadvantages. Direct particle renderings takes too long for interactive exploration of large

datasets. Benchmarks have shown that it takes approximately 50 seconds to render 300 million points on a

Silicon Graphics InfiniteReality engine, and PC workstations are unable to even hold this much data in their

main memory. Reducing the data to fit on a smaller system can result in artifacts, and can hide fine

structures, especially in the critical low-density halo region of the beam.

Ideally, a visualization tool would be able to interactively visualize the beam halo of a large simulation

at very high resolutions. It would also provide real-time modification of the transfer function, and run on

high-end PCs rather than a supercomputer. This tool would be used to quickly browse the data, or to locate

regions of interest for further study. Once identified, these regions can be rendered offline at maximum

quality using a parallel supercomputer.

To address these needs, the system uses a combined particle- and volume-based rendering approach.

The low-density beam halo is represented by directly rendering its constituent particles. This preserves all

fine structures of the data, especially the lowest-density regions consisting of only one or two particles that

would be invisible using a volumetric approach. The high-density beam core is represented by a low-

resolution volumetric rendering. This area is of lesser importance, and is dense enough so that individual

particles do not have a significant effect on the rendering. The volume-rendered area provides context for

the particle rendering, and, with the right parameters, is not even perceived as a separate rendering style.

3.3.2 Data representation

To prepare for rendering, a multi-resolution, hierarchical representation is generated from the original,

unstructured, point data. The representation currently implemented is an octree, which is generated on a

distributed-memory parallel computer. This pre-processing step is performed once for each plot type

desired (since there are six values per point, many different three-dimensional plots can be generated from

each dataset). This data is later loaded by a viewing program for interactive visualization.

The hierarchical data consists of two parts: the octree data and the point data. At each octree node, the

density of points in the node and the minimum density of all sub-nodes is stored. At the leaf octree nodes

85

(the nodes at the finest level of subdivision), the index into the point data of the node’s constituent points is

stored. The leaf nodes should be small enough so that the boundary between point-rendered nodes and

volume-rendered nodes appears smooth on the screen. Simultaneously, the nodes need to be big enough to

contain enough points to accurately calculate point density.

Since the size of the point data is several times the available memory on the workstation used for

interaction, not all of the points can be loaded at once by the viewing program. Having to load points from

disk to display each frame would result in a loss of interactivity. Instead, the application takes advantage of

the fact that only low-density regions are rendered using the point-based method. High-density regions,

consisting of the majority of points in the dataset, are only volume rendered, and the point data is never

needed. Therefore, the points belonging to lower-density nodes are stored separately from the rest of the

points in the volume. The rendering program pre-loads these points from disk when it loads the data. It can

then generate images entirely from in-core data as long as the display threshold for points does not exceed

that chosen by the partitioning program. For this reason, the partitioning program generates approximately

as much pre-loaded data as there is memory available on the viewing computer.

86

Figure 28 The user interface, showing the volume transfer function (black box in the top right of the window) and the point transfer function (below it) with the phase plot (x, Px, z) of frame 170 loaded. This image consists of 2.7 million points and displays in about one second on a GeForce 3.

87

3.3.3 User interaction

The rendering program is used to view the partitioned data generated by the parallel computer. As shown in

Figure 28, it displays the rendering of the selected volume in the main portion of the window, where it can

be manipulated using the mouse. Controls for selecting the transfer functions for the point-based rendering

and the volume-based rendering are located on the right panel.

The volume transfer function maps point density to color and opacity for the volume-rendered portion

of the image. Often, a step function is used to map low-density regions to zero (fully transparent) and higher

density regions to some low constant so that one can see inside the volume. Ramp functions are used for the

transition between the high and low values, so the boundary of the volume-rendered region is less visible.

More complex transfer functions could also be used. However, they would not necessarily hide the tran-

sition between the point and volumetric data, and a great amount of control is not necessary for this

application: three-dimensional rendering is most commonly used for high-level exploration, while tradi-

tional full-resolution two-dimensional renderings are used for more detailed study of identified features.

The point transfer function maps density to number of points rendered for the point-rendered portion

of the image. Below a certain threshold density, the data is drawn rendered as points; above that threshold,

no points are drawn. Intermediate values are mapped to the fraction of points drawn. When the transfer

function’s value is at 0.75 for some density, for example, it means that three out of every four points are

drawn for areas of that density. This allows the user to see fewer points if too many points are obscuring

important features, or to make rendering faster. It also allows a smooth transition between point-rendered

portions of the image and non-point-rendered portions. Point opacity is given as a separate control, a

feature that can be useful when many points are being drawn.

By default, the two transfer functions are inverses of each other. Changing one results in an equal and

opposite change in the other. This way, there is always an even transition between volume- and point-

rendered regions of the image. In many cases, this transition is not even visible. The user can unlink the

functions, if desired, to provide more or less overlap between the regions.

88

3.3.4 Rendering

The octree data structure allows efficient extraction of the information necessary to draw both the volum-

etric- and point-rendered portions of the image. Volumetric data is extracted directly from the density

values of all nodes at a given level of the octree.

To draw the volume, a palette is loaded that is based on the transfer function the user specified for the

volumetric portion of the rendering. This palette maps each 8-bit density value of the texture to a color and

an opacity; regions too sparse to be displayed for the given transfer functions are simply given zero opacity

values. Then, a series of planes is drawn through the volume. The accumulation of these planes gives the

impression of a volume rendering. While often the highest possible resolution supported by the hardware

can be used for rendering, lower resolutions are often in this application. This is because the core of the

beam is typically diffuse, rendered mostly transparent, and is obscured by points. All beam images presented

here were produced using a volume resolution of only 643.

In contrast to the volume rendering, in which only the palette is changed in response to user input,

point rendering requires that the appropriate points from the dataset be selected each time a frame is

rendered. Therefore, those regions that are too dense to require point-based rendering should be quickly

eliminated. To display a frame, the maximum density node that can be represented with points is computed

based on the transfer function given by the user. Since each octree node contains the minimum density of

any of its sub-nodes, only octree paths leading to renderable leaf nodes must be traversed; octree nodes

leading only to dense regions in the middle of the beam need never be expanded.

Once the program decides that a leaf node must be rendered, it uses the point transfer function to

estimate the fraction of points to draw. Most commonly, this value is one (meaning all points are drawn in

the cell), but may be less than one depending on the transfer function specified by the user. The list of points

is then processed and every n-th one is drawn. The first point drawn is selected from a random index

between 0 and n. This eliminates possible visual artifacts resulting in the selection of a predictable subset of

points from data that may have structure in the order it was originally written to disk. Figure 29 illustrates

the two regions of the volume used in the image generation process.

89

3.3.5 Results

The system was tested using the results from a self-consistent simulation of charged particle dynamics in an

alternating-focused transport channel. The simulation, which was based on an actual experiment, was done

using 100 million particles. Each particle was given the same charge-to-mass ratio as a real particle. The

particles moving inside the channel were modeled including the effects of external fields from magnetic

quadrupoles and self-fields associated with the beam’s space charge. The three-dimensional mean-field

space-charge forces were calculated at each time step by solving the Poisson equation using the charge

density from the particle distribution. The initial particle distribution was generated by sampling a 6d

waterbag distribution (i.e. a uniformly filled ellipsoid in 6d phase space). At the start of the simulation, the

distribution was distorted to account for third-order nonlinear effects associated with the transport system

upstream of the starting point of the simulation. In the simulation, as in the experiment, quadrupole settings

at the start of the beam line were adjusted so as to generate a mismatched beam with a pronounced halo.

The output of the simulation consisted of 360 frames of particle phase space data, where each frame

contained phase space information at one time step.

Volume-rendered portionParticle rendered portion

Original octree representingparticle density Result

Figure 29 The image is created by classifying each octree node as belonging to a volume-rendered region or a point-rendered region, depending on the transfer functions for each region (the regions can overlap, as inthis example). The combination of the two regions defines the output image.

90

Figure 30 Comparison of a volume rendering (top) and a mixed volume/point rendering (bottom) of thephase plot (x, Px, y) of frame 170. The volume rendering has a resolution of 2563. The mixed rendering has a volumetric resolution of 643, two million points, and displays at about three frames per second. The mixed rendering provides more detail than the volume rendering, especially in the lower-left arm. Note that the renderings look different due to slight differences in the renderer. The volume renderer is also limited in thevery light areas by the resolution of the frame buffer.

91

Several frames of this data were moved onto a PC cluster for partitioning, although the data could have

been partitioned on the large IBM SP that was used to generate the data. Eight PCs were used, each with a

1.33 GHz AMD Athlon processor and one gigabyte of main memory. A typical partitioning step took a few

minutes, with most of the time being spent on disk I/O. The resulting data was visualized on one of the

cluster computers equipped with an nVidia GeForce 3.

Figure 30 shows a comparison of a standard volumetric rendering, and a mixed point and volumetric

rendering of the same object. The mixed rendering is able to more clearly resolve the horizontal

stratifications in the right arm, and also reveals thin horizontal stratifications in the left arm not visible in

the volume rendering from this angle.

Figure 31 shows how the view program can be used to refine the rendering from a low-quality, highly

interactive view, to a higher-quality less interactive view.

3.3.6 Review of hybrid visualization for particle rendering

Large particle accelerator simulations can be difficult to visualize because the extremely large size of the data

overwhelms most visualization systems. With the hybrid storage and rendering method presented here,

those areas of near-constant point density are replaced with the density value itself. The majority of the

points are in the dense but homogeneous beam center, so the resulting volume is able to represent the

majority of the data in any frame in a compact way. The less homogeneous and less dense areas of the beam

halo are kept in their original representation, preserving possible features of interest. The result is a

substantially smaller data set that can be rendered with the illusion of a full particle rendering.

3.4 Conclusions on hybrid visualization

Hybrid point/volumetric storage and rendering method improves rendering quality available for a given

amount of graphics hardware, and significantly reduces the amount of storage space required. Volume

rendering lacks the spatial resolution and the dynamic range to resolve regions with very high frequency,

areas which may be of significant interest to researchers. Point-based rendering lacks the interactive speed

92

and the ability to run on a desktop workstation for many large data sets. The hybrid point/volume approach

presented here combines the strengths of each to produce significantly smaller data with minimal loss in

quality.

The hybrid approach replaces areas with slowly varying density with a low-resolution volume

representation. For volumetric source data, this means generating a low-resolution version of the original.

For point-based source data, this means producing a low-resolution volume representing density. Points are

then added to this low-resolution volume data in those areas poorly represented in the original. In typical

data sets, these areas are localized: they are material boundaries in volume data, and they are the beam halo

regions for particle accelerator simulations. This means that relatively few points are necessary to achieve

renderings with quality close to that of the original.

93

Figure 31 A progression showing how exploration is performed. (a) Shows the initial screen, with a volume-only rendering. (b) The boundary between the high-density volume rendering and the low-density particle rendering has been moved to show more particles. (c) The transfer functions have been unlinked to show more particles while keeping the volume-rendered portion relatively transparent. (d) The point opacity has been lowered to reveal more structure. (e) The volume has been rotated to view it end-on. (f) A higher-resolution version similar to (d).

(a) (b)

(c) (d)

(e)

(f)

94

Conclusion

The increasing maturity of the computer graphics fields and the power of today’s computers have made

graphics and visualization a more integral part of many parts of society, including science and the arts. This

means, however, that many of the traditional methods and styles of rendering are no longer seen as unusual

or innovative. Now, the challenge is to produce the most useful and attractive images and animations

possible.

The production of artistically attractive renderings is often seen as completely separate from

visualization. But aesthetics and utility are intrinsically intertwined. Improving the speed and fluidity of an

animation makes it more attractive, but also increases the ability of viewers to comprehend the content.

Likewise, improving the clarity and simplicity of a visualization will simultaneously create an image that is

more aesthetically focused.

Attractive scientific renderings are also an important part of communication with the general public.

Attractive images are more likely to be published and viewed by the public, increasing the visibility of the

research in question. For example, the Hubble Space Telescope often takes many different exposures at

different wavelengths of a given object, useful for scientists but not understandable by the general public.

The Hubble Heritage Project was formed to take these measurements and generate attractive color images

for dissemination to the public. These color images are less scientifically useful than the originals, but “by

emphasizing compelling HST images distilled from scientific data, [they] hope to pique curiosity about our

astrophysical understanding of the universe we all inhabit” [Hubble Heritage]. It is efforts such as this that

make the Hubble telescope so widely known and appreciated by the general public. Many other scientific

95

projects can benefit from this type of public exposure for funding, for disseminating new research results,

and to inspire today’s students to enter careers in science.

This research has used an understanding of the human visual system to achieve better results than

traditional rendering for a variety of applications and with a variety of goals. Certain kinds of optimized

representations and rendering can increase efficiency while being visually very similar to a full rendering of

all the data. Also, an understanding of the way images are perceived helps create more simple and

understandable renderings. With an explosion of complex data, simplicity is one of the most important

aspects of a successful visualization; the eye should be directed to the areas of primary interest, and there

should be enough surrounding data to put those details into proper context.

The approach of using discrete primitives has worked very well for the applications addressed by this

research. Looking forward to the future of rich visualization, it is likely that the most significant gains will be

made through the intelligent combination of these rendering approaches with others. For example, using

image processing to understand human perception of a scene can be applied to any rendering technique,

including traditional volume and surface rendering. Combining many different rendering styles such as

stippling with surface meshes and volume rendering can help make sense of a complex multimodality data

set. Combining point-based and volume structures with an even wider range of representations such as

irregular grids and surfaces might further improve efficiency for new classes of problems.

The great range of representations and rendering styles gives a nearly limitless potential. The key to

developing this potential is the intelligent application and combination of these approaches. For example, if

image segmentation indicates ambiguous areas in a volume rendering with an embedded stippled surface,

how can this ambiguity best be resolved? On a machine with specific performance characteristics, what is

the best combination of volume and point-based techniques that will balance rendering performance with

storage performance? By optimally combining a variety of rendering and storage methods with little or no

human interaction, visually rich and informative visualizations can be open to people of all scientific

disciplines.

96

References

[Adams 1983] Ansel Adams. Examples: The Making of 40 Photographs. Little, Brown, and Company, Boston, 1983.

[Adams 1994] Rolf Adams, Leanne Bischof. Seeded Region Growing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 166, pp. 641–647, 1994.

[Alexa 2001] Marc Alexa, Johannes Behr, Daniel Cohen-Or, Shachar Fleishman, David Levin, and Claudio T. Silva. Point Set Surfaces. Proceedings of IEEE Visualization, pp. 21–28, 2001.

[Barnard 1991] Michael Barnard. Introduction to the Printing Process. Blueprint, London, 1991.

[Beuchemin 1995] Steven S. Beauchemin and John L. Barron. The Computation of Optical Flow. ACM Computing Surveys CSUR, 273, pp. 433–467, 1995.

[Buchanan 2000] John W. Buchanan and Mario C. Sousa. The Edge Buffer: A Data Structure for Easy Silhouette Detection. Proceedings of the Symposium on Non-Photorealistic Animation and Rendering NPAR, pp. 39–42, 2000.

[Cabral 1994] Brian Cabral, Nancy Cam, and Jim Foran. Accelerated Volume Rendering and Tomographic Reconstruction Using Texture Mapping Hardware. Proceedings of the Workshop on Volume Visualization, pp. 91–98, 1994.

[Cavanagh 1989] Patrick Cavanagh and Yvan G. Leclerc. Shape from Shadows. Journal of Experimental Psychology, 151, pp. 3–27, 1989.

[Chamberlain 1996] Bradford Chamberlain, Tony DeRose, Dani Lischinski, David Salesin, and John Snyder. Fast Rendering of Complex Environments Using a Spatial Hierarchy. Proceedings of Graphics Interface, pp. 132–141, May 1996.

[Chamberlain 1996] Bradford Chamberlain, Tony DeRose, Dani Lischinski, David Salesin, and John Snyder. Fast Rendering of Complex Environments Using a Spatial Hierarchy. Proceedings of Graphics Interface, pp. 132–141, 1996.

[Claes 2001] Johan Claes, Fabian Di Fiore, Gert Vansichem, and Frank Van Reeth. Fast 3D Cartoon Rendering with Improved Quality by Exploiting Graphics Hardware. Proceedings of Image and Vision Computing New Zealand IVCNZ, pp. 13–18, 2001

[Cline 1988] Harvey E. Cline, William E. Lorensen, Sigwalt Ludke, Carl R. Crawford, and Bruce C. Teeter. Two Algorithms for the Three-Dimensional Construction of Tomograms. Medical Physics, 153, pp. 320–327, 1988.

[Cohen 1991] Laurent D. Cohen. On Active Contour Models and Balloons. CVGIP: Image Understanding, 532, pp. 211–218, 1991.

97

[Curtis 1997] Cassidy Curtis, Sean Anderson, Josh Seims, Kurt Fleischer, and David Salesin. Computer-Generated Watercolor. Proceedings of SIGGRAPH, pp. 421–430, 1997.

[Deussen 2000 (a)] Oliver Deussen, Stefan Hiller, Cornelius van Overveld, and Thomas Strothotte. Floating Points: A Method for Computing Stipple Drawings. Computer Graphics Forum, 193 (Proceedings of Eurographics), pp. 41–51, 2000.

[Deussen 2000 (b)] Oliver Deussen and Thomas Strothotte. Computer-Generated Pen-and-Ink Illustration of Trees. Proceedings of SIGGRAPH, pp. 13–18, 2000.

[Deussen 1999] Oliver Deussen, Jörg Hamel, Andreas Raab, Stefan Schlechtweg, and Thomas Strothotte. An Illustration Technique Using Hardware-Based Intersections and Skeletons. Proceedings of Graphics Interface, pp. 175–182, 1999.

[Dong 2001] Feng Dong, and G.J. Clapworthy, and M. Krokos. Volume Rendering of Fine Details Within Medical Data. Proceedings of IEEE Visualization, pp. 387–394, 2001

[Dong 2003] Feng Dong, Gordon J. Clapworthy, Hai Lin, and Meleagros A. Krokos. Nonphotorealistic Rendering of Medical Volume Data. IEEE Computer Graphics and Applications, 234, pp. 44–52, 2003.

[Du 1999] Qiang Du, Vance Faber, and Max Gunzburger. Centroidal Voronoi Tesselations. SIAM Review, 414, pp. 637–676, 1999.

[Durand 2001] Frédo Durand, Victor Ostromoukhov, Mathieu Miller, François Duranleau, and Julie Dorsey. Decoupling Strokes and High-Level Attributes for Interactive Traditional Drawing. Proceedings of the Eurographics Workshop on Rendering, pp. 71–82, 2001.

[Ebert 2000] David Ebert and Penny Rheingans. Volume Illustration: Non-Photorealistic Rendering of Volume Models. Proceedings of IEEE Visualization, pp. 195–202, 2000.

[Freeman 1999] William T. Freeman, Joshua B. Tenenbaum, and Egon Pasztor. An Example-Based Approach to Style Translation for Line Drawings. Technical report tr99-11, Mitsubishi Electric Research Laboratories MERL, 1999.

[Freudenberg 2001] Bert Freudenberg, Maic Masuch, and Thomas Strothotte. Walk-Through Illustrations: Frame-Coherent Pen-and-Ink Style in a Game Engine. Computer Graphics Forum 203, 2001.

[Gauch 1999] John M. Gauch. Image Segmentation and Analysis via Multiscale Gradient Watershed Hierarchies. IEEE Transactions on Image Processing, 81, pp. 69–79, January 1999.

[Girshick 2000] Ahna Girshick, Victoria Interrante, Steven Haker, and Todd Lemoine. Line Direction Matters: An Argument for the Use of Principal Directions in 3D Line Drawings. Proceedings of the Symposium on Non-Photorealistic Animation and Rendering NPAR, pp. 43–52, 2000.

[Gooch 1998] Amy Gooch, Bruce Gooch, Peter Shirley, and Elaine Cohen. A Non-Photorealistic Lighting Model for Automatic Technical Illustration. Proceedings of SIGGRAPH, pp. 447–452, 1998.

[Grossman 1998] J. P. Grossman and William J. Dally. Point Sample Rendering. Proceedings of the Eurographics Rendering Workshop, pp. 181–192, 1988.

[Guptill 1997] Arthur L. Guptill. Rendering in Pen and Ink. Watson-Guptill Publications, New York, 1997.

[Halper 2002] Nick Halper, Stefan Schlechtweg, and Thomas Strothotte. Creating Non-Photorealistic Images the Designer’s Way. Proceedings of the Symposium on Non-Photorealistic Animation and Rendering NPAR, pp. 97–105, 2002.

[Hamel 1999] Jörg Hamel and Thomas Strothotte. Capturing and Re-Using Rendering Styles For Non-Photorealistic Rendering. Proceedings of Eurographics, pp. 173–182, 1999.

98

[Hansen 1997] Michael W. Hansen and William E. Higgins. Relaxation Methods for Supervised Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 199, pp. 949–962, September 1997.

[Haris 1998] Kostas Haris, Serafim N. Efstratiadis, Nicos Maglaveras, and Aggelos K. Katsaggelos. Hybrid Image Segmentation Using Watersheds and Fast Region Merging. IEEE Transactions on Image Processing, 712, pp. 1684–1699, 1998.

[Hertzmann 2000] Aaron Hertzmann and Denis Zorin. Illustrating Smooth Surfaces. Proceedings of SIGGRAPH, pp. 517–526, 2000.

[Hertzmann 2001 (a)] Aaron Hertzmann, Charles E. Jacobs, Nuria Oliver, Brian Curless, and David H. Salesin. Image Analogies. Proceedings of SIGGRAPH, pp. 327–340, 2001.

[Hertzmann 2001 (b)] Aaron Hertzmann. Paint by Relaxation. Proceedings of Computer Graphics International, pp. 47–54, 2001.

[Hiller 2001] Stefan Hiller, Oliver Deussen, and Alexander Keller. Tiled Blue Noise Samples. Proceedings of Vision, Modeling, and Visualization, pp. 265–272, 2001.

[Hiller 2003] Stefan Hiller, Heino Hellwig, and Oliver Deussen. Beyond Stippling—Methods for Distributing Objects on the Plane. Computer Graphics Forum, 223, pp. 515–522, 2003.

[Hoff 1999] Kenneth E. Hoff III, Tim Culver, John Keyser, Ming Lin, and Dinesh Manocha. Fast Computation of Generalized Voronoi Diagrams Using Graphics Hardware. Proceedings of SIGGRAPH, pp. 277–286, 1999.

[Hubble Heritage] Hubble Heritage Project. http://heritage.stsci.edu/

[Humphrey 1998] William Humphrey, Robert Ryne, Timothy Cleland, Julian Cummings, Salman Habib, Graham Mark, Ji Qiang. Particle Beam Dynamics Simulations Using the Pooma Framework. Proceedings of the International Symposium on Computing in Object-Oriented Parallel Environments ISCOPE, pp. 25–34, 1998.

[Interrante 1995] Victoria Interrante, Henry Fuchs, and Stephen Pizer. Enhancing Transparent Skin Surfaces With Ridge and Valley Lines. Proceedings of IEEE Visualization, pp. 52–59, 1995.

[Interrante 1997] Victoria L. Interrante. Illustrating Surface Shape in Volume Data via Principal Direction-Driven 3D Line Integral Convolution. Proceedings of SIGGRAPH, pp. 109–116, 1997.

[Kalnins 2002] Robert D. Kalnins, Lee Markosian, Barbara J. Meier, Michael A. Kowalski, Joseph C. Lee, Philip L. Davidson, Matthew Webb, John F. Hughes, and Adam Finkelstein. WYSIWYG NPR: Drawing Strokes Directly on 3D Models. ACM Transactions on Graphics, 213, pp. 755–762, 2002.

[Kass 1987] Michael Kass, Andrew Witkin, and Demetri Terzopoulos. Snakes: Active Contour Models. International Journal of Computer Vision, 14, pp. 321–331, 1987.

[Kniss 2001] Joe Kniss, Patrick McCormick, Allen McPherson, James Ahrens, Jamie Painter, Alan Keahey, and Charles Hansen. Interactive Texture-Based Volume Rendering for Large Data Sets. IEEE Computer Graphics and Applications, 214, pp. 52–61, 2001.

[LaMar 1999] Eric LaMar, Bernd Hamann, and Kennith I. Joy. Multiresolution Techniques for Interactive Texture-Based Volume Visualization. Proceedings of IEEE Visualization, pp. 355–361, 1999.

[Laur 1991] David Laur and Pat Hanrahan. Hierarchical Splatting: A Progressive Refinement Algorithm for Volume Rendering. Proceedings of SIGGRAPH, pp. 285–288, 1991.

[Levoy 1985] Marc Levoy and Turner Whitted. The Use of Points as a Display Primitive. Technical Report tr-85-022, The University of North Carolina at Chapel Hill, 1985.

99

[Lloyd 1982] S. Lloyd. Least Square Quantization in PCM. IEEE Transactions in Information Theory, 282, pp. 129–137, 1982.

[Lu 2002 (a)] Aidong Lu, Christopher J. Morris, David S. Ebert, Penny Rheingans and Charles Hansen. Non-Photorealistic Volume Rendering Using Stippling Techniques. Proceedings of IEEE Visualization, pp. 211–218, 2002.

[Lu 2002 (b)] Aidong Lu, Joe Taylor, Mark Hartner, David Ebert, and Charles Hansen. Hardware-Accelerated Interactive Illustrative Stipple Drawing of Polygonal Objects. Proceedings of Vision, Modeling, and Visualization, pp. 61–68, 2002.

[Lum 2001] Eric B. Lum and Kwan-Liu Ma. Non-Photorealistic Rendering Using Watercolor Inspired Textures and Illumination. Proceedings of Pacific Graphics, pp. 322–331, 2001.

[Lum 2002] Eric B. Lum and Kwan-Liu Ma. Hardware-Accelerated Parallel Non-Photorealistic Volume Rendering. Proceedings of the Symposium on Non-Photorealistic Animation and Rendering NPAR, pp. 67–74, 2002.

[Ma 2002] Kwan-Liu Ma, Greg Schussman, Brett Wilson, Kwok Ko, Ji Qiang, and Robert Ryne. Advanced Visualization Technology for Terascale Particle Accelerator Simulations. Proceedings of Supercomputing, pp. 1–11, 2002.

[Majumder 2002] Aditi Majumder and M. Gopi. Hardware Accelerated Real Time Charcoal Rendering. Proceedings of the Symposium on Non-Photorealistic Animation and Rendering NPAR, pp. 59–66, 2002.

[Malik 2001] Jitendra Malik, Serge Belongie, Thomas Leung, and Jianbo Shi. Contour and Texture Analysis for Image Segmentation. International Journal of Computer Vision, 431, pp. 7–27, 2001.

[Mao 1996] Xiaoyang Mao. Splatting of Non-Rectilinear Volumes Through Stochastic Resampling. IEEE Transactions on Visualization and Computer Graphics, 22, pp. 156–170, June 1996.

[Mardia 1988] Kanti V. Mardia and T. J. Hainsworth. A Spatial Thresholding Method for Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 106, pp. 919–927, November 1988.

[Max 1995] Nelson Max. Optical Models for Direct Volume Rendering. IEEE Transactions on Visualization and Computer Graphics, 12, pp. 99–108, 1995.

[McCool 1992] Michael McCool and Eugene Fiume. Hierarchical Poisson Disk Sampling Distributions. Proceedings of Graphics Interface, pp. 94–105, 1992.

[McCormick 1999] Patrick S. McCormick, Ji Qiang, and Robert D. Ryne. Visualizing High-Resolution Accelerator Physics. IEEE Computer Graphics and Applications, 195, pp. 11–13, 1999.

[Meier 1996] Barbara J. Meier. Painterly Rendering for Animation. Proceedings of SIGGRAPH, pp. 477–484, 1996.

[Meissner 1999] Michael Meißner, Ulrich Hoffmann, and Wolfgang Straßer. Enabling Classification and Shading for 3D Texture Mapping Based Volume Rending Using OpenGL and Extensions. Proceedings of IEEE Visualization, pp. 207–214, 1999.

[Meredith 2001] Jeremy Meredith and Kwan-Liu Ma. Multiresolution View-Dependent Splat Based Volume Rendering of Large Irregular Data. Proceedings of the Parallel and Large Data Visualization Symposium, pp. 93–99, 2001.

[Meruvia 2003] Oscar Meruvia Pastor, Bert Freudenberg, and Thomas Strothotte. Real-Time Animated Stippling. IEEE Computer Graphics and Applications, 234, pp. 62–68, 2003.

100

[Meyer 1990] Fernand Meyer and Serge Beucher. Morphological Segmentation. Journal of Visual Communication and Image Representation, 11, pp. 21–46, 1990.

[Mitchell 1987] Don P. Mitchell. Generating Antialiased Images at Low Sampling Densities. Proceedings of SIGGRAPH, pp. 65–72, 1987.

[Nagy 2002] Zoltan Nagy, Jens Schneider, and Rüdiger Westermann. Interactive Volume Illustration. Proceedings of Vision, Modeling, and Visualization, pp. 497–504, 2002.

[Nice 1993] Claudia Nice. Sketching Your Favorite Subjects in Pen and Ink. North Light Books, Cincinnati, Ohio, 1993.

[Nice 1995] Claudia Nice. Creating Textures in Pen and Ink with Watercolor. North Light Books, Cincinnati, Ohio, 1995.

[Northrup 2000] J. D. Northrup and Lee Markosian. Artistic Silhouettes: A Hybrid Approach. Proceedings of the Symposium on Non-Photorealistic Animation and Rendering NPAR, pp. 31–37, 2000.

[Ostromoukhov 2000] Ostromoukhov. Artistic Halftoning—Between Technology and Art. SPIE Volume 3963, pp. 489–509, 2000.

[Ostromoukhov 2001] Victor Ostromoukhov. A Simple and Efficient Error-Diffusion Algorithm. Proceedings of SIGGRAPH, pp. 567–572, 2001.

[Ostromoukhov 1999] Victor Ostromoukhov. Digital Facial Engraving. Proceedings of SIGGRAPH, pp. 417–424, 1999.

[Pavlidis 1990] Theodosios Pavlidis and Yuh-Tay Liow. Integrating Region Growing and Edge Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 123, pp. 225–233, 1990.

[Pfister 1999] Hanspeter Pfister, Jan Hardenbergh, Jim Knittel, Hugh Lauer, and Larry Seiler. The VolumePro Real-Time Ray-Casting System. Proceedings of SIGGRAPH, pp. 251–260, 1999.

[Praun 2000] Emil Praun, Adam Finkelstein, and Hugues Hoppe. Lapped Textures. Proceedings of SIGGRAPH, pp. 465–470, 2000.

[Praun 2001] Emil Praun, Hugues Hoppe, Matthew Webb, and Adam Finkelstein. Real-Time Hatching. Proceedings of SIGGRAPH, pp. 579–584, 2001.

[PRMan App. Note № 24] PhotoRealistic RenderMan Application Note 24: Using Arbitrary Output Variables in PhotoRealistic Renderman (With Applications). Pixar, 1998. http://graphics.stanford.edu/lab/soft/prman/Toolkit/AppNotes/appnote.24.html

[Qiang 2000 (parallel)] Ji Qiang, Robert D. Ryne, Salman Habib, Viktor Decyk. An Object-Oriented Parallel Particle-In-Cell Code for Beam Dynamics Simulation in Linear Accelerators. Journal of Computational Physics, 1632, pp. 434–451, 2000.

[Qiang 2000 (halo)] Ji Qiang and Robert D. Ryne. Beam Halo Studies Using a 3-Dimensional Particle-Core Model. Physical Review Special Topics—Accelerators and Beams Volume 3, 2000.

[Raskar 1999] Ramesh Raskar and Michael Cohen. Image Precision Silhouette Edges. Proceedings of the Symposium on Interactive 3D Graphics I3DG, 1999.

[Reeves 1983] William T. Reeves. Particle systems—A Technique for Modeling a Class of Fuzzy Objects. ACM Transactions on Graphics, 22, pp. 91–108, April 1983.

[Reeves 1985] William T. Reeves and Ricki Blau. Approximate and Probabilistic Algorithms for Shading and Rendering Structured Particle Systems. Proceedings of SIGGRAPH, pp. 313–322, 1985.

101

[Rosenfeld 1976] A. Rosenfeld, R. Hummel, and S. Zucker. Scene Labeling by Relaxation Operations. IEEE Transactions on Systems, Man, and Cybernetics, 66, pp. 420–433, 1976.

[Rosenfeld 1981] A. Rosenfeld and R. C. Smith. Thresholding Using Relaxation. IEEE Transactions on Pattern Analysis and Machine Intelligence PAMI, 35, pp. 598–606, September 1981.

[Rössl 2000] Christian Rössl and Leif Kobbelt. Line-Art Rendering of 3D-Models. Proceedings of Pacific Graphics, pp. 87–97, 2000.

[Röttger 2000] Stefan Röttger, Martin Kraus, and Thomas Ertl. Hardware-Accelerated Volume and Isosurface Rendering Based on Cell-Projection. Proceedings of IEEE Visualization, pp. 109–116, 2000.

[Rüdiger 1997] Rüdiger Westermann and Thomas Ertl. The VSBUFFER: Visibility Ordering of Unstructured Volume Primitives by Polygon Drawing. Proceedings of IEEE Visualization, pp. 35–42, 1997.

[Rusinkiewicz 2000] Szymon Rusinkiewicz and Marc Levoy. Qsplat: A Multiresolution Point Rendering System for Large Meshes. Proceedings of SIGGRAPH, pp. 343–352, 2000.

[Sahoo 1988] P. K. Sahoo, S. Soltani, A. K. C. Wang, and Y. C. Chen. A Survey of Thresholding Techniques. Computer Vision, Graphics, and Image Processing, 412, pp. 233–260, 1988.

[Salisbury 1994] Mike Salisbury, Sean Anderson, Ronen Barzel, and David Salesin. Interactive Pen-and-Ink Illustration. Proceedings of SIGGRAPH, pp. 101–108, 1994.

[Salisbury 1997] Michael P. Salisbury, Michael T. Wong, John F. Hughes, and David Salesin. Orientable Textures for Image-Based Pen-and-Ink Illustration. Proceedings of SIGGRAPH, pp. 401–406, 1997.

[Sato 2000] Mie Sato, Sarang Lakare, Ming Wan, Arie Kaufman, and Masayuki Nakajima. A Gradient Magnitude Based Region Growing Algorithm for Accurate Segmentation. Proceedings of the International Conference on Image Processing, pp. 448–451, 2000.

[Schumann 1996] Jutta Schumann, Thomas Strothotte, Stefan Laser, and Andreas Raab. Assessing the Effect of Non-Photorealistic Rendered Images in CAD. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 35–41, 1996.

[Secord 2002] Adrian Secord, Wolfgang Heidrich, and List Streit. Fast Primitive Distribution for Illustration. Thirteenth Eurographics Workshop on Rendering, pp. 215–226, 2002.

[Secord 2002] Adrian Secord. Weighted Voronoi Stippling. Proceedings of the Symposium on Non-Photorealistic Animation and Rendering NPAR, pp. 37–43, 2002.

[Shi 1997] Jianbo Shi and Jitendra Malik. Normalized Cuts and Image Segmentation. IEEE Conference on Computer Vision and Pattern Recognition, pp. 731–743, 1997.

[Shirley 1990] Peter Shirley and Allan Tuchman. A Polygonal Approximation to Direct Scalar Volume Rendering. ACM Computer Graphics (San Diego Workshop on Volume Visualization), 245, pp. 63–70, 1990.

[Simmons 1992] Gary Simmons. The Technical Pen. Watson-Guptill Publications, New York, 1992.

[Sousa 2003] Mario Costa Sousa and Przemyslaw Prusinkiewicz. A Few Good Lines: Suggestive Drawing of 3D Models. Computer Graphics Forum, 223 (Proceedings of Eurographics), pp. 381–390, 2003.

[Strassmann 1986] Steve Strassmann. Hairy Brushes. Proceedings of SIGGRAPH, pp. 225–232, 1986.

[Swan 1997] J. Edward Swan II, Klaus Mueller, Torsten Möller, Naeem Shareef, Roger A. Crawfis, and Roni Yagel. An Anti-Aliasing Technique for Splatting. Proceedings of IEEE Visualization, pp. 197–204, 1997.

[Treavett 2000] Steve M. F. Treavett and Min Chen. Pen-and-ink Rendering in Volume Visualization. Proceedings of IEEE Visualization, pp. 203–210, 2000.

102

[Treavett 2001] Steve M. F. Treavett, Min Chen, Richard Satherly, and Mark W. Jones. Volumes of Expression: Artistic Modeling and Rendering of Volume Datasets. Proceedings of Computer Graphics International, pp. 99–106, 2001.

[VanGelder 1996] Allen Van Gelder and Kwansik Kim. Direct Volume Rendering with Shading via Three-Dimensional Textures. Proceedings of the IEEE Symposium on Volume Visualization, pp. 23–30, 1996.

[Vincent 1991] Lee Vincent and Pierre Soille. Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 136, pp. 583–598, 1991.

[Wand 2001] Michael Wand, Matthias Fischer, Ingmar Peter, Friedhelm Meyer auf der Heide, and Wolfgang Straßer. The Randomized Z-Buffer Algorithm: Interactive Rendering of Highly Complex Scenes. Proceedings of SIGGRAPH, pp. 361–370, 2001.

[Weber 2001] Gunther H. Weber, Hans Hagen, Bernd Hamann, Kenneth I. Joy, Terry J. Ligocki, Kwan-Liu Ma, and John M. Shalf. Visualization of Adaptive Mesh Refinement Data. Proceedings of SPIE, pp. 121–132, 2001.

[Weiler 2000] Manfred Weiler, Rüdiger Westermann, Charles Hansen, Kurt Zimmerman, and Thomas Ertl. Level-of-Detail Volume Rendering via 3D Textures. Proceedings of the IEEE Symposium on Volume Visualization, pp. 7–13, 2000.

[Westover 1989] Lee Westover. Interactive Volume Rendering. Proceedings of the Chapel Hill Workshop on Volume Visualization, pp. 9–16, 1989.

[Wilhelms 1996] Jane Wilhelms, Allen Van Gelder, Paul Tarantino and Jonathan Gibbs. Hierarchical and Parallelizable Direct Volume Rendering for Irregular and Multiple Grids. Proceedings of IEEE Visualization, pp. 57–64, 1996.

[Williams 1998] Peter L. Williams, Nelson L. Max, and Clifford M. Stein. A High Accuracy Volume Renderer for Unstructured Data. IEEE Transactions on Visualization and Computer Graphics, 41, pp. 37–54, 1998.

[Wilson 1994] Orion Wilson, Allen Van Gelder, and Jane Wilhelms. Direct Volume Rendering via 3D Textures. Technical Report UCSC-CRL-94-19, Jack Baskin School of Engineering, University of California at Santa Cruz, 1994.

[Winkenbach 1994] Georges Winkenbach and David H. Salesin. Computer-Generated Pen-and-Ink Illustration. Proceedings of SIGGRAPH, pp. 91–100, 1994.

[Winkenbach 1996] Georges Winkenbach and David H. Salesin. Rendering Parametric Surfaces in Pen and Ink. Proceedings of SIGGRAPH, pp. 469–476, 1996.

[Yellot 1983] John I. Yellot, Jr. Spectral Consequences of Photoreceptor Sampling in the Rhesus Retina. Science, volume 221, pp. 382–385, 1983.

[Zhu 1996] Song Chun Zhu and Alan Yuille. Region Competition: Unifying Snakes, Region Growing, and Bayes/MDL for Multiband Image Segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 189, pp. 884–900, 1996.

[Zwicker 2001] Matthias Zwicker, Hanspeter Pfister, Jeroen van Baar, and Markus Gross. EWA Volume Splatting. Proceedings of IEEE Visualization, pp. 29–36, 2001.