modelling the texture of paint

EUROGRAPHICS '92 / A. Kilgour and L. Kjelldahl (Guest Editors), Blackwell Publishers © Eurographics Association, 1992

Volume 11, (1992), number 3

Modelling the Texture of Paint

Tunde Cockshott, John Patterson, David England

Glasgow Interactive Systems cenTre (GIST), Department of Computer Science, University of Glasgow, Glasgow G12 8QQ, UK.

Tel: +44 (0)41 339 855 Fax: +44 (0)41 330 4913 EMail: ( tundejwp,de)@dcs.glasgow.ac.uk,

Abstract We present an extension to the WET & STICKY model. A method of modelling textured and shiny paint is described using bump mapping and standard illumination models . The qualities of the paint are used to supply input to the texture and lighting models. We demonstrate that a dynamic model of textured paint provides the user with valuable visual clues for the production of realistic painted images. We also propose further extensions which would allow the use of different lighting models for different paints, allowing for paint attributes which are unachievable either with traditional painting or with earlier forms of electronic painting.

Keywords: simulation, electronic paint system, bump mapping, interaction, parallelism.

1. Introduction

Electronic paint systems usually model the process of applying 'paint' solely in terms of changing the colour of a brush footprint to or towards a specified paint colour. This is a quite incomplete model of what can be seen when real paint is applied and in particular such properties as the surface texture of the paint, which can put local variations into its nominal colour, are not taken into account. This paper is about an electronic paint system, the WET & STICKY system[Cockshott 91], which models the 'naturalistic' properties of real paints, and in particular about how surface texture is modelled and rendered to give the appearance of the superficial textures of 'real' paint. The paper is essentially in two parts, the first describing how WET & STICKY simulates the behaviour of real paint through computationally lightweight analogues of its properties and concentrating on those properties which affect surface texture, and the second describes how we extract a quantity which can be interpreted as the instantaneous height of the paint above the substrate and how we calculate the necessary illumination parameters from it. Photographs showing the texture of flowing paint produced using WET & STICKY and rendered using well-known illumination models appear near the end of the paper.

The origins of the WET & STICKY system arise from the expressed dissatisfaction of many critics with the sterile quality which tends to pervade the images produced with of electronic paint systems. To understand this dissatisfaction one only has to walk round almost any exhibition of paintings. If one closely examines the marks that make up the complete image it immediately becomes apparent the incredible range, variety, and complexity of these fundamental building blocks. If one now carries out a similar examination of electronic paint images one will see a much narrower range, a lesser variety, and a reduced level of complexity. The aim of WET & STICKY was to produce a model which would support the production of images whose marks possess

mailto:tundejwp,de)@dcs.glasgow.ac.uk

C-218 T. Cockshott et al. / Modelling the Texture of Paint

the same degree of complexity and variety as those found in real paintings, and to allow for the accidental effects inherent in real painting. To achieve this WET & STICKY simulates traditional artists’ media to a much greater degree of fidelity than attempted hitherto.

WET & STICKY is still evolving but its present incarnation runs too slowly on sequental processors to be an effective interactive tool. A version of WET & STICKY is being developed to run on multiprocessors [England&Cockshott 1991] and we are beginning to see speed-ups which, with careful attention to the style of interaction, could lead to a viable interactive system. At present the application of paint is shown only as an area of colour which behaves in much the way one would expect real paint to behave and most of the work in maintaining the simulation involves managing that behaviour. However, the potential for an interactive application has encouraged us to consider modelling the actual appearance of the paint, with ridges and patterns resulting from the application and flow of the paint before it dries. Some artwork (e.g. that of Van Gogh) derives much of its character from the textures of thickly-applied paint. This paper is directed to the problem of rendering the textures of various kinds of paint with a view to producing a visually convincing simulation and visually interesting images.

2. Philosophical approach

Although the model is based on real world elements, the role of WET & STICKY is not that of an accurate simulation of reality in the same sense that a physicist might build an accurate simulation of fluid flow. To the artist user it is the apparent , not the actual correctness of the visual and behavioural characteristics of the medium that is important. The model’s responses and appearance should be consistent with the artist’s expectations. We could use [Hayes’ 1978] concept of “naive physics” expanded to cover computation by [Owen 1986], as a good way of describing the level of competence required from the model. Naive physics can be regarded as an informal, often phenomenological, precis of real laws of physics, which people develop from their encounters with the world. As long as the model responds in accordance with the artist’s naive physical concepts, then how (and the exact reasons why), these responses occur, is not of importance to the artist.

To explain this, one can use the example of a stage set of a street. A stage set is a simulation in as far as it provides a thin veneer that attempts to give the appearance of reality. The viewer knows that what he is seeing is not really a street. All he sees is the facades of the buildings. They may even appear to function like real buildings, with opening doors and windows, and functioning staircases. But the structure behind and supporting the veneer is nothing like that encountered with real buildings. The similarity between the reality that the stage-set is depicting and the original need only go as far as is required to persuade and convince the viewer that, for the period or purpose of the play, what he is seeing is a real street.

The way this philosophy has been interpreted in WET & STICKY involves an exploration of what constitutes a minimalist simulation of physical reality. Early models [Cockshott 1991] did not include analogues of certain physical properties like surface tension, and the absence of their mimics led to unrealistic behaviour, e.g. paint dripping unnaturally. The development of WET & STICKY has been one of progressively introducing minimal simulations of these properties until realistic behaviour, that is behaviour like that seen when an artist applies real paint on to a canvas, was achieved.

A question that is often asked at this stage is: why bother? Most paint systems are easy to make interactive and there are a number of new paint systems, eg Hairy Brushes [Strassman], Painter [Fractal Design], Monet [Delta Tao] which supposedly extend the range of marks achievable by electronic paint systems. None of these systems requires anything like the scale of resources that WET & STICKYs simulation demands. The answer to this question has more parts than might be expected from the introduction to this paper. The ‘obvious’ answer is that because the WET & STICKY model is based on a simulation of the attributes and behaviour of real paint it allows more highly textured marks than i s possible with any other paint system. In addition these marks are more like the kinds of result an artist, who has spent many years developing mark making skills which rely on the natural characteristics of the physical media, would expect. Accordingly WET & STICKY allows traditional artists to capitalize on their existing skills while working in the electronic

T. Cockshott et al. / Modelling the Texture of Paint C-219

medium. Artists develop inutitions about how a medium may behave. WET & STICKY, by capturing some of the behaviour of traditional media allows the artist to develop similar intuitions about a purely electronic medium. In particular the professional artist recognizes the value of accommodating the accidental into a work and many paintings have had their character reinforced by such accommodation. Most of WET & STICKYs effects cannot be undone for the simple reason that the amount of data that would have to be kept would defeat even a computer powerful enough to run the simulation.

The least obvious answer to the question posed only became apparent as the research progressed. This i s that it is possible to achieve novel and interesting effects by switching attribites and treating simulation elements as interchangeable. For example WET & STICKY normally simulates the behaviour of paint flowing on an upright canvas where the direction of gravity is always the same. However, the direction (and magnitude) of gravity is just an attribute which can be associated with a given region of the canvas or just painted on so that wet paint runs in different directions at different places on the canvas. An example is shown in Plate 1*. In Plate 1 the canvas has been treated with gravity paint of opposite polarities and then had strokes of three different colours of paint added. The result has been a clearly visible mixing of the paint at the ‘gravity boundary’ which looks like a realistic fluid flow simulation although the formulae of fluid mechanics were not involved. For this reason we refer to this kind of simulation as a minimalist simulation. These minimalist simulations are of particular relevance to the topic of the present paper as it is possible to exploit the parameters required to model textures to produce novel images based on the illumination characteristics of naturally unworkable or impossible materials.

3. Overview of the basic model

The origins of the WET & STICKY model are rooted in the ideas of cellular automata, [Codd 1968]. A cellular automaton can be thought of as a large collection of interconnected finite automata. One usually refers to these finite automata as cells. In WET & STICKY these cells are arranged in the form of a 2D array of interacting array elements.

There are three parts to the model, the Intelligent Canvas, Paint Particles, and the Painting Engine. These correspond to the elemental building blocks, interaction and behavioural rules as follows;

Paint Particles represent the paint,

the Intelligent Canvas

the Painting Engine

is the substrate ,The term intelligent is used because it also has knowledge of what is upon its surface plus its orientation and

is the set of rules and environmental effects that act upon and govern the behaviour and interaction of the Paint particles and the Intelligent Canvas.

The intelligent canvas is an array of cells each of which has the ability to hold paint particles. The term particle is not used to describe a particle system, [Reeves 1983], but as a way of describing discrete units of paint. Paint particles may be thought of as self-contained blocks of paint. Each block contains information about its attributes, such as colour, and liquid content. Just like the Paint Particles, each canvas cell contains information about its attributes, such as horizontal and vertical orientation and absorbency. The cells also know about the type and volume of paint that they hold. These cells may be thought of as open topped containers which can hold Paint Particles. The canvas is rather like a large ice-cube tray: as paint i s poured into one canvas cell it will fill it up and then try to overflow into its neighbours, but unlike an ice-cube tray each cell may have a different capacity before it starts to overflow, and each cell may also have a different idea of which is the prefered direction of flow. Adjacent cells may differ in their attributes in order that the canvas may mimic different characteristics across its surface, such as sized and un-sized areas. Size is an animal based glue used to seal the surface of a substrate to reduce its absorbency It also allows the model to mimic entirely different surfaces on the same canvas. Such an example would be a canvas which had pieces of paper or other materials stuck to its surface.

* See page C-476 for Plate 1.


In the versions of the WET & STICKY model described in this paper the cells that comprise the canvas map directly onto pixels on the display screen. As this is a 1: 1 mapping we would expect to see aliasing effects, but in fact none have been seen in any of the images we have produced. The explanation would appear to be that the system is self-anti-aliasing by virtue of the fact that paint progressively enters adjacent cells and mixes with what is there. More generally it does not force its colour onto a new cell immediately. If thin, or thinly-applied, paint is treated as not wholly opaque [Oddy&], that is its transparency is determined by thickness up to a certain depth of paint, the paint which moves onto unpainted canvas will take on intermediate colours until it is thick enough to assert its own colour.

The artist paints onto the surface of the Intelligent Canvas with a brush in the same way that a traditional system would allow the user to paint colour onto the frame buffer. Although in essence the brush works in the same way as a traditional simple paint system brush, in this case it now deposits paint particles onto the canvas rather than colour into a frame buffer. Any canvas cells under the path of the brush receive a number of paint particles. The quantity deposited is determined by the characteristics of the brush The number of particles held by a cell is known as the volume of paint held by the cell. The artist can control the attributes of paint and canvas via a set of sliders representing the values decribed in the next section. However, sliders are not the ideal interface to attribute selection. A better approach would be a set of small ‘playpens’ where the artist could experiment with paints and a canvas of differing values before applying them to the image. Experimentation would be particularly important for the non-visible attributes such as gravity and canvas absorption.

The state of a cell is determined by the state of its attributes and those of the paint it contains. By painting on the surface of the canvas the artist alters the state of the individual canvas cells. If this was the only way for the state of the canvas to change then the system would be static in between inputs from the artist and any changes to the image or state would be explicit. That is to say the artist could only change the image by applying more paint or by using a tool that affected the paint already on the surface. In reality paints will run and bleed on their own accord if their attributes permit, not because the artist uses a special tool. In the studio the painting will evolve by itself without the artist’s intervention.

4. Painting Engine

To mimic this independent behaviour of real paint the system has what is termed a painting engine. The engine runs as an omnipresent process continually visiting, interrogating, and updating the state of the paint held by the canvas cells.

On visiting a cell the painting engine attempts to perform set of rules upon the cell’s paint. It does these in the order illustrated in Figure 1. The first purpose of the painting engine is to mimic the passage of time by ageing the paint held by the cell. Ageing the paint has the effect of reducing its liquid content, which has a direct influence upon how the paint behaves in response to environmental conditions. In effect this process is mimicking the results of evaporation. The more liquid a paint contains the longer it will take for that liquid to evaporate. The rate of evaporation is dependent on the surface area and the drying rate of the paint. A quantity of paint with a high liquid content spread out over a large area would dry faster than if it were spread over a smaller area. By reducing the surface area one reduces the rate of evaporation. In the WET & STICKY model the unit of surface area, the cell, is fixed thereby creating a uniform cellular unit area of evaporation across the entire canvas. Altering the drying rate alters the rate at which liquid evaporates from the cell.

Secondly, the painting engine sees which environmental conditions would affect the state of the cells which it visits. The effects it is modeling in this role can be classed as dripping and spreading effects, that is,

the effect of gravity acting upon the paint, a notion of sideways spreading, similar to the effect of diffusion, and, the forces of surface tension acting against the movement of paint.


Figure 1 One cycle of painting engine. The engine may only perform a partial cycle dependant on the effects of ageing, diffusion, or gravity.

The painting engine looks at the state of a cell and refers to its set of rules regarding the behaviour of paint and determines if there is the potential for paint to flow from that cell to one of its neighbours. It then looks at the states of all its neighbours and decides if any of them will accept any paint. If so it then changes the cell's state and updates any of its neighbouring cells which are affected by this change.

The third way the painting engine works is in response to the application of paint by the user and the redistribution of paint between the cells caused by the dripping and spreading effects. In this role the engine is attempting to mix paint. It takes any new paint that has arrived in a cell, by whatever means, and sees if it is compatible with that already in the cell. If so, the model mixes all the attributes of the paints proportionally according to the relative volumes each of the two paints will contribute to the new paint, otherwise it makes the incoming paint the surface paint. Currently the paint engine does not alter its rules as liquid evaporates from the canvas. This, and similar minor improvements to the paint engine, would add computational cost without any great improvements in the realism of the paint's behaviour.

A detailed description of the algorithms used by the painting engine can be found in Cockshott [1991].

5. Wet & Bumpy

The use of the concept of volumes of paint associated with individual cells of the intelligent canvas leads naturally to the notion that these volumes may be interpreted as heights. Each paint cell has unit area and consitutes a resevoir which, when filled with a given volume of paint will be filled to a height defined by the magnitude of the paint volume. Where the reservoirs are fuller than their neighbours paint will flow out of them into less full adjacent cells subject to the restraining mechanisms described earlier. Another way of looking at this is to say that the reservoirs form a quantization of the continuous spread of paint on a real


canvas and the restraining mechanisms are adequately detailed analogues of physical processes which affect paint so applied. Accordingly the height values can be interpreted as the instantaneous and locally-varying heights of the paint as it flows on the canvas and, as they form a regular array, can be interpreted as a texture which could be revealed by Bump Mapping [Blinn 1978]. We note that this differs from the application of a pre-defined texture maps as might be found in other conventional paint systems. Such systems might allow the application of a pre-defined oil, charcoal or wood texture to a drawn image. In our approach, however, the surface texture is derived from the behaviour of paint over time.

The key step in bump mapping is the calculation of the perturbation of the surface normal by including consideration of a texture mapped onto the surface. This texture is normally provided as a regular array of height samples,as here, but i s precalculated whereas we are obtaining them as the result of lightweight simulation.This does not affect the calculation, but dies affect the significance of the outcome. Here the calculation of the normal can be hugely simplified as the base surface is a plane oriented so that its normal N is always in the direction N = [0, 0, 1].

Figure 2a: Cell Reservoir Model Figure 2b: Quantized Height Model

Although this case seems very straightforward the computationally simplest method of deriving the perturbed normal is only obtained using Blinn's[1978] formulae. The model we use for the paint surface is that of a mesh of triangular facets which form a pyramid-like surface around each vertex. The triangles are made up out of planes which pass through the point x, y, in the cell x,y with reservoir height , and through the points given by the x, y coordinates of the diagonally adjacent cells and the height of the vertically or horizontally adjacent cell between them. The resultant triangular facets may thus not necessarily meet along edges which are exactly parallel to the diagonals of the cell mesh but a closed pyramid is defined around each pixel in this way. In the terminology of Figure 2b, the four facets contain the lines XP, XQ, XR, XS respectively, with the facets containing XP, XR presenting in the x-direction and those containing XQ, XS in the y-direction.

Our formula for the perturbed normal at point X is determined by calculating the normals for each of the facets containing XP, XQ, XR, XS and then averaging them. Rather than just calculate the normal for each facet f directly, we use Blinn's [1978] formula:

where is the unperturbed normal (always in the plane), and for a plane we can equate u, v with x, y so


Here surface whose normal is being perturbed. So P defines the plane z = 0, that is P = [x, y, 0 ]. Accordingly,

are the slopes for the facet in the x, y directions respectively, and P is the position vector for the

and

where out the vector products,

and x and y are the x and y-axis vectors respectively Since we get, on multiplying

So is the vertical component of and the horizontal component.

If we identify each facet by the point (P, Q, R, S) adjacent to X which contributes its height value to determine the baseline of the facet, then

The average normal at point is given by some weighted average of the individual normals Since each formula for contains , which is equal to N, it is unnecessary to scale the component normals differently as the vertical components at X are the same. Accordingly we just average the horizontal components to get

i.e.

This normal may now be used in illumination calculations after itself being normalized.

In developing the WET & STICKY model our goal has first been to see what was originally required to simulate reality accurately enough, and then to see what exploitable effects can be obtained through interchanging the model components or altering the parameters or attributes until they lie outside what can be achieved in nature. (One particularly useful example i s the ability to paint on dryness to control the flow of wet paint interactively.) Examples of the foregoing model being used to simulate the surface appearance of applied paint


using the Phong model are in Plate 2. In each case the paint has very slow drying attributes and has been allowed to drip for a while.

The specular component of the Phong model requires two parameters, a specular weighting factor and a power term which results in tight specular highlights for high values of the term. These tight highlights make a surface look shiny, a characteristic which is also true of wet paint. Accordingly we included in our model an (empirical) multiplier for the specular weighting, and for the power term, whose combined effect is to make paint look shiny and dry paint look matt.

The Phong model also requires a vector V pointing towards the eye and this is obtained as where d is the distance of the observer for the screen. This vector V also has to be normalized to take part in illuminated calculations. The quantities required for practically any other illumination model involving isotropic scattering are thus available. The interesting aspect of this, as far as WET & STICKY is concerned, is that the illumination model may itself be associated with the paint. Clearly paints with compatible illumination models can mix and their parameters arranged in proportion, but if two paints with radically different illumination models mix then they will have to be averaged as early in the calculation tree[Cook 1984] at which it is possible to identify compatible steps. Possibilities include physically-based illumination models, allowing shiny metal or gemstone paint, or materials with physical characteristics not found in nature (eg by manipulating the Fresnel Extinction Coefficient).

Finally the simulation of the appearance of the paint once it is applied requires that the simulation of the applicator has to allow for the possibility of the uneven application of paint into the canvas. An early WET & STICKY model included a a simulation of [Strassman 1987] Hairy Brushes model, but once wet paint was modelled the effects gained by the Hairy Brushes simulation vanished. Now that the uneven application of paint can be seen, a model, effectively a superset of Hairy Brushes, will be required for the process of applying paint in the uneven way artists have come to expect.

6. Conclusions

Simulating the surface texture of paint has always been a goal of the WET & STICKY programme but, although the method of rendering that texture is much simpler than the general case, the normal perturbation and illumination calculation is significantly more time-consuming than the other steps in the cycle of the paint engine. The problems in parallelizing WET & STICKY have all been in contention for framestore access, so extending the amount of work done per cell by the paint engine can be tolerated so long as sufficient processor resources are available. The merits of modelling paint texture have therefore to be judged entirely on their aesthetic qualities and the realism of the simulation. Plate 2*shows a quite plausible-looking dripping wet paint so the technical problems of texture modelling have been resolved, for previously applied paint, using a minimalist model in keping with other elements of WET & STICKY.

Also in keeping with experience with WET & STICKY is the fact that no aliasing or Mach banding effects can be seen although no special provision has been made to prevent them. Regular wavy surfaces are particularly prone to aliasing effects although bump-mapping usually isn't. The reason bump-mapping does not usually show intrinsic aliasing is because there is an interpolation step involved in reconstructing the relevant height value for the projection of the sample point into texture space. There is no such interpolation in WET & STICKY texture mapping because the reservoir heights always coincide with the required samples. This doesn't alias because the simulation of the fluid properties (effectively of the flow) induces correlations in adjacent reservoir height values.

Now that we can simulate surface texture the next step will be to model accurately the process of applying the paint. This will produce an initial surface texture and will have to be rendered continuously, which effectively forces a parallel implementation. However, with all but the thinnest paints we would expect the texture to be furrowed by the passage of a brush-like applicator in the direction of the brush stroke and this would require the use of anisotropic reflection models [Poulin & 1990], so there are new issues still to address in dealing with the texture of applied paint.

* See page C-476 for Plate 2.


We have shown that only a relatively simple extension to the basic WET & STICKY model is required to produce fully textured paints. A paint system which allows interaction with dynamic three dimensional paint has many potential advantages over systems which model only the flat, static nature of paint. The texture of the paint adds more than an extra physical dimension to the image, it also expands its expressive possibilities.

This work follows the initial aims of WET & STICKY and is a further attempt to bring the computer nearer to the artist. The system rewards the traditonally trained artist by exploiting his existing skills.

Acknowledgments

The authors would like to thank Kevin Waite for his programming assistance and Professor Steven Todd of IBM Research Centre Winchester for his encouragment and general assistance.

References

[Blinn] Blinn, J.F. Simulation of Wrinkled Surfaces, Computer Graphics, proc SIGGRAPH '78 12, (2) pp 286-292 (August 1978)

[Cockshott 1991] Cockshott, T, Wet and Sticky: A novel model for computer based painting. PhD. Thesis, Computing Science Department (Research Report 91/R20), University of Glasgow, 1991.

[Cockshott & England 1991], Cockshott, T. & England, D. Wet and Sticky: Supporting Interaction with Wet Paint, Editors Diaper, D. and Hammond, H. People and Computers VI: Proceedings of the HCI '91 Conference, University Press Cambridge, August 1991

[Codd 1968] Codd, E. F., Cellular Automata, New York: Academic Press Inc, 1968

[Cook 1984] Cook, R.L. Shade Trees Computer Graphics, Proc SIGCRAPH '84,18 (3) p223-232 (July 1984)

[Delta Tao] Delta Tao inc.Monet Users Manual, Delta Tao, 1991

[England & Cockshott] England, D., Cockshott, T. Painting with wet paint on a transputer-based system, Proceedings of Applications of Transputers 3, ( T S Durani et al. eds), IOS Press, 1991.

[Fractal Design] Fractal Design Corp.Painters User Manual, Fractal Design, Aptos California, 1991

[Foley &] Foley, J.D. et al Computer Graphics: Principles & Practice (2nd Edition), Addison- Wesley, Reading, Mass 1990.

[Hayes 1978] Hayes P. J.The Naive Physics manifesto in Expert systems in the microelectronic age, Editor D. Michie, Edinburgh University Press, 1978


[Oddy & Willis 1991] Oddy, R, J and Willis, P, J A Physically Based Colour ModelComputer Graphics Forum, Volume 10, Number 2, 1991

[Owen 1986] Owen D.Naive Theories of Computation, page 187 in User Centered System Design, Editors Donald E. Norman, Stephen W. Draper, Lawrence Erlbaum Associates Publishers, 1986

[Poulin & 1990] Poulin, P., Fournier, A. A Model for Anisotropic Reflection, Computer Graphics, Proc. SIGGRAPH '90, 24 (4) p273-282 (August 1990)

[Reeves 1983] Reeves W T Particle systems - a Technique for Modelling a Class of Fuzzy Objects, ACM Transactions on GraphicsVol. 2, No. 2, ACM, 1983

[Strassman 1987] Strassman, S Hairy Brushes, Computer Graphics, proc SIGGRAPH '86 20(4) pp225 - 232, August 1986

modelling the texture of paint

Documents