concept-based retrieval of art documentsmarissac/conceptpaper/new... · concept-based retrieval of...

E. M. Bakker et al. (Eds.): CIVR 2003, LNCS 2728, pp. 383-393, 2003. Springer-Verlag Berlin Heidelberg 2003

Concept-Based Retrieval of Art Documents

Jose A. Lay1 and Ling Guan2

1 University of Sydney, Electrical and Information Engineering,Sydney, NSW 2006, [email protected]

2 Ryerson University, Electrical and Computer Engineering,Toronto, Ontario M5B-2K3, Canada

[email protected]

Abstract. This paper presents our work on the retrieval of art docu-ments for color artistry concepts. First we show that the query-by-example paradigm popularly used in content-based retrieval can sup-port only limited queryability. The paper then proposes a concept-basedretrieval engine based on the generative grammar of elecepts methodol-ogy. In the latter, the language by which color artistry concepts arecommunicated in art documents is used to operate the retrieval proc-esses. The concept language is explicated into a lexicon of elecepts andthe associated generative grammar. Documents are then indexed withelecept indices, while the generative grammar is used to facilitate thequery operation. More extensive color artistry concept queries can thenbe supported by post-coordination of the elecept indices.

1 Introduction

Leveraging content-based indexing techniques for the retrieval of art documents isboth appealing and challenging. It is appealing as the techniques hold potentials todiscern rich content information of an art document just as full-text indexing unveilskeywords from a text document. It is also challenging, as evident by the fact that intothe third decade of the content-based retrieval (CBR) practice, search tools at muse-ums and art galleries across the world remain mostly text-based.

Two representative systems of CBR concerned with art documents are the QBIC atthe State Hermitage Museum [1] and PICASSO [2]. QBIC is used at the site to sup-port two types of syntactic search: (1) the dominant color search where a user speci-fies one or more colors to search for artworks matching the color specifications; (2)the color layout search where a user specifies an arrangement of color areas to searchfor artworks matching the spatial color structure of the example query. PICASSO, onthe other hand, also supports semantic queries on contrast and harmony based on theinter-region color relationships. Typical queries supported by the system are: (1) tworegions of certain sizes are sketched and the property of contrasting luminance is

384 Jose A. Lay and Ling Guan

selected. This example query searches for paintings of contrasting luminance; (2) tworegions of certain sizes are sketched, one region is filled with color green and theproperties of hue and contrasting warmth are selected. This example query is used tosearch for paintings showing contrasting warmth where one of the two regions isgreen in color; (3) three uncolored regions of certain sizes are sketched and the prop-erty of harmony is selected. The latest query is used to search for painting matching aternary accordance such as paintings of blue, orange, and green regions.

We construe that a key issue impeding the wider applicability of CBR on artdocuments lies with the inflexibility of the Query-by-Example (QBE) paradigm.Intuitively, the retrieval of perceptually similar documents by example queries isuseful for finding variants of art documents for which examples are available at hand.Beyond this functionality, the usefulness of QBE appears to be ill-fated, as posingvisual example queries is difficult to be operationalized. For instance to pose an ex-ample query to retrieve paintings of primary triadic color scheme which are brilliantbut must not contain the complementary pairs of purple is at best a very tedious op-eration. Consequently, access to color artistry concepts has remained restricted tointellectually indexed entities operated by the traditional cataloging practice.

This paper presents our work on operationalizing a concept-based retrieval enginefor color artistry concepts based on the generative grammar of elecepts (G2E) meth-odology [9]. We demonstrate that once the language by which color artistry conceptsare communicated in art documents is identified; retrieval for color artistry conceptscan be operationalized by using the post-coordination indexing scheme where morequeries can be supported through post-coordination of the index terms.

2 Color Artistry Concepts

Color artistry deals with the artful skill to communicate thought and to render per-ceptual experience with colors. In this section, we briefly present the color opponenttheory of Ewald Hering [3] and the color harmony schemes of Faber Birren [5].

Color Opponent Theory

In the last decade of the 19th century, Ewald Hering noted that trichromatic humanvision based on long-, medium-, and short-wavelength (LMS or RGB) cones com-monly referred to as the Young-Helmholtz theory is inadequate for explaining humanperceptual experience. Hering observed that yellow is as elementary as red or greenand mixtures of red-green and yellow-blue are inconceivable, for there is not a red-dish-green or a yellowish-blue. He also noted that the effect of chromatic contrastdoes not seem to apply to achromatic black and white, which blend to produce arange of gray. Thus he added yellow to the trichromatic RGB to introduce RYGB asperceptual primaries and proposed luminosity as an independent process to form whatis currently known as the three color-opponent pairs of light-dark, red-green, andyellow-blue [3]. Hering then went on to devise a color-opponent wheel. On thewheel, perceptual primaries RYGB are arranged as color complements: red opposedto green, and yellow contrasted with blue.

Concept-Based Retrieval of Art Documents 385

Fig. 1. Color Harmony Schemes. [a] Monochromatic. [b] Analogous. [c] Complementary.[d] X-complementary. [e] Triadic. [f] Quad

Color Harmony Schemes

In Arts, a number of color schemes are known for exhibiting certain effects on theiruse. These color schemes are often described as principles for attaining color har-mony [5]. An illustration of a few color harmony schemes is given in Figure 1.

The monochromatic scheme deals with the harmony of a hue. It is instantiated byapplying shades (adding black), tones (adding gray) and tints (adding white) on asingle hue. Tones can also be introduced by varying saturation of a color. On theother hand, the analogous scheme is concerned with the harmony of similar hues. It isnormally created by mixing no more than three adjoining colors on a 12-color wheel.Typically one of the three colors is used more predominantly than the others. Next thecomplementary scheme relates to the equilibrium harmony of complementary colors.In the simplest form, it is created by using two colors opposite to each other on thewheel. Alternatively, double- and split- complementary can be used. Two hues adja-cent to each other can be coordinated with their respective complements to form adouble complementary pairs of an X structure, while the split complementary is cre-ated by pairing a hue with either sides of its complement on a color wheel to form a Ystructure. Color schemes are also commonly communicated by geometric metaphors.For example, the basic complementary is also known as a dyad, the scheme for thethree primary hues is known as primary triadic, etc. Geometric color schemes can beexpanded both in terms of structure (e.g. pentad) and dimensionality (from colorwheel to color sphere).

The monochromatic color scheme is particularly popular in Arts. The school ofImpressionism, for instance, is devoted to the combination of saturated color, tint, andwhite and avoided the darker tones and black. The eight monochromatic colorschemes created through combinations of the tint-tone-shade processes [5] are repli-cated in Figure 2.


[a] [b] [c]

[d] [e] [f]

[g] [h] [i]

Fig. 2. Monochromatic Schemes. [a] Triangle of a color sphere. [b]-[i] Tint-tone-shade colorharmony schemes

3 Generative Grammar of Elecepts

In G2E, an art document is seen as a communication medium whose contents com-prise a set of artistry concepts communicated by a painterly language. The languagecomprises a set of artistry structures constructed out of a lexicon of finite color art-istry elecepts. For the purpose of this work, we can think of colors as words and theperceptual elements of colors as phonemes. It follows that an artistry structure has afinite series of colors (respectively perceptual elements) just as a sentence has a finitesequence of words (respectively phonemes). Furthermore, just as not every sequenceof words constitutes a sentence, not every composition of colors is a well-formedartistry structure. Based on [6], the set of explicit rules used to generate all and onlywell-formed artistry structures is called the generative grammar of color artistry.Figure 3 shows the hierarchical abstraction of the generative concept expression. Acolor artistry concept is seen as a compound concept constructed out of a set of colorartistry elecepts coordinated by certain rules of the color artistry generative grammar.

A fundamental process in G2E is thus to derive the elecepts and generative gram-mar of the color artistry language such that color artistry concept queries can be con-ceived and expressed in terms of the color artistry elecepts and retrieval can be treatedas a process of collocating the set of documents for which the query expression has amodel. The elecepts are used to index the documents, while the generative grammar isused to facilitate the query operation.


Fig. 3. Coordinating elecepts for color artistry concepts

Table 1. Color artistry relevance-concepts

Color Contrast Color Scheme OtherHue Monochromatic Color Names

Light-dark Analogous PaleCold-warm Complementary Vivid

Complementary Triadic ToneSimultaneous Y-complementary Colorfulness

Saturation X-complementary Dull, etc.Extension

In this work, the color artistry concepts (CACs) in [4][5] are used. A summary ofthe CACs introduced in the works is listed in Table 1.

To derive the elecepts and the generative grammar, a generative concept analysis iscarried out. We begin by introducing a concept language LANG for the color artistryconcepts. Principally, an expression in LANG takes the form of a color structure. Theexpression thus can be interpreted at two abstract levels which we shall call thetopological language LANG1 and the compositional language LANG2. The former isconcerned with the geometric properties and the spatial relationship of colors, whilethe latter deals with the perceptual quality and interactivity of colors. Intuitively,CACs are primarily a set of expressions in the compositional language.

Treating CACs as expressions of LANG-2 confers a rather simple lexicon to thecolor artistry language. Clearly, CACs can be factored into certain compositions ofcolors. The elecepts and generative grammar of LANG2 can then be defined by usinga color opponent color model—a color wheel or sphere where contrasting hues arearranged as complements. The lexicon of the elecepts comprises all colors specifiablein that color model, while the generative grammar comprises the various topologicalrelationships needed to represent concepts presented in Table 1.

To offer a formal syntax for LANG2, we use ALCN from the family of DescriptionLogics (DL) introduced in [7]. The basic elements of DL are concepts and roleswhich respectively correspond to classes and binary relations. Concepts and roles areconstructed from elecepts along with some constructors. The syntax of the grammaris given by:


C, D → CE | ¬C | C ⊓ D | C ⊔ D |

∀R.C |∃R.C |∃≥nR.C |∃≤nR.C

where CE denotes elecepts, C and D are color artistry concepts, R represents grammarroles, ≥nR and ≤nR are respectively the at-least and at-most number restrictions wheren ranges over positive integers. The constructors are negation (¬), union (⊔), andintersection (⊓).

Meanwhile to represent the topological relationships on a color model, we intro-duce three operators: adjacent, opposite, and triad. On a color wheel, the adjacents atdistance n of h are the pairs (h1,h2) such that (h1,h2) located on the counter-clock-wiseand clock-wise at distance n from h on the color wheel. The opposite of h is h1 suchthat ∡(h,h1) = 180 degree. The triad of h is a triple (h,h1,h2) such that (h,h1,h2) is aequilateral triangle. Lastly the temperature operator maps the hues on the color wheelinto two sequence of warmth Wi and Ci based on some interpretation of warmth.While on a color sphere, the hue operators are extended to deal with the dimensionsof chroma and lightness. Accordingly three elemental adjacent operations are percep-tible for each color on the surface of the sphere: towards other hues along the equator(hue operation); up towards white and down towards black (lightness operation); andinwards towards grey at the centre and continuingly toward its complementary hue(chroma operation).

By using the syntax and the topological relationship operators, we now definesome concepts and grammar roles:

color ≐ hue ⊓ saturation ⊓ lightnesscolor-name ≐ color-constantwarmth ≐ hue-constantlightness-name ≐ lightness-constantcolorfulness-name ≐ saturation-constantrAnalogous ≐ ∃≥nadjacent.x ⊓ ∃≤nadjacent.x

rMono ≐ ∃≥0adjacent.x ⊓ ∃≤0adjacent.x

rComplement ≐ ∃opposite.x

rTriadic ≐ ∃triad.x

rTone(n) ≐ ∃≥nadjacent.sat.

rTint(n) ≐ ∃≥nadjacent.lightness

rShade(n) ≐ ∃≤nadjacent.lightness

rY-complement(n) ≐ ∃≥nadjacent.(complement.x) ⊓ ∃≤nadjacent.(complement.x)

rWarmer(n) ≐ ∃≥ntemperature

Subsequently, arbitrary CACs can be written as expression in LANG2, for example:

(1) Purple ⊓ rTriadic.hue(2) Purplish-red ⊓ rComplement.hue ⊓

¬ Analogous.sat

(1) represents the triadic scheme for hue purple; and (2) stands for the hue comple-ment of color purplish-red which must not have analogous saturation.


The query operation thus can be seen as a satisfiability check. Given an interpreta-tion I a pair of (∆I,•I) where ∆I is the domain of the interpretation and •I is aninterpretation function which assigns to every concept A a set AI ⊆∆I and everygrammar role R a binary relation RI ⊆∆I x ∆I. The interpretation function of othercompound concepts is defined inductively by the following rules: ∆

⊤I = CI ∪¬CI = ∆I

⊥I = CI ∩¬CI = Ø(¬C)I = ∆I \ CI

(C⊓D)I = CI ∩ DI

(C⊔D)I = CI ∪ DI

(∃R.C)I = {a∈∆I |∃b.(a,b)∈RI ∧ b∈ CI}(≥n R)I = {a∈∆I | ∣{b∣(a,b)∈RI }|≥ n}(≤n R)I = {a∈∆I | ∣{b∣(a,b)∈RI }|≤ n}

Thus a query of concept C is satisfiable with respect to the grammar G (G C) ifthere exists a model I of G such that CI is a nonempty set. It follows that two con-cepts C and D are equivalent (or disjoint) with respect to G, if CI=DI (or CI∩DI = Ø)

for every model I of G.

4 Retrieving Color Artistry Concepts

To represent the elecepts, the CIE-LAB color system is used. The choice is judicious,as CIE-LAB is a color-opponent model. The long axis of CIELAB represents light-ness (L), while the other two color axes are based on Hering's color opponent pairs ofred-green (A: a,-a) and yellow-blue (B: b,-b). However, as CACs are conceived interms of hue, saturation, and lightness; the LAB coordinates are impractical and willneed to be transformed. In this work, the normalized color histograms of lightness(L), chroma/saturation (C), and hue (H) of CIE-LCH are used. The latter is derived bytransforming the CIE-LAB cube into polar coordinates:

)arctan(;; 22

abHbaCLL =+==

The lexicon of LANG-2 thus comprises all colors in the LCH sphere, while its gen-erative grammar is defined by the extended adjacent, opposite, triad, and the tem-perature operators on the LCH color sphere. However, color is hardly perceived insuch a granularity for general artistry use. Thus, a coarser color specification is oftendesirable. In this work, we define a color cube by introducing a tolerance block. Acolor cube of a color is obtained by extending the color point with ∆L, ∆C, and ∆Hdefined by the tolerance block. Meanwhile, natural language color names are sup-ported by mapping of the ISCC-NBS color system [8].

We now demonstrate a naïve retrieval mechanism by using the relational datamodel. The implementation on the latter is often desirable as a large number of art


document collections are maintained as relational database systems. To operate theretrieval as set theoretic operations of relational database, elecepts of hue, chroma,and lightness are treated as relations. The database schema are (hue) = {H1, H2, …,H360}, (chroma) = {C1, C2, …, C60}, and (lightness) = {L1, L2, …, L100}. The lexiconof LANG-2 is thus the Cartesian products of the domain of the three elemental con-cept relations: dom{H}x dom{C}x dom{L} where the domains of {L},{C},{H}range over 0 to 100 for each attribute (histogram bin) representing normalized per-centage distribution over the elecepts.

As elecepts of an artwork is indexed by the LCH tuples, collocating for a conceptquery is essentially a two-step process of expanding the LANG-2 expression intorelations of elecepts and a satisfiability check for its model in the database. The ex-pression expansion is carried out by using the generative grammar, while the satisfi-ability check utilizes the projection, selection, join, difference, and other set theoreticoperators in the relational data model. For instance, the concept of lightness may bedifferentiated as {very-light, light, normal, dark, and very-dark}, by the grammar;these concepts are expanded as a function over the tuple {L}. Each concept is thenoperationalized by a projection onto the set of attributes in {L}. Suppose the percep-tuality over the set of attributes of {L} is evenly distributed, then t[very-light]=<t(L81), t(L82), …, t(L100)> where t[very-light] is the projection of the eleceptonto the set of attributes in the relation schema of (lightness). Thus the queryQ1: green ⊓ rComplementary.hue can be operationalized as the selection over:

1: . ( )( ) ( )green red

green rComplementary hue greenhue hueπ π

< >≡ ∩

Q

G GQ Q

where G is the generative grammar of the LANG-2 and Q is the operational satisfi-ability qualifier for the concepts in the relational data model. In practice, Q can bedefined based on some heuristic or by adaptive learning. In the simplest form, Q mayjust be defined as a threshold function.

Intuitively, other query models are also supported. For example, a system based oninverted elecept indices can be devised. In the latter, each elecept index is treated as aterm. Then for each elecept term ti, a posting (dj, wji) is created to point to all docu-ments dj for which the term ti has a normalized histogram weight wji.

In a search operation, the query is again treated as a two-step process. First it isexpanded into a relation of elecept terms along with their qualifiers. Then each asso-ciated term vector in the inverted elecept indices is evaluated, qualified documents arethen retrieved. Relationships such as triadic AND warm are supported by intersecting


the associated document sets. The use of inverted elecept index is operationally ad-vantageous. Its efficiency has been well exemplified by the capacity of online searchengines to support billions of web pages.

5 Retrieval Examples

Figure 4 shows several retrieval examples of the SoloArt1. The examples were basedupon a database comprising 40,000+ images from the Corel Photo Collection. Figure4[a] shows the retrieval of the simple analogous scheme for the hue green. The searchwas interpreted as a projection onto the adjacent range of the hue green and thensorted by decreasing cumulative hue containments. The sort operation resembles thesimilarity ranking of content-based retrieval. Figure 4[b] and Figure 4[c] respectivelyshow the retrieval for analogous color scheme with dominant warmth and the re-trieval for highly saturated color of blue or red hues. In LANG-2, a structured querycan be posed by combining various color artistry concepts with Boolean operators.Structured queries are a major advantage of the G2E methodology. Boolean operatorsallow complex concepts to be expressed intuitively, while the search can be supportednaturally by using relational data model or inverted elecept indices. Meanwhile byusing the few operators defined in the generative grammar, artists can instantiate,experiment, and store various color artistry concepts. Figure 4[d] demonstrate theretrieval example in supporting the customization of color artistry concepts whereimages constituting models for the major triadic harmony scheme are retrieved. Theformulation of well-known concepts is a convenience for novice users. However,artists often contemplate with new ideas and may seek to retrieve examples on them.The latter in turn highlights another salient feature of the G2E methodology. Userscan make use of the elecepts and the generative grammar to build customized con-cepts and when appropriate saving them as personal concepts.

6 Conclusions

In this paper, we presented a new treatment for the retrieval of art documents by colorartistry concepts. The color artistry language was identified and explicated into alexicon of elecepts and the associated generative grammar. Documents are then in-dexed with the elecept indices, while the generative grammar is used to facilitate thequery operation. As elecept indices and generative grammar are rendered accessible,more extensive queryability is operationalized without the need to devise a visualexample query.

1 A collaborative project supported by the Art Gallery of NSW Australia.


[a] Analogous of green hues [b] Analogous color scheme with

dominant warm

[c] Bright saturated colors of Blue or Red [d] Major triadic harmony scheme

of the RYB primaries

Fig. 4. Retrieval examples of the SoloArt

References

[1] IBM QBIC Online at the State Hermitage Museum:http://www.hermitagemuseum.org/.

[2] J. M. Corridoni, A. del Bimbo, and P. Pala, Retrieval in Paintings using Effectsinduced by Color Features, IEEE Multimedia, Vol.6, No.3, July-September1999, pp.38-53.

[3] E. Hering, Outlines of a Theory of the Light Sense, translated by L. Hurvichand D.H. Jameson, Harvard University Press, Cambridge MA, 1964.

[4] J. Itten, The Art of Color: the Subjective Experience and Objective Rationale ofColor, translated edition by Ernst van Haagen, Reinhold Publishing, NewYork, 1961.


[5] F. Birren, Principles of Color – A review of past traditions and modern theoriesof color harmony, VN Reinhold, New York, 1969.

[6] N. Chomsky, Aspects of the Theory of Syntax, M.I.T. Press, Cambridge, 1965.[7] M. Schmidt-Schauß and G. Smolka, Attributive Concept Description with

Complements, Artificial Intelligent, 48, 1, 1991, pp. 1-26.[8] K.L. Kelly and D.B. Judd, Color: Universal Language and Dictionary of

Names, National Bureau of Standards, 1976.[9] J.A Lay, Concept-based Retrieval of Images and Audiovisual Documents,

Ph.D. Thesis, University of Sydney, 2003.