Alves

Bill AlvesHarvey Mudd College301 E. 12th St.Claremont, California 91711alves@hmc.edu

Digital Harmony of Soundand Light

45

In the late 1980s and early 1990s I was privileged towork with the computer animation pioneer JohnWhitney Sr. and was profoundly influenced by hisideas on how to apply musical concepts of harmonyto visual arts of motion. At the time of his death in1995, he and I were planning composition softwarein which an artist could apply these concepts to cre-ate harmonic patterns simultaneously in sound andanimation.

Though this idea was never realized beyond cer-tain tests, I have taken this step in my subsequentwork in computer-generated music and video inways inspired by, if distinct from, Whitney’s earlyartworks. This article examines ways in whichWhitney’s ideas can be applied to musical composi-tion, and in particular to ways in which I have ex-trapolated principles from these ideas to create anartistic correspondence between abstract animationand computer music.

Visual Harmony

We use the word harmony today to refer not just tothe vertical dimension of music, but also a generalsense of agreement or peace—the original meaningof harmonia to the ancient Greeks (Franklin 2002).Pythagoras was credited with defining this equilib-rium in mathematical terms, as whole number pro-portions that represented an ideal not just in musicalscales, but in sculpture and architecture. Indeed thesame mathematical proportions formed the veryfabric of Platonic universe (Cornford 1937).

This close connection between music and visualart remained a vital force in European and Islamicart well into the Renaissance period. One does nothave to subscribe to Pythagorean numerology tofeel the arresting impact of the pure and crystallineharmonies created by musical tunings based on in-teger ratios, or Just intonation, as well as the beau-ties of proportional symmetries in visual design.

The sounds and compositional resources availablein Just intonation first attracted me as a composer.

Though the Pythagorean tradition receded withthe adoption of musical temperament and othercompositional structures for visual arts and archi-tecture, Sir Isaac Newton and others proposed aconnection of sound and vision through the waveproperties of color in light and pitch in music (Jones1972). Arguably the first efforts to use technology toapply concepts of musical composition to abstractarts of motion came with a series of experimentaldevices from the 18th through the 20th centuriesgenerally called “color organs.” Most famously, thecomposer Aleksandr Skryabin’s Prométhée includesa part for such an instrument which would projectthe colors of Skryabin’s synesthetic associations.However, the technology these innovators used al-lowed only poor and vague definition of visual form(Collopy 2000).

The technology of filmed animation offered muchgreater control over form and direct synchroniza-tion to music. The great abstract animator OskarFischinger created a remarkable series of films inthe first half of the twentieth century in which herepresented the sounds of accompanying music of-ten with delicately fluid forms (Moritz 2004). Manyof his films are a kind of supple choreography dis-tilled to abstraction, but relying on an intuitivesense of connection to musical form.

Mapping Vision to Sound

Distinct approaches to the correspondences be-tween sound and light are at least as important asdifferences in technique in distinguishing the workof artists in this tradition. Many artists, from the18th-century inventor of a “color harpsichord,”Louis-Bertrand Castel, to modern creators of musicvisualizer software, have attempted to directly illus-trate music, mapping pitch to spatial height or colorhue, for example (Klein 1937). The Music Anima-tion Machine (www.musanim.com/) of Stephen Ma-linowski is especially interesting in this approach.

Computer Music Journal, 29:4, pp. 45–54, Winter 2005© 2005 Massachusetts Institute of Technology.

However, though artists often speak of “color har-mony,” attempts to create a direct mapping of colorrelationships to the immediacy of musical conso-nance and dissonance have largely failed.

To Whitney, such a direct, synesthetic mapping ofmusic’s most basic parameters (pitch, loudness, andso forth) failed to capture the expressive vision ofgreat works of music, which, to him, depended moredirectly on multidimensional interplay of tensionand resolution. Moreover, he advocated an approachin which animation, instead of being a direct repre-sentation of music, corresponds to this higher levelof aesthetic intention, creating what he termed“complementarity” (Whitney 1984).

Whitney’s Differential Dynamics

John Whitney recognized that digital computerscould uniquely and directly realize in animatedform the same kind of harmonic movement foundin music in ways never imagined by the Greeks. Be-ginning in the 1960s, Whitney created a series of ex-traordinary films of abstract animation that usedcomputers to create a harmony not of color, space,or musical intervals, but of motion.

In his 1980 book Digital Harmony: On the Com-plementarity of Music and Visual Art, Whitney hy-pothesized that Pythagorean harmony could bematched in visual art: “This hypothesis assumesthe existence of a new foundation for a new art. Itassumes a broader context in which Pythagoreanlaws of harmony operate. . . . In other words, the hy-pothesis assumes that the attractive and repulsiveforces of harmony’s consonant/dissonant patternsfunction outside the dominion of music” (p. 5).More particularly, Whitney discovered that if he seta large number of elements into repetitive motionsuch that the motion of the second was two timesthe speed of the first, the third three times the speedof the first, and so on, the animation that would re-sult would demonstrate beautiful patterns of sym-metry at points corresponding to the same ratiosthat define musical consonances.

(Tenney [1988] reminds us that the terms conso-nance and dissonance have been used in many dif-

ferent senses throughout history. In particular, wemust be careful to distinguish between consonanceas a psychoacoustical phenomenon, which may de-pend in large part on the closeness of the frequencyproportions to ratios of relatively small whole num-bers, my focus here, and consonance in a more gen-eral sense as harmonic stasis or resolution, whichmay be created in many different ways.)

Though Whitney’s films brilliantly demonstratehow computer animation can effectively create atemporal sensation of attraction and repulsion, oftension and resolution, he was never able to takethe next step of creating music to correspond di-rectly to those visual images.

To take a very simple example of what Whitneycalled “differential dynamics,” imagine a set ofpoints going around a circle, the second travelingtwice the speed of the first, the third three times thespeed of the first, and so on, all starting together atthe twelve o’clock position. By the time the slowestpoint reaches halfway around the circle, all the pointswill align with either the six or twelve o’clock posi-tion. If the slowest point is at either the one-third ortwo-thirds position, all points will line up at thethirds: twelve, four, or eight o’clock. When the slow-est point is at a position around the circle which isnot a very clear integer divisor of the circle’s perime-ter, the points will appear scattered somewhat ran-domly, and the eye will not detect a clear pattern.

Whitney’s Works

Of course points moving around a circle is a verysimple example. In his films Matrix I and III (1970and 1972), Whitney had points or other shapesmove around parametric curves. In Permutations(1968), the points move in various rose curves (sinefunctions in polar coordinates), and in Arabesque(1973) points initially arranged in a circle movelinearly but wrap around at the edge of the screen(see Figure 1). In all of these examples, the samepoints of resonance, of striking symmetry, occur atthe same points of harmonic proportions.

Each of these early films, created by photograph-ing frames from a CRT monitor with an animation

46 Computer Music Journal

Alves

camera and then combining them and adding colorwith an optical printer, is an intricate compositionarranged in sections in which various themes aredeveloped. Most distinctively, these films createform through the manipulation of expectation, reso-lution, and surprise, mostly achieved through thetechnique of differential dynamics. However, withthe exception of Matrix I, Whitney avoided sound-tracks with conventional tonal harmonic progres-

sions, whose sequences of harmonic tension andresolution might conflict with those he was pre-senting visually.

An effort to move to an interactive compositionalsystem led him to create works of very low colorand spatial resolution on an IBM PC in the 1980s.Although not nearly as immediately attractive ashis earlier works, these pieces, collected as MoonDrum (1991), allowed him to develop works of com-positional complexity and nuance. With this soft-ware, developed by Jerry Reed, he was also able tosend out MIDI messages at keyframes, creating hisown soundtrack of non-metrical FM tones anddrum machine crashes. Whereas this approach al-lowed for musical articulation of important pointsof the animation, it hardly demonstrated the possi-bilities of a true complementarity, to use Whitney’sterm, of visual and musical composition.

Differential Dynamics in Music

In the composing software that Whitney and Iplanned, he envisioned multitudes of sine wavesrising and falling in correspondence to the spatialmotion of the elements on the screen. However, Isoon realized that a literal mapping of pitch space toheight on the screen was only intuitive because ourculture has adopted that particular arbitrary meta-phor of “low” and “high” to describe pitch. Thoughother pioneers of connections between music andthe visual arts have explored such a mapping, itdoes not retain the possibility of musical conso-nance resulting in visually “consonant” patterns.

The fact that whole number proportions createthese arresting patterns of visual resonance suggesta correspondence, or complementarity, to conso-nant musical sonorities created by whole numberfrequency ratios, that is, Just intonation. Seeing thatJust musical intervals occur naturally in the har-monic series, many Just intonation composers, suchas La Monte Young, have focused on the composi-tional resources of frequency ratios related implic-itly to a common fundamental, a relationshipPartch (1974) terms an otonality.

However, the proportions of Whitney’s differential

47

Figure 1. Images from JohnWhitney’s 1973 filmArabesque. Images repro-duced with the permissionof the Estate of John andJames Whitney.

48 Computer Music Journal

dynamics suggest a different kind of relationship.In the example of points moving around a circle,when the first point reaches the one-third point inthe circle, all the points are aligned to the 12:00,4:00, and 8:00 positions. By treating the 12:00 posi-tion as the fundamental, the one-third point wouldrepresent a 3/1 frequency ratio and the two-thirdspoint the 3/2 frequency ratio. (Just intonation ratiosare conventionally notated with the larger numberin the numerator. I have adopted this convention forratios of both frequencies and visual proportions, asin this example.)

Similar patterns would occur at other whole-number divisions, but with increasing complexityas that whole number increases. Thus a five-fold di-vision of the circle would correspond to frequencyratios of 5/5 (that is, 1/1, the fundamental), 5/4, 5/3,5/2, and 5/1. Unlike the harmonic series, this set ofmusical intervals has a constant factor in the nu-merator. This set defines what Partch called anutonality and what others have called a subhar-monic series (that is, the inversion of the harmonicseries). Subharmonic series occur with combina-tions of any integer divisions of a whole, as with di-visions of string lengths or evenly spaced holes onaerophones, for example.

The higher the numerator, or number of divi-sions, what Partch calls the numerary nexus, thefurther along the spectrum we are towards disso-nance. The smaller this common factor or numer-ary nexus, the more stable but less dynamic thesound and image. Thus the frequency ratio of 2:1 isthe octave, which, as with the differential dynamicsexample, results in a profoundly stable but not veryinteresting consonance. Somewhat more complexfactors such as 5 or 7, which offer more interestingmusical consonances, result in fascinating symmet-rical patterns that catch the eye’s attention whenapplied the differential dynamics system.

Figure 2 demonstrates examples of these points ofconsonance, first in the simple example of pointsmoving in a circle, then in a more complex rose-curve pattern, and finally as a set of tones in thesame proportions. The number in the first columnrepresents the proportion that the slowest elementhas traversed through the entire cycle, and hencethe common factor or numerary nexus of all the ele-

ments. At points where this number represents a ra-tio of relatively small whole numbers, symmetricalpatterns emerge.

The right-hand column of Figure 2 shows thechords that would result if a set of sixteen pitcheswent through the same differential cycle. The be-ginning position (12:00 in circular motion), the fun-damental, is here set to C2. One quarter through thecycle, for example, would represent a 4/1 ratio, or afrequency four times C2. To represent some of thepitches poorly approximated by twelve-tone nota-tion, I have used an accidental of an arrow up ordown to represent a pitch inflection of about 33cents and a plus sign to indicate raising the pitchabout 15 cents. The final row is an example of howboth visual and aural dissonance results when thecommon factor is irrational or sufficiently complex.

Artists such as Ronald Pellegrino have discoveredsimilar points of correspondence between Just into-nation and visual symmetry through the use of ana-log electronics, especially the oscilloscope (Pellegrino1983). When two tones of relatively simple wholenumber frequency relationships are connected tothe x and y inputs of an oscilloscope (or the vibrat-ing mirrors of a laser scanner), the resulting visualform will be a Lissajou figure of relative stabilityand symmetry. The algorithms of Monro and Press-ing (1998) demonstrate similar relationships betweenJust intonation harmonies and visual symmetries inmore extensible and elaborate ways. However,Whitney intended differential dynamics to provide aset of principles which could be applied composi-tionally in many different forms, rather than algo-rithms for visualization.

New Explorations

In my first video based on these principles, Hiway70 (1997), I extended the polar coordinate curves ofWhitney’s Permutations to three-dimensionalgraphics. But the most important way in which mywork was distinguished from his is that, approach-ing this work as a composer, I created a soundtrackin tandem with the visual composition, carefullysynchronizing movement between points of tensionand dissonance and points of stability and tonal

Differential dynamics Differential dynamicsCommon example—spheres moving example—spheres moving Musical correspondencefactor in a circle in a rose curve pattern pitches are approximate

1

2

3

4

5

Figure 2. Correspondencesbetween points of visualresonance in two examplesof differential motion andmusical pitches (see text).

continued

50 Computer Music Journal

Differential dynamics Differential dynamicsCommon example—spheres moving example—spheres moving Musical correspondencefactor in a circle in a rose curve pattern pitches are approximate

6

7

8

9

10

8.45

Alves

consonance. I created the music entirely in Just in-tonation, using harmonies which were often directanalogues of the patterns of visual symmetry (seeFigures 3 and 4).

Just as digital technology allowed me to controlthe visual elements with precision necessary for dif-ferential dynamics, so did Csound realization of themusic allow me to create a Just intonation systemwhich could freely modulate between tonal centers.(The Csound code for Hiway 70 is on the CD-ROMaccompanying Boulanger 2000. See also Alves 2000in the same volume.) I mostly used POV-ray to real-ize the images. Csound and POV-ray are comparablein that they are languages freely available for a vari-ety of platforms that allow the power and speci-ficity to implement my vision of differentialdynamics and dynamic Just intonation, amongmany other advantages, of course.

Static Cling (2000) explores the ways in whichharmonic patterns can emerge from visual or audi-tory chaos. The video begins with television staticaccompanied by multiple tracks of overlappingchatter of news anchors resynthesized through lin-ear prediction and then driven with noise. Gradu-ally the noisy chatter is resonated into specificharmonic frequencies emphasizing prime relation-ships of 7 and 11 while the random fluctuations ofthe video static align themselves into particularshapes and grids.

The principles of differential motion can also bereflected the rhythmic dimension of the music. InStatic Cling sets of vertical lines flash in correspon-

dence to several layers of repetitive polyrhythmicpatterns. Each set of lines moves in harmonic pro-portion with the others, so that they line up at met-rically important points. The lines and other shapeswhich follow in this work move in a wider varietyof curve types than in previous works, but the prin-ciple is the same. The elements line up into sym-metrical patterns at points of stability in thecomposition, reflecting the simultaneously conso-nant Just ratios in the music (see Figures 5 and 6).

aleph

In my next work, aleph (2002), I explored the waysin which Whitney’s principles of patterned motioncreating and resolving expectation could be extrapo-lated to approaches sometimes not involving differ-ential dynamics at all. I was inspired by ancientIslamic arts of geometric abstraction that find theirhigh point in the intricate tessellations of the Al-hambra in Spain or the Isfahan mosque in Iran, ex-quisite frozen examples of Whitney’s points ofvisual resonance. In aleph, I “unfroze” those elabo-rate patterns so that they could reform into newones at points of tonal stability in the accompany-ing music.

The great philosophical schools of medieval Bagh-dad and elsewhere in the Islamic world deeply feltthe influence of Pythagoreanism and Platonism, at-testing to the power of number in music, art, callig-raphy, and religious symbolism. Critchlow (1976)

51

Figure 3. Still fromHiway 70.

Figure 4. Still fromHiway 70.

showed how Islamic art is built up through patternsof number, starting with the monad, unity, whichsymbolically recapitulates creation when it is ex-tended into two dimensions into the circle, the tes-sellating polygons of the triangle, square, andhexagon, and finally more ornate manifestations.He relates this extension of unity to the first letterof the Arabic alphabet, aleph, as the “creative raywhich initiates existence at the diacritical point ofbey [the second character] and thence proceeds toexpand horizontally with the lateral gesture of thesecond character. By this the fundamental three-foldnature of reality is established—the descent of thelight, the expansion into creation and (in the sym-bolism of the written words of the Quran) the meanswhereby the ‘light’ returns to its source” (p. 8).

Rather than relying on harmonics or subharmon-ics to form the basis for the Just intonation, I in-stead turned to structures known as “hexanies” toestablish points of musical repose and stasis in thesoundtrack. Hexanies, a type of “combination prod-uct set” invented by the visionary tuning theoristErvin Wilson, are sets of six pitches created throughthe various possible pairings of four selected integerfactors (Wilson 1989; Grady 1991). A characteristicof combination product sets that distinguishesthem from traditional approaches to Just intonationis that no one pitch in the set necessarily has prior-ity over the others. Like ever-expanding tessellationpatterns that offer multiple perspectives of what

functions as a center, this approach allows pitchsets to retain the integrity of Just harmonic struc-ture together with the ability to float free.

In the beginning of aleph, a formlessness gradu-ally coalesces into visual and musical patterns. Res-onant drones are eventually replaced with bell-liketones that follow the mathematical patterns ofchange-ringing. In this English folk art, groups ofbell ringers follow strict permutations of the orderof a set of bells, so that each ringer follows a “path”through the successive orders that I envision ratherlike the paths that the eye may traverse in followingthe interlocking weaves of Islamic design. In myversion other attributes such as octave, density, andpitch sets gradually shift together with the changesof bells. The frequencies of the bell partials aretuned to the same pitches in the hexany then in use,so that the timbres will retain a consonant relation-ship in the context of the pitch sets (Sethares 1999).

By the time the visual patterns coalesce intostriking symmetries, so too do the pitch sets arriveat stable hexanies (see Figures 7 and 8). In betweenpitches change gradually from one set to the next,reflecting the visual disorder as one pattern slowlytransforms into the next. Again, the flexibility ofthe Csound language enabled me to freely shift be-tween pitch sets without having to establish asingle fixed gamut of pitches. (This approach con-trasts with that of some other composers who preferto explore the structure of a single combination

52 Computer Music Journal

Figure 5. Still from StaticCling.

Figure 6. Still from StaticCling.

Alves

product set within a particular composition. Amongthe composers who have explored combinationproduct sets in their works are Warren Burt, KraigGrady, Greg Schiemer, and Stephen James Taylor.)

Future Harmony

John Whitney recognized that his work representedthe infancy of an art form, but suggested that,“Composers will discover a congruence of aural-visual partnership as productive as that which theyfound for centuries in writing for combinations of

all kinds. . . . That partnership will be grounded onvalid harmonic interrelationships equally applicableto sound and image” (Whitney 1980, p. 18). Even ashe predicted the digital computer and video tech-nology would launch a new golden age of artisticpossibilities, he remained convinced that the suc-cess of a new art of visual motion would depend onits basis in fundamental principles extending fromthe canon of Pythagoras to the artful play of tensionand resolution in European musical harmonies.

This article illustrates some new possibilitiesthat creatively apply Whitney’s ideas as a founda-tion to achieve a complementarity of music andvisual art in motion. Within the psychologicallypowerful but indefinitely flexible applications ofthese general principles lie boundless opportunitiesfor new visions of a fluid architecture of form, mo-tion, and music.

References

Alves, Bill. 2000. “Using Alternate Tunings in Csound.”In CD-ROM accompanying Boulanger, Richard, ed. TheCsound Book. Cambridge, Massachusetts: MIT Univer-sity Press.

Boulanger, Richard, ed. 2000. The Csound Book. Cam-bridge, Massachusetts: MIT University Press.

Collopy, Fred. 2000. “Color, Form, and Motion: Dimen-sions of a Musical Art of Light.” Leonardo 33(5):355–360.

Cornford, Francis Macdonald. 1937. Plato’s Cosmology.London: K. Paul, Trench, Trubner & Co.

Critchlow, Keith. 1976. Islamic Patterns: An Analyticaland Cosmological Approach. London: Thames &Hudson.

Franklin, John Curtis. 2002. “Harmony in Greek andIndo-Iranian Cosmology.” The Journal of Indo-European Studies 30(1/2):1–25.

Grady, Kraig. 1991. “Ervin Wilson’s Hexanies.” 1/1: Jour-nal of the Just Intonation Network 7(1):8–15.

Jones, Tom Douglas. 1972. The Art of Light and Color.New York: Van Nostrand Reinhold.

Klein, Adrian Bernard. 1937. Coloured Light: An ArtMedium. London: The Technical Press.

Monro, G., and J. Pressing. 1998. “Sound VisualizationUsing Embedding: The Art and Science of AuditoryAutocorrelation.” Computer Music Journal 22(2):20–34.

53

Figure 7

Figure 8

Figure 7. Still from aleph. Figure 8. Still from aleph.

Moritz, William. 2004. Optical Poetry: The Life andWork of Oskar Fischinger. Bloomington: Indiana Uni-versity Press.

Partch, Harry. 1974. Genesis of a Music. 2nd ed. NewYork: Da Capo.

Pellegrino, Ronald. 1983. The Electronic Arts of Light andSound. New York: Van Nostrand Reinhold.

Sethares, William A. 1999. Tuning, Timbre, Spectrum,Scale. London: Springer Verlag.

Tenney, James. 1988. A History of Consonance and Dis-sonance. New York: Excelsior.

Whitney, John. 1980. Digital Harmony: On the Comple-mentarity of Music and Visual Art. New York: ByteBooks.

Whitney, John. 1984. “To Paint on Water: The Audiovi-sual Duet of Complementarity.” Computer Music Jour-nal 18(3):44–52.

Wilson, Ervin. 1989. “D’Alessandro, Like a Hurricane.”Xenharmonikon XII:1–39.

54 Computer Music Journal