son if ication

8/12/2019 Son if Ication

1/34

Java Prog. Techniques for Games. Java Art Chapter 5. Sonification Draft #1 (4th May 09)

1 Andrew Davison 2009

Java Art Chapter 5. Program Sonification

Program sonification (also called auralization) is the transformation of an executingprogram into auditory information. Im not talking about an application playing asound clip, but the entire program becomingthe clip or piece of music. Themotivation for this unusual transformation is the same as for program visualization as a way of better understanding what's happening inside code, to aid with itsdebugging and modification.

Music is inherently structured, hierarchical, and time-based, which suggests that itshould be a good representation for structured and hierarchical code, whose executionis also time-based of course. Music offers many benefits as a notation, being bothmemorable and familiar. Even the simplest melody utilizes numerous attributes, suchas sound location, loudness, pitch, sound quality (timbre), duration, rate of change,

and ordering. These attributes can be variously matched to code attributes, such asdata assignment, iteration and selection, and method calls and returns. Moving beyonda melody into more complex musical forms, lets us match recurring themes,orchestration, and multiple voices to programming ideas such as recursion, codereuse, and concurrency.

A drawback of music is the difficulty of representing quantitative information (e.g.that the integer x has the value 2), although qualitative statements are relatively easyto express (e.g. that the x value is increasing). One solution is lyrics: spoken (or sung)words to convey concrete details.

Ill be implementing program sonification using the tracer ideas discussed in the lasttwo chapters (i.e. employing the Java Platform Debugger Architecture (JPDA),specifically its Java Debug Interface (JDI) API). The resulting system is shown inFigure 1.

Figure 1. Sonification of a Java Application.

When a method is called in the monitored application, a word is spoken (anabbreviation of the methods name), when Java keywords are encountered in the code,musical notes are played, and when the program starts, ends, and when a methodreturns, sound clips are heard.

Sound generation is managed by the SoundGen thread, which reads messages from aqueue filled by the tracer. The generator utilizes three sound APIs:


2/34



java.applet.AudioClip for playing clips, the MIDI classes in the Java Sound API forplaying notes, and the FreeTTS speech synthesis system(http://freetts.sourceforge.net/), which is a partial implementation of the Java SpeechAPI 1.0 (JSAPI, http://java.sun.com/products/java-media/speech/).

Ill start this chapter by explaining the three sound subsystems for playing clips,notes, and speaking. These are of general use outside of sonification; for example, theSpeaker class (implemented with FreeTTS) can be used to pronounce any string,using a range of different voices.

The tracer code is not much different from that shown in previous chapters, but doesillustrate two new ideas. Firstly, the queue between the tracer and sound generationthread deals with uneven execution speeds. The tracer generates messages at intervalsof tens of milliseconds, but the generator is constrained to run much more slowly,waiting for each message to be sounded out before moving onto the next.

Secondly, the code shows the advantages of mixing static and dynamic analysis.Static analysis allows the Java keywords in the program text to be mapped to musicalnotes before the program begins execution (making it less costly to run). However,dynamic optimization of the message queue is also needed to reduce the amount of

processing at runtime.

Although this chapter focuses on sonification, the implementation details are sosimilar to those used in the previous chapter on program visualization that it wouldn't

be difficult to combine the two so that the application could be both visualized andheard at the same time.

More Information on Sonification

One of the best known sonification systems is CAITLIN(http://www.auralisation.org), which helps novice Pascal programmers withdebugging. The auralization is based on point-of-interests (POIs), such as if-tests andloops. Each construct is represented by a musical motif, or a short recurring theme.

JListen auralizes Java program to report on events, which are specified by theprogrammer using a rule-based language called LSL(http://www.cs.purdue.edu/homes/apm/listen.html).

CodeSounding is a Java sonification library(http://www.codesounding.org/indexeng.html), which can add sound-generation

methods to "if", "for", and other statements.LYCAY is another Java library for the sonification of code, using a mix of parsingand execution strategies (http://lycay.sourceforge.net/textLetYourCodePlay.html).

The Wikipedia entry on sonification includes a good set of starter references(http://en.wikipedia.org/wiki/Sonification). TheInternational Community for

Auditory Display(ICAD) is a more technical resource on sonification(http://www.icad.org/). Links to sonification-related research can be found athttp://www.dcs.gla.ac.uk/~stephen/otherlinks.shtml andhttp://computing.unn.ac.uk/staff/cgpv1/caitlin/links.htm.


3/34



1. Playing a Clip

Javas AudioClip class offers a high-level means of loading and playing audio clips.It supports a large number of file formats, and multiple AudioClips can be played atthe same time.

A major problem with AudioClip is that it doesn't report when a clip finishes. MySoundGen thread needs this information so that it can wait until the audio has finished

before playing the next sound. There are several hacky work-arounds, such as callingsleep() for a period based on the audio file's byte size (which can be obtained via aFile object).

Another issue is the lack of low-level access to the sound data (or the audio device it'splaying on), to permit run-time effects like volume changing, panning betweenspeakers, and echoing.

The best solution to these problems is to move to the Java Sound API, and utilize thejavax.sound.sampled package for manipulating the audio clips. Unfortunately, this

entails a lot of low-level programming involving sound formats (e.g. ALAW andULAW), and an understanding of frame sizes, rates, and buffering. Once this high

plateau of knowledge has been reached, calculating a clips duration is a (long) one-liner:

doubl e dur at i onI nSecs = cl i p. get Buf f er Si ze( ) /( cl i p. get For mat ( ) . get FrameSi ze( ) *cl i p. get For mat ( ) . get FrameRat e( ) ) ;

For readers interested in this approach, I explain the details in an online chapter called"Chapters 7-10. Sound, Audio Effects, and Music Synthesis" athttp://fivedots.coe.psu.ac.th/~ad/jg/. But in this chapter I'll take a much easier path:

Ill assume that the user supplies the running time of a clip when requesting that it beplayed.

My ClipsPlayer class stores a collection of AudioClip objects in a HashMap whosekeys are their filenames (minus the ".wav" extension).

/ / gl obal s i n t he Cl i psPl ayer cl asspr i vat e f i nal st at i c St r i ng SOUND_DI R = "Sounds/ " ;

/ / di r ectory hol di ng t he cl i pspr i vat e HashMap cl i psMap;

/ / st r i ng i s t he f i l ename ( mi nus . wav) hol di ng t he cl i p

publ i c Cl i psPl ayer ( ){ cl i psMap = new HashMap( ) ; }

When a clip is played, the play() method doesn't return until a specified delay timehas passed.

publ i c bool ean pl ay(St r i ng name, i nt del ay)/ / st ar t pl ayi ng t he cl i p, and wai t f or del ay ms bef or e r et ur ni ng{

i f ( ! cl i psMap. cont ai nsKey( name) )addClip( name) ;

Audi oCl i p cl i p = cl i psMap. get ( name) ;


4/34



i f ( c l i p == nul l ) {System. out . pr i nt l n( "No cl i p f ound f or " + name) ;ret ur n f al se;

}cl i p. pl ay( ) ;wait( del ay) ;

r et ur n t r ue;} / / end of pl ay( )

pr i vat e bool ean addCl i p( St r i ng name)/ / st ore the Audi oCl i p obj ect f or name i n t he hashmap{

Audi oCl i p cl i p = loadClip( name) ;i f ( c l i p != nul l ) {

cl i psMap. put ( name, cl i p) ;System. out . pr i nt l n( "Loaded cl i p f or " + name) ;r et ur n t r ue;

}

retur n f al se;} / / end of addCl i p( )

pr i vat e Audi oCl i p l oadCl i p( St r i ng name)/ / l oad name. wav f r om SOUND_DI R{

St r i ng f nm = SOUND_DI R + name + " . wav";Audi oCl i p cl i p = nul l ;try {

cl i p = Appl et . newAudi oCl i p( get Cl ass( ) . get Resour ce( f nm) ) ;}cat ch ( Except i on e) {

System. out . pr i nt l n( "Coul d not l oad " + f nm) ;}return cl i p;

} / / end of l oadCl i p( )

pr i vat e voi d wai t ( i nt del ay){ t r y {

Thread. sl eep( del ay) ; / / i n ms}cat ch ( I nt er r upt edExcept i on e){ e. pr i nt St ackTr ace( ) ; }

}

ClipsPlayer can be called like so:

Cl i psPl ayer cp = new Cl i psPl ayer ( ) ;cp. pl ay( "wi ndChi me" , 1930) ; / / a wi nd chi me cl i p of 1. 93 secscp. pl ay(" keyDr op", 430) ; / / a key dr oppi ng cl i p of 0. 43 secscp. pl ay( "wi ndChi me" , 1930) ;

When "windchime.wav" and "keyDrop.wav" are first played, they're loaded intoClipsPlayers HashMap. This will delay their playing slightly, but I'm assuming that

the clips are quite short, and so will be quick to load. Another drawback of the class isthe need to supply a clips duration every time it's played. It would be more


5/34



convenient to store it with the AudioClip when it's first loaded, and use thatinformation each time the clip is played.

2. Playing a Note

The simplest way to play a note is to employ a MIDI synthesizer via the Java SoundAPIs javax.sound.midi package.

A Java program can communicate with a synthesizer by sending it a stream ofmessages (as MidiMessage objects). Each message is routed to a particular channel,which represents a different kind of musical instrument. The idea is shown in greatlysimplified form in Figure 2.

Figure 2. A MIDI Synthesizer.

The javax.sound.midi.MidiSystem class provides access to the MIDI resources on amachine. A synthesizer is sent MIDI messages via a receiver port, which is obtainedlike so:

Synt hesi zer synt hesi zer = Mi di Syst em. get Synt hesi zer ( ) ;synt hesi zer . open( ) ;Recei ver r ecei ver = synt hesi zer . get Recei ver ( ) ;

MIDI messages can be encoded using three subclasses of MidiMessage:ShortMessage, SysexMessage, and MetaMessage. The ShortMessage is the mostuseful for my needs, since it includes the NOTE_ON and NOTE_OFF messages forstarting and terminating note playing. The MidiChannel class offers noteOn() andnoteOff() methods for building the messages:

voi d noteOn( i nt not eNumber , i nt vel oci t y) ;voi d not eOf f ( i nt not eNumber , i nt vel oci t y);voi d noteOf f ( i nt not eNumber) ;

A note number is a MIDI number assigned to a musical note, while velocity isequivalent to loudness. A note will keep playing after a noteOn() call, until it's


6/34



terminated with noteOff(). The two-argument form of noteOff() affects how quicklythe note fades away.

Playing a note corresponds to sending a NOTE_ON message, letting it play for awhile, and then killing it with a NOTE_OFF message. This can be wrapped up in a

playNote() method (which Ill be revising a little later on in this chapter):

/ / gl obal spr i vat e st at i c f i nal i nt VOLUME = 90; / / f i xed vol ume f or not es

pr i vat e Recei ver r ecei ver ;

publ i c voi d pl ayNot e( i nt not e, i nt dur at i on, i nt channel )/ / first version; second ver si on gi ven bel ow{

Shor t Message msg = new Short Message( ) ;try {

msg. set Message( Shor t Message. NOTE_ON, channel , note, VOLUME) ;r ecei ver . send( msg, - 1) ; / / - 1 means pl ay i mmedi at el y

wai t ( dur at i on) ;

/ / r euse t he Short Message obj ectmsg. set Message( Shor t Message. NOTE_OFF, channel , note, VOLUME) ;r ecei ver . send( msg, - 1) ;

}cat ch ( I nval i dMi di Dat aExcept i on e){ Syst em. out . pr i nt l n( e) ; }

} / / end of pl ayNot e( )

pr i vat e voi d wai t ( i nt dur at i on){ t r y {

Thread. sl eep( durat i on) ; / / i n ms}cat ch ( I nt er r upt edExcept i on e){ e. pr i nt St ackTr ace( ) ; }

} / / end of wai t ( )

A MIDI message must be sent with a time-stamp. The -1 used above means that themessage should be processed immediately.

The following code fragment plays a round of applause:

f or ( i nt i =0; i < 10; i ++)pl ayNot e( 39, 1000, 9) ; / / note 39 sent t o the dr um channel , 9

Channel 9 plays different percussion and audio effects depending on the note numberssent to it. Note 39 corresponds to a "hand clap". A complete list of the mappings fromMIDI numbers to drum sounds can be found athttp://www.midi.org/techspecs/gm1sound.php.


7/34



2.1. Note Names and Numbers

Its a lot easier (at least for programmers with some musical knowledge), to specify anote using a name derived from the piano keyboard rather than as a MIDI number.

A piano keyboard has a mix of black and white keys, as in Figure 3.

Figure 3. Part of a Piano Keyboard.

Keys are grouped into octaves, each octave consisting of twelve consecutive whiteand black keys. The white keys are labeled with the letters 'A' to 'G' and an octavenumber. For example, the note C4 is the white key closest to the center of thekeyboard, often referred to as "middle C". The '4' means that the key is in the fourthoctave, counting from the left of the keyboard.

A black key is labeled with the letter of the preceding white key and a sharp (#). Forinstance, the black key following C4 is C#4. A note for musicians: for simplicity'ssake, I'll be ignoring flats in this discussion.

Figure 4 shows the keyboard fragment of Figure 3 again, but labeled with note names.I've assumed that the first white key is C4.

Figure 4. Partial Piano Keyboard with Note Names.

Figure 4 utilizes the C Major scale, where the letters appear in the order C, D, E, F, G,A, and B.

After B4, the fifth octave begins, starting with C5 and repeating the same sequence asin the fourth octave. Before C4 is the third octave, which ends with B3.

Having introduced note names, I can now explain their MIDI note numbers. MIDInotes can range between 0 and 127, extending well beyond the pianos scope, whichonly has 88 standard keys. This means that the note naming scheme gets a littlestrange below note 12 (C0), since we have to start talking about octave -1 (e.g. see thetable at http://www.harmony-central.com/MIDI/Doc/table2.html). Additionally, amaximum value of 127 means that note names only go up to G9; there is no G#9.


8/34



Table 1 shows the mapping of MIDI numbers to notes for the 4th octave.

MIDI Number Note Name

60 C4

61 C#4

62 D4

63 D#4

64 E4

65 F4

66 F#4

67 G4

68 G#4

69 A4

70 A#4

71 B4

Table 1. MIDI Numbers and Note Names.

This table can be used as the basis of the method getKey() which converts a notename string (e.g. "C4") into a MIDI note number (60):

/ / gl obal s/ * The note of f set s use t he "C" maj or scal e, whi ch

has t he order "C D E F G A B" , but t he of f set s arest or ed i n t he or der "A B C D E F G" t o si mpl i f y thei rl ookup. */

pr i vat e stati c f i nal i nt [ ] cOf f set s = {9, 11, 0, 2, 4, 5, 7};/ / A B C D E F G

pr i vat e st at i c f i nal i nt C4_KEY = 60;/ / C4 i s t he "C" i n t he 4t h oct ave on a pi ano

pr i vat e st at i c f i nal i nt OCTAVE = 12; / / not e si ze of an oct ave

pr i vat e i nt get Key( St r i ng not eSt r )/ / Conver t a not e st r i ng ( e. g. "C4", "B5#") i nt o a key.{

char [ ] l et t er s = not eSt r . t oChar Ar r ay( ) ;i f ( l et t er s. l engt h < 2) {

Syst em. out . pr i nt l n( "I ncor r ect not e synt ax; usi ng C4") ;r et ur n C4_KEY;

}

/ / l ook at not e l et t er i n l et t er s[0]

i nt c_of f set = 0;i f ( ( l et t er s[0] >= ' A' ) && ( l et t er s[0]


9/34



el seSystem. out . pr i nt l n( "I ncor r ect l et t er : " + l et t er s[0] +

", usi ng C") ;/ / l ook at oct ave number i n l et t er s[ 1]i nt r ange = C4_KEY;i f ( ( l et t er s[1] >= ' 0' ) && ( l et t er s[1] 2) && ( l et t er s[ 2] == ' #' ) )

sharp = 1; / / a shar p i s 1 note hi gher/ / ( r epr esent ed by the bl ack keys on a pi ano)

i nt key = r ange + c_of f set + shar p;/ / Syst em. out . pr i nt l n( "not e: " + not eSt r + "; key: " + key);

r et ur n key;

} / / end of get Key( )

The parsing of the string is simpler if I abuse the note name notation by moving thesharp # to the end of the string. Therefore, the user must write "C4#" rather than thestandard C#4.

playNote() can now be revised to accept a note name:

/ / gl obal/ / pi ano channel used by t he synt hesi zerpr i vat e st at i c f i nal i nt CHANNEL = 0;

publ i c voi d pl ay( St r i ng not eSt r , i nt dur at i on)/ / second version; pl ay not e st r i ng f or speci f i ed dur at i on{

i f ( r ecei ver == nul l ) {Syst em. out . pr i nt l n( "No synt hesi zer t o pl ay not e: " + not eSt r ) ;r et ur n;

}

i nt note = getKey( not eSt r ) ;

Shor t Message message = new Short Message( ) ;try {

message. set Message( Short Message. NOTE_ON, CHANNEL, not e, VOLUME) ;r ecei ver . send( message, - 1) ;wai t ( dur at i on) ; / / i n msmessage. set Message( Short Message. NOTE_OFF, CHANNEL, not e, VOLUME) ;r ecei ver . send( message, - 1) ;

}cat ch ( I nval i dMi di Dat aExcept i on e){ Syst em. out . pr i nt l n( e) ; }

} / / end of pl ay( )

Ive also hard-wired the code to utilize channel 0, which plays notes as if on a piano.


10/34


11/34



3. Using a Speech Synthesizer

The Java Speech API (JSAPI, http://java.sun.com/products/java-media/speech/)covers two speech-related technologies: speech recognition, which converts spoken

language into text, and speech synthesis that goes the other way, rendering text intospoken audio. I only need the latter, so can make use of FreeTTS,(http://freetts.sourceforge.net/), a fairly complete implementation of the speechsynthesis parts of JSAPI. The main missing feature is JSML processing (the JavaSpeech Markup Language), which can be employed to fine-tune pronunciation, stress,

pauses, and other aspects of speaking, to make a computerized voice sound morenatural.

I downloaded the Windows binary version of FreeTTS v.1.2.2, unzipped it, and storedit in a convenient location (d:\freetts-1.2\ in my case).

Rather confusingly, theres initially no JSAPI JAR file in the FreeTTS lib\ directory

(d:\freetts-1.2\lib\); it only appears after the user executes jsapi.exe in that directory.

Many FreeTTS programming examples will not work unless the speech.properties filein d:\freetts-1.2\ is copied either to your home directory or to the JREs lib\ directory(which for me is c:\Program Files\Java\jre1.6.0_03\lib).

More details on these set-up issues can be found athttp://freetts.sourceforge.net/docs/jsapi_setup.html

Having done all this, the easiest way of testing things is to go to d:\freetts-1.2\bin\,and type:

> j ava j ar Hel l oWor l d. j ar

It uses an US male voice called "kevin16" to say "hello world".

3.1. Adding Voices

FreeTTSs "kevin16" voice isnt that great, as the HelloWorld example illustrates.Fortunately, FreeTTS supports several better US voices, available from the MBROLAwebsite (http://tcts.fpms.ac.be/synthesis/mbrola.html).

I downloaded four things: the PC/DOS version of the MBROLA software (inmbr301d.zip), and three voices called us1, us2, and us3. I moved the unzippedexecutable, mbrola.exe, to d:\freetts-1.2\mbrola\, along with the three unzipped voice

folders (us1\, us2\, and us3\).

I checked if the voices had been correctly installed by writing a program that lists allthe voices known to FreeTTS. My ListVoices.java application starts by obtaining alist of the machine's synthesizer engines by querying javax.speech.Central.

pr i vat e stati c Engi neLi st get Engi nes( )/ / first version usi ng J SAPI j avax. speech. Cent r al{

/ / Don' t set any pr oper t i es so al l synt hesi zer s wi l l be ret ur nedSynt hesi zer ModeDesc empt yDesc = new Synt hesi zer ModeDesc( ) ;

Engi neLi st engi neLi st = nul l ;try {engi neLi st = Central.availableSynthesizers ( emptyDesc) ;


12/34



}cat ch ( Except i on e){ Syst em. out . pr i nt l n( e) ;

Syst em. exi t ( 1) ;}r et ur n engi neLi st ;

} / / end of get Engi nes( )

Central.availableSynthesizers() is called with a description of the required synthesizerattributes, which in this case can be anything. Other examples of how to use Centralcan be found in its online documentation at http://java.sun.com/products/java-media/speech/forDevelopers/jsapi-doc/.

Central is a JSAPI class, which is implemented using lower-level FreeTTS classes. Ifyou dont mind calling these classes directly, then FreeTTSs installation can besimplified. In particular, it's no longer necessary to move the speech.properties file tothe JRE lib\ directory.

I rewrote getEngines() as:

pr i vat e stati c Engi neLi st get Engi nes( )/ / second version uses Fr eeTTS FreeTTSEngi neCent r al{

/ / Don' t set any pr oper t i es so al l synt hesi zer s wi l l be ret ur nedSynt hesi zer ModeDesc empt yDesc = new Synt hesi zer ModeDesc( ) ;

Engi neLi st engi neLi st = nul l ;try {

Fr eeTTSEngi neCent r al cent r al = new FreeTTSEngineCentral ( ) ;engi neLi st = cent r al .createEngineList( emptyDesc) ;


Syst em. exi t ( 1) ;}r et ur n engi neLi st ;

} / / end of get Engi nes( )

The use of the lower-level FreeTTSEngineCentral class means that I dont needCentral, and so dont need to move the speech.properties file at FreeTTS installationtime. The documentation for the FreeTTS-specific classes can be found at d:/freetts-1.2/javadoc/index.html.

Once I have an engines list, the voices they offer can be displayed:

Engi neLi st engi neLi st = getEngines( ) ;

/ / l oop over t he synt hesi zer sf or ( i nt e = 0; e < engi neLi st . si ze( ) ; e++) {

Synt hesi zer ModeDesc desc = ( Synt hesi zer ModeDesc) engi neLi st . get ( e) ;

/ / l oop over al l t he voi ces f or t hi s synt hesi zerVoi ce[ ] voi ces = desc. get Voi ces( ) ;f or ( Voi ce voi ce : voi ces)

Syst em. out . pr i nt l n( " " + desc. get Engi neName() +

" : Voi ce: " + voi ce. getName( ) +" ; Gender : " + genderStr( voi ce. get Gender ( ) ) ) ;


13/34



}

The gender information is represented by integer constants, so genderStr() changes itto more useful text:

pr i vat e st at i c St r i ng gender St r ( i nt gender ){

swi t ch ( gender) {case Voi ce. GENDER_FEMALE: r et urn " f emal e";case Voi ce. GENDER_MALE: r et urn "mal e";case Voi ce. GENDER_NEUTRAL: r etur n "neut r al " ;case Voi ce. GENDER_DONT_CARE:def aul t : r et ur n "unknown";

}} / / end of gender St r ( )

All this code is wrapped up inside a ListVoices class, which can be compiled and

executed like so:

> set " TTS=d: \ f r eet t s- 1. 2\ l i b"

> j avac - cp "%TTS%\ j sapi . j ar ; %TTS%\ f r eet t s. j ar ;%TTS%\ f r eet t s- j sapi 10. j ar ; . " Li st Voi ces. j ava

> j ava - Dmbr ol a. base=d: \ f r eet t s- 1. 2\ mbr ol a- cp "%TTS%\ j sapi . j ar ; %TTS%\ f r eet t s. j ar ;

%TTS%\ f r eet t s- j sapi 10. j ar ; . " Li st Voi ces

Not every program requires jsapi.jar, freetts.jar, and freetts-jsapi10.jar, in the

compilation and execution calls, but many do.

If we want to employ the three MBROLA US voices, then the call to java.exe mustinclude the mbrola.base property (which states the location of the directory holdingthe mbrola.exe executable).

In general, its useful to package up these lengthy compilation and execution calls inDOS batch files, to save on typing.

The output of the call to ListVoices will be something like:

Fr eeTTS en_US t i me synt hesi zer : Voi ce: al an; Gender : mal e

FreeTTS en_US gener al synt hesi zer : Voi ce: kevi n; Gender : mal eFr eeTTS en_US gener al synt hesi zer : Voi ce: kevi n16; Gender : mal eFr eeTTS en_US gener al synt hesi zer : Voi ce: mbr ol a_us1; Gender : f emal eFr eeTTS en_US gener al synt hesi zer : Voi ce: mbr ol a_us2; Gender : mal eFr eeTTS en_US gener al synt hesi zer : Voi ce: mbr ol a_us3; Gender : mal e

The alan, kevin, and kevin16 voices come with FreeTTS, and the three US voicesfrom MBROLA. I need a good quality, general voice for speaking words, so will usembrola_us1 from now on.


14/34



3.2. Making a Program Speak

I'll introduce the main elements of JSAPI by writing a Speaker class, which reads astring from the command line and speaks it. An example call to Speaker:

> j ava - Dmbr ol a. base=d: \ f r eet t s- 1. 2\ mbr ol a- cp "%TTS%\ j sapi . j ar ; %TTS%\ f r eet t s. j ar ;%TTS%\ f r eet t s- j sapi 10. j ar ; . " Speaker "my name i s j ane"

Speaker is passed "my name is jane", and says it using the US female voicembrola_us1.

A speech synthesis application (such as Speaker) typically passes through sevenstages:

1) The desired properties for the synthesizer are collected together in a modedescriptor. The descriptor might include the name of the synthesizer that we

require, its intended mode of use, and the chosen language.2) The synthesizer engine is created, using the mode descriptor as a guide.3) The synthesizer is moved to the ALLOCATED state, which causes it to load

resources.

4) Once in the ALLOCATED state, various tweaks can be applied to the engine,such as changing its voice.

5) The synthesizer is moved into an ALLOCATION sub-state called RESUMED,which makes it ready to process text into speech. It's also a good idea to make surethat its speaking queue, where text is stored waiting for processing, is in the

QUEUE_EMPTY state. This ensures that the supplied text will be processed asquickly as possible.

6) The text is passed to the engine, which renders it into speech. Its possible to havethe application wait for the speaking to finish by making it suspend until thespeaking queue is once again empty.

7) The synthesizer is closed down by moving it to the DEALLOCATED state,making it release its resources.

All these stages are present in the Speaker class, called from main():

publ i c st at i c voi d mai n( St r i ng[ ] ar gs){

St r i ng msg;i f ( ar gs. l engt h ! = 1) {

Syst em. out . pr i nt l n( "Usage: Speaker \ "sent ence to be sai d\ "" ) ;msg = "You must t el l me what t o say" ;

}el se

msg = args[ 0] ;

Speaker speaker = new Speaker( ) ;speaker .say( msg) ;speaker .closeDown( ) ;

} / / end of mai n( )


15/34



The Speaker constructor carries out stages 1-5, say() implements stage 6, andcloseDown() performs stage 7.

The Speaker() method:

/ / gl obal spr i vat e st at i c f i nal St r i ng VOI CE = "mbr ol a_us1";pr i vat e Synt hesi zer synt hesi zer ;

publ i c Speaker ( ){

try {Syst em. out . pr i nt l n( "General Synt hesi zer usi ng \ " " + VOI CE +

" \ " voi ce bei ng i ni t i al i zed. . . " ) ;

/ / speci f y t he r equi r ed synt hesi zer pr oper t i es ( st age 1)Synt hesi zer ModeDesc desc = new Synt hesi zer ModeDesc(

nul l , / / engi ne name ( don' t car e)"general " , / / mode name - - general usageLocal e. US, / / l ocal enul l , / / pr ef er a r unni ng synt hesi zer ( don' t car e)nul l ) ; / / voi ce ( none speci f i ed yet )

/ / creat e t he synt hesi zer ( st age 2)FreeTTSEngi neCent r al cent r al = new FreeTTSEngi neCent r al ( ) ;

/ / so no need f or " speech. pr oper t i es"Engi neLi st l i st = cent r al . creat eEngi neLi st ( desc);i f ( l i st . s i ze( ) > 0) {

Engi neCr eat e cr eat or = ( Engi neCr eat e) l i st . get ( 0) ;synt hesi zer = ( Synt hesi zer ) creat or . creat eEngi ne( ) ;

}

i f ( synt hesi zer == nul l ) {Syst em. er r . pr i nt l n( "Cannot creat e synt hesi zer ") ;Syst em. exi t ( 1) ;

}

/ / al l ocat e r esour ces ( st age 3)synt hesi zer . al l ocat e( ) ;synt hesi zer . wai t Engi neSt at e( Synt hesi zer . ALLOCATED) ;

/ / modi f y synt hesi zer pr oper t i es ( st age 4)Synt hesi zerProper t i es synProps =

synt hesi zer . get Synt hesi zer Pr oper t i es( ) ;

synPr ops. set Voi ce( getVoice( VOI CE) ) ;

/ / get synt hesi zer r eady t o speak ( st age 5)synt hesi zer . r esume( ) ;

/ / wai t unt i l t he synt hesi zer i s r eady t o speaksynt hesi zer . wai t Engi neSt at e( Synt hesi zer . RESUMED) ;synt hesi zer . wai t Engi neSt at e( Synt hesi zer . QUEUE_EMPTY) ;Syst em. out . pr i nt l n( "Synt hesi zer r eady") ;


Syst em. exi t ( 1) ;

}


16/34



} / / end of Speaker ( )

The SynthesizerModeDesc class is used to create a mode descriptor. The nullarguments in its constructor mean that the description doesn't care about the enginename, running behavior, or voices that the synthesizer offers. The descriptor only

requires that the synthesizer can manage general speech, in American English.

The several calls to Synthesizer.waitEngineState() force the Speech object to waituntil the synthesizer enters a desired state. For example, the call toSynthesizer.resume() in stage 5 requests that the synthesizer moves into theRESUMED state. The subsequent waitEngineState() calls waits for that state to beattained.

Once the synthesizer is in an ALLOCATED state, its behavior can be adjusted using aSynthesizerProperties object. The code above only sets the voice, but its also possibleto change attributes such as the pitch and speaking rate. SynthesizerProperties settingsare considered to be hints, so may have no effect, depending on the synthesizer.

Specifying a Voice

The getVoice() method searches through the voices associated with the synthesizer,looking for the specified voice. If it's present then its corresponding Voice object isreturned, and subsequently employed by the synthesizer.

pr i vat e Voi ce get Voi ce( St r i ng voi ceName){

/ / get t he pr oper t i es f or t hi s synt hesi zer engi neSynt hesi zer ModeDesc desc =

( Synt hesi zerModeDesc) synt hesi zer . get Engi neModeDesc( ) ;Voi ce[ ] voi ces = desc. get Voi ces( ) ;

/ / check i f voi ceName i s a known Voi ce f or t hi s engi neVoi ce voi ce = nul l ;f or ( i nt i = 0; i < voi ces. l engt h; i ++) {

i f ( voi ces[ i ] . get Name( ) . equal s( voi ceName) ) {voi ce = voi ces[ i ] ;br eak;

}}

i f ( voi ce == nul l ) {Syst em. er r . pr i nt l n( "Synt hesi zer coul d not f i nd t he " +

voi ceName + " voi ce") ;Syst em. exi t ( 1) ;

}r et ur n voi ce;

} / / end of get Voi ce( )

Speaking a Response

The say() method requests that a message is spoken, and returns when the speech isfinished.

publ i c voi d say(St r i ng msg)


17/34


18/34



4. Back to Sonification

Over the last 15 pages or so, Ive described three audio building blocks: theClipsPlayer class for playing clips, NotesPlayer which uses a MIDI synthesizer togenerate notes, and Speaker which reads out strings using the FreeTTSimplementation of JSAPI. Now its time to return to my Sonification application, firstshown in Figure 1 (and repeated here as Figure 5 to save you having to flick back tothe beginning).

Figure 5. Sonification of a Java Application (Again).

SoundTrace is a tracer employing the Java Debug Interface (JDI) API, while soundgeneration is managed by a thread which reads messages from a queue filled by thetracer. SoundGen utilizes ClipsPlayer, NotesPlayer, and Speaker to generate audiooutput.

Figure 6 shows the UML class diagrams for the sonifier, with only public methodslisted for each class.

Figure 6. SoundTrace Class Diagrams


19/34



The three classes at the top left of Figure 6 (SoundTrace, StreamRedirecter, andSoundEventsMonitor) contain the tracer code.

SoundTrace.java is little more than a renamed SimpleTrace.java from Java ArtChapter 3. It sets up the command-line launching connection which starts the JVMand creates a local link with the JVM on the same machine. It passes the applicationsname and input arguments over to the JVM, and employs the StreamRedirecter classto redirect the JVMs output and error streams to stdout and stderr. StreamRedirecteris unchanged from previous chapters.

SoundEventsMonitor is a subset of JDIEventMonitor from Java Art Chapter 3, andIll only describe the parts which are new or differ significantly from before.SoundEventsMonitor monitors incoming JDI events for a program running in theJVM. When a method is called, a word is spoken, when keywords in the code areencountered, musical notes are played, and when the program starts, ends, and when amethod returns, sound clips are played.

SoundEventsMonitor maintains two types of audio information:

1) it uses the MethodWords and MethodWord classes to store the words that will bespoken when methods are called in the application being traced;

2) it calls AppNotes and FileNotes to analyze the application's code, and all the linescontaining keywords are assigned musical notes. This is a form of static analysissince the program files are scanned before their code is executed, at class loadtime.

Three types of messages are placed onto the blocking queue going to the SoundGenthread:

$w

$n $c

For example:

$w f oo / / say t he word " f oo"$n C4 / / pl ay t he not e C4$c zap 500 / / pl ay zap. wav whi ch l ast s f or 500ms

SoundGen reads these audio messages, passing them to the relevant audio object(ClipsPlayer, NotesPlayer, or Speaker).

4.1. Initializing Event MonitoringThe SoundEventsMonitor constructor initializes the AppNotes and MethodWordsobjects, and sets up the blocking queue connection to the SoundGen thread.

/ / gl obal spr i vat e f i nal Vi r t ual Machi ne vm; / / t he J VM/ / musi cal dat a obj ect spr i vat e AppNot es appNotes;

/ / st ores musi cal not es r epr esent i ng t he appl i cat i on code

pr i vat e Met hodWor ds met hodWor ds;

/ / st ores words t hat wi l l be spoken when met hods ar e cal l ed


20/34



/ / sound gener at i on obj ect spr i vate Bl ocki ngQueue sndQueue; / / hol ds sound messagespr i vate SoundGen soundGen; / / pr ocesses t he sound messages

publ i c SoundEvent sMoni t or ( Vi r t ual Machi ne j vm)

{ super ( "SoundEvent sMoni t or" ) ;vm = j vm;

appNot es = new AppNot es( ) ;met hodWor ds = new Met hodWor ds( ) ;

/ / i ni t i al i ze t he message queue and t he sound gener at i on t hr eadsndQueue = new Li nkedBl ocki ngQueue( ) ;soundGen = new SoundGen( sndQueue) ;soundGen. st ar t ( ) ;

setEventRequests( ) ;

} / / end of SoundEvent sMoni t or ( )

Four types of program events are requested in setEventRequests().

pr i vat e voi d set Event Request s( ){

Event Request Manager mgr = vm. eventRequest Manager ( ) ;

/ / r epor t ' met hod ent r i es'Met hodEnt r yRequest menr = mgr . cr eat eMet hodEnt r yRequest ( ) ;f or ( i nt i = 0; i < excl udes. l engt h; ++i )

menr . addCl assExcl usi onFi l t er ( excl udes[ i ] ) ;

menr . set SuspendPol i cy( Event Request . SUSPEND_EVENT_THREAD) ;menr . enabl e( ) ;

/ / r epor t ' met hod exi t s'Met hodExi t Request mexr = mgr . cr eat eMet hodExi t Request ( ) ;f or ( i nt i = 0; i < excl udes. l engt h; ++i )

mexr . addCl assExcl usi onFi l t er ( excl udes[ i ] ) ;mexr . set SuspendPol i cy( Event Request . SUSPEND_EVENT_THREAD) ;mexr. enabl e( ) ;

/ / report ' cl ass l oads 'Cl assPrepareRequest cpr = mgr. cr eat eCl assPrepareRequest ( ) ;f or ( i nt i = 0; i < excl udes. l engt h; ++i )

cpr . addCl assExcl usi onFi l t er ( excl udes[i ] ) ;cpr . enabl e( ) ;

/ / report ' t hread star t s 'ThreadSt ar t Request t sr = mgr . cr eat eThreadSt ar t Request ( ) ;t sr . enabl e( ) ;

} / / end of setEvent Request s( )

Method entry and exits are monitored so that various sounds can be generated. Classloads must be observed so that the corresponding Java file can be analyzed byAppNotes and FileNotes to create musical notes for its keywords. A thread starttriggers code stepping so the correct note can be played when a line is executed.


21/34



4.2. Handling Events

As in JDIEventMonitor in Java Art Chapter 3, SoundEventsMonitor callshandleEvent() to process an incoming event:

pr i vat e voi d handl eEvent ( Event event ){

/ / met hod ent r y/ exi t event si f ( event i nst anceof Met hodEnt r yEvent )

methodEnt r yEvent ( ( MethodEntr yEvent ) event ) ;el se i f ( event i nst anceof Met hodExi t Event )

met hodExi t Event ( ( Met hodExi t Event ) event ) ;

/ / cl ass l oad event sel se i f ( event i nst anceof Cl assPr epar eEvent )

cl assPrepar eEvent ( ( Cl assPrepar eEvent ) event ) ;

/ / t hr ead st ar t event sel se i f ( event i nst anceof Thr eadSt ar t Event )

t hr eadSt ar t Event ( ( Thr eadSt ar t Event ) event ) ;

/ / st ep event - - a l i ne of code i s about t o be execut edel se i f ( event i nst anceof St epEvent )

st epEvent ( ( St epEvent ) event ) ;

/ / VM event sel se i f ( event i nst anceof VMSt ar t Event )

vmSt art Event ( ( VMSt ar t Event ) event ) ;el se i f ( event i nst anceof VMDeat hEvent )

vmDeathEvent ( ( VMDeat hEvent ) event ) ;el se i f ( event i nst anceof VMDi sconnect Event )

vmDi sconnect Event ( ( VMDi sconnect Event ) event ) ;

el set hr ow new Err or ( "Unexpect ed event t ype") ;

} / / end of handl eEvent ( )

handleEvent() deals with methods, classes, threads, and step events. The JVM-relatedprocessing in vmStartEvent(), vmDeathEvent(), and vmDisconnectEvent() isunchanged from JDIEventMonitor.

Entering a Method

Method entry triggers the playing of a clip if the method is main(), and a method wordis spoken.

/ / gl obalpr i vat e Met hodWor ds met hodWor ds;

/ / st ores t he words t hat wi l l be spoken when methods ar e cal l ed

pr i vat e voi d methodEntr yEvent ( MethodEnt r yEvent event ){

Met hod met h = event . met hod( ) ;St r i ng cl assName = meth. decl ar i ngType( ) . name( ) ;

System. out . pr i nt l n( ) ;


22/34



i f ( met h. i sConst r uct or ( ) ) / / use cl ass name f or t he const r uct orSystem. out . pr i nt l n( "ent er ed " + cl assName + " const r uct or" ) ;

el seSyst em. out . pr i nt l n( "ent ered " + cl assName+" . "+met h. name( ) +" ( ) " ) ;

i f ( met h. name( ) . equal s( "mai n") ) / / pr ogr am st ar t i ng so pl ay cl i p

saySound( "$c wi ndChi me 1930" ) ;

/ / r epor t method ent r y by sayi ng a wordSt r i ng wor d = met hodWor ds. get Wor d( cl assName, met h) ;i f ( wor d ! = nul l ) {

Syst em. out . pr i nt l n( " sayi ng " + wor d) ;saySound( "$w " + word) ;

}el se

System. out . pr i nt l n( "Coul d not f i nd met hod wor d t o say") ;} / / end of methodEnt r yEvent ( )

The clip played when main() starts is a wind chime which lasts for 1.93 seconds. Theword spoken for a method is retrieved from the MethodWords object.

saySound() doesnt say the sound directly; it puts a message onto the blocking queueso it can be processed by the SoundGen thread.

/ / gl obalpr i vate Bl ocki ngQueue sndQueue; / / hol ds sound messages

pr i vat e voi d saySound( St r i ng msg){ t r y {

sndQueue. put ( msg) ;

}cat ch( I nt er r upt edExcept i on e) {}

}

Leaving a Method

methodExitEvent() is called when all the code in a method has been executed, and theapplication is about to return to the calling function. A return from main() (i.e. thetermination of the application) is signaled by playing a clip of a key dropping, whileother returns are represented by a zapping sound.

pr i vat e voi d methodExi t Event ( Met hodExi t Event event ){

Met hod met h = event . met hod( ) ;St r i ng cl assName = meth. decl ar i ngType( ) . name( ) ;

i f ( met h. i sConstr uctor ( ) )System. out . pr i nt l n( "exi t i ng " + cl assName + " const r uct or" ) ;

el seSystem. out . pr i nt l n( "exi t i ng " + cl assName + " . "

+ meth. name( ) + " ( ) " ) ;

/ / r epor t met hod r et ur n/ pr ogr am exi ti f ( met h. name( ) . equal s( "mai n") ) / / pr ogr am i s exi t i ngsaySound( "$c keyDr op 430") ;

el se / / t he met hod i s r et ur ni ng


23/34



saySound( "$c zap 500" ) ;System. out . pr i nt l n( ) ;

} / / end of methodExi t Event ( )

Clip playing is only requested in methodEntryEvent() and methodExitEvent(), and so

Ive hard-wired the details of the clip messages into those methods.If more clips were used, across more methods, then it would be better to store the clipdetails (i.e. the event trigger name, wave filename, and duration) in a separate class,which could perhaps load its information at start-up time.

Loading a Class

classPrepareEvent() is called after a new class has been loaded, which is a good timeto parse the classs file to build a collection of musical notes, and use the classsmethod names to create the words stored in methodWords.

/ / gl obal spr i vat e AppNot es appNotes;pr i vat e Met hodWor ds met hodWor ds;

pr i vat e voi d cl assPrepar eEvent ( Cl assPrepar eEvent event ){

Ref er enceType ref = event . r ef erenceType( ) ;

Li st f i el ds = r ef . f i el ds() ;Li st met hods = r ef . met hods( ) ;St r i ng cl assName = r ef . name( ) ;

St r i ng f nm;try {

f nm = r ef . sour ceName( ) ; / / get f i l ename of t he cl assappNotes. add( f nm) ; / / cr eat e musi cal notes f or code i n f nm

}cat ch ( Absent I nf ormat i onExcept i on e){ f nm = "??" ; }

System. out . pr i nt l n( "l oaded cl ass: " + cl assName + " f r om " +f nm + " - f i el ds=" + f i el ds. si ze( ) +

" , met hods=" + methods. si ze( ) ) ;

met hodWor ds. addMet hods( cl assName, met hods) ;/ / convert met hod names i nto words} / / end of cl assPrepar eEvent ( )

Starting a Thread

When a new thread starts running, the tracer needs to switch on single stepping so itcan monitor which lines are being executed.

pr i vat e voi d thr eadSt ar t Event ( Thr eadSt ar t Event event )

{ ThreadRef er ence t hr = event . t hread( ) ;


24/34



i f ( t hr . name( ) . equal s(" Si gnal Di spat cher ") | |t hr . name( ) . equal s( "Dest r oyJ avaVM") | |t hr . name( ) . st ar t sWi t h( "AWT- " ) ) / / AWT t hr eads

r et ur n;

i f ( t hr . t hr eadGr oup( ) . name( ) . equal s( "syst em") )

r et ur n; / / i gnor e syst em t hr eads

setStepping( t hr ) ;} / / end of t hr eadSt ar t Event ( )

setStepping() asks the JVM to issue step events, which are sent out just before eachline is executed.

pr i vat e voi d set St eppi ng( Thr eadRef er ence thr ){

Event Request Manager mgr = vm. eventRequest Manager ( ) ;St epRequest sr = mgr. cr eat eSt epRequest ( t hr, StepRequest . STEP_LI NE,

St epRequest . STEP_I NTO) ;sr . set SuspendPol i cy( Event Request . SUSPEND_EVENT_THREAD) ;f or ( i nt i = 0; i < excl udes. l engt h; ++i )

sr. addCl assExcl usi onFi l t er ( excl udes[i ] ) ;sr . enabl e( ) ;

} / / end of set St eppi ng( )

There are a few different kinds of step requests, but the most common is acombination of STEP_INTO and STEP_LINE which means that every line in everymethod will be examined.

Single Stepping Event Handling

After setStepping() has requested single stepping, StepEvents will start to be added tothe event queue, and handleEvent() will process them by calling stepEvent().

An event contains the location of the line that's about to be executed, which includesthe codes filename and line number. These are passed to the AppNotes object to helpretrieve a note string if that line contains a keyword. If a note is found (e.g. "C4"),then its sent as an "$n" message over to SoundGen.

pr i vat e voi d st epEvent ( St epEvent event ){

Locat i on l oc = event . l ocat i on( ) ;t r y { / / get the l i ne of code

St r i ng f nm = l oc. sour ceName( ) ; / / get f i l ename of codei nt l i neNum = l oc. l i neNumber ( ) ;St r i ng not e = appNot es. get Not e( f nm, l i neNum) ;

/ / pl ay not e f or t he l i nei f ( not e ! = nul l ) {

Syst em. out . pr i nt l n( f nm + ". " + l i neNum + ": pl ay " + not e) ;saySound( "$n " + note) ;

}}cat ch ( Absent I nf ormat i onExcept i on e) {}

} / / end of st epEvent ( )


25/34



4.3. Extracting Words from Method Names

The MethodWords class is little more than a wrapper around an ArrayList ofMethodWord objects, along with several methods for adding to and searching through

the list:

/ / gl obal i n MethodWords cl asspr i vat e Ar r ayLi st mWor ds;

/ / st ores t he words t hat wi l l be spoken when methods ar e cal l ed

publ i c MethodWords( ){ mWor ds = new Arr ayLi st ( ) ; }

addMethod() is typical of the class's code. It adds a MethodWord object to the list, ifthat object isnt already there.

publ i c voi d addMet hod(St r i ng cl assName, Met hod met h){

i f ( get Word( cl assName, met h) ! = nul l ) / / al r eady pr esentr et ur n; / / don' t add

Met hodWor d mw;i f ( met h. i sConst r uct or ( ) ) / / use cl ass name f or the const r uct or

mw = new Met hodWor d( cl assName, cl assName, t hi s) ;el se

mw = new Met hodWor d( cl assName, met h. name( ) , t hi s) ;mWor ds. add( mw) ;

} / / end of addMethod( )

A minor complication of addMethod() (and several other methods in MethodWords)is the need to map the Method object into a method name. I set the constructor'smethod name to be its class name, rather than the "" string returned byMethod.name().

The primary aim of a MethodWord object is to map the class and method names ontoa word, which can be spoken by the FreeTTS speech synthesizer. The tricky part isthat each word must be unique and preferable short.

/ / gl obal s i n Met hodWord cl asspr i vate St r i ng cl assName, met hodName;pr i vat e St r i ng wordToSay;

publ i c Met hodWor d( St r i ng cName, St r i ng mName, Met hodWor ds mWor ds){

cl assName = cName;met hodName = mName;wor dToSay =buildWord( met hodName, mWor ds) ;Syst em. out . pr i nt l n( cl assName + " : " + met hodName +

"( ) i s r epr esent ed by t he word \ " " + wordToSay + " \ "" ) ;} / / end of Met hodWord( )


26/34



The trickiness is hidden inside buildWord() which tries a number of different wordcreation strategies, testing each one for uniqueness against the existing words in theMethodWords object, mWords.

pr i vat e St r i ng bui l dWord( St r i ng mName, Met hodWor ds mWords)

{i f ( mName. l ength( ) < MI N_LEN)

/ / i f met hod name i s shor t , use i t as t he word t o sayi f ( mWords. i sUni que( mName) )

r et ur n mName;

/ / shor t en met hod nameSt r i ng shor t Name;f or ( i nt i = MI N_LEN; i


27/34



4.4. Converting Files into Musical Notes

The AppNotes and FileNotes classes parse the files involved in the application,converting the lines containing specified keywords into musical note strings. At runtime, when the tracer executes a line of code, its corresponding note is played (if oneexists).

AppNotes is mostly just a wrapper around a TreeMap of FileNotes objects, andseveral methods for adding to and searching through the map:

/ / gl obal i n AppNot es cl asspr i vat e TreeMap f i l esNotesMap;

publ i c AppNotes( ){ f i l esNot esMap = new TreeMap( ) ; }

The String used as the map's key is the filename.

add() adds a FileNotes object to the map, if that object isnt already there.

publ i c voi d add( St r i ng f nm){

i f ( f i l esNot esMap. cont ai nsKey( f nm) ) {Syst em. out . pr i nt l n( f nm + "al r eady st or ed") ;r et ur n;

}f i l esNotesMap. put ( f nm, new Fi l eNot es( f nm) ) ;Syst em. out . pr i nt l n( f nm + " added to f i l es Not es") ;

} / / end of add( )

FileNotes does all the real work. It loads a file line-by-line, analyzing each line ofcode. Those lines containing a keyword are converted into musical note strings, while"" is stored for the lines without keywords.

/ / gl obal i n Fi l eNot es cl asspr i vat e Ar r ayLi st codeNotes;

/ * each l i ne of t he i nput f i l e i s r epr esent ed bya musi cal not e st r i ng, or "" */

publ i c Fi l eNot es( St r i ng f i l eName){

codeNot es = new Ar r ayLi st ( ) ;

i nt l i neNum = 0;St r i ng l i ne = nul l ;Buf f er edReader i n = nul l ;St r i ng[ ] wor ds;

Syst em. out . pr i nt l n( "Anal yzi ng " + f i l eName) ;try {

i n = new Buf f eredReader( new Fi l eReader( f i l eName) ) ;whi l e ( ( l i ne = i n. r eadLi ne( ) ) ! = nul l ) {

l i neNum++;wor ds = l i ne. spl i t ( "\ \ W+") ; / / spl i t on wor d boundar i es

storeNote( wor ds, l i neNum) ;}


28/34



}cat ch ( I OExcept i on ex) {

Syst em. out . pr i nt l n( "Pr obl em r eadi ng " + f i l eName) ;}f i nal l y {

try {

i f ( i n != nul l )i n. cl ose( ) ;}cat ch ( I OExcept i on e) {}

}} / / end of showLi nes( )

A line of input is split into tokens based on word boundaries, and the resulting array isexamined for keywords by storeNote().

If storeNote() finds a keyword in the words array, then a corresponding note is addedto the codeNotes list. If no keyword is found, then "" is stored instead, which ensures

that every program line has an entry in the list.

/ / gl obal spr i vat e f i nal St r i ng[ ] keywor ds =

{ "el se", "br eak", "swi t ch", "case", " f i nal l y" , "cat ch", " f or" ," i f " , " t r y" , "cont i nue", "r et urn", "def aul t " , "do", "whi l e"

};

/ / C D E F G A B i n t he 4th and 5th oct ave on a pi anopr i vat e f i nal St r i ng[ ] not es =

{ "C4", "D4", "E4", "F4", "G4", "A4", "B4","C5", "D5", "E5", "F5", "G5", "A5", "B5"

};

pr i vat e voi d st or eNot e( St r i ng[ ] wor ds, i nt l i neNum){

f or ( St r i ng wor d : wor ds)f or ( i nt i =0; i < keywor ds. l engt h; i ++)

i f ( wor d. equal sI gnor eCase( keywor ds[ i ] ) ) {codeNot es. add( not es[ i ] ) ;System. out . pr i nt l n( " " + l i neNum +

": f ound " + keywor ds[ i ] ) ;r et ur n;

}codeNotes. add( " " ) ; / / no keyword f ound

} / / end of st or eNot e( )

The correspondence between keywords and notes is encoded by the ordering of thekeywords[] and notes[] arrays. For example, if the "break" keyword is found on a line,which is the second string in keywords[], then the second note in notes[] ("D4") will

be added to the codeNotes list.

I've restricted myself to control structure keywords, so that the played notes will givean indication of the control flow in the application. A complete list of Java keywordscan be found athttp://java.sun.com/docs/books/tutorial/java/nutsandbolts/_keywords.html.


29/34



To simplify matters a little, storeNote() stops searching a line as soon as a singlekeyword is found. Even when a line has multiple keywords, it will only berepresented by one note in the list.

4.5. Static and Dynamic AnalysisAppNotes and FileNotes carry out a form of static analysis the text of theapplication code is converted into lists of musical notes. This can be carried out at anytime, for example in a separate pre-processing stage before the tracer begins, so itneedn't have any speed impact.

Although static analysis is a great tool, it must often be combined with dynamicanalysis (i.e. analysis performed at run-time), since it's only at run-time that certaininformation comes available, such as the user's input, the results of random-numbergeneration, or the processing order between threads.

The dual use of static and dynamic analysis can be seen in SoundTrace staticanalysis is performed in AppNotes and FileNotes (as weve just seen), and dynamicanalysis of audio messages is carried out by SoundGen.

4.6. Generating Sounds

The SoundGen thread monitors a blocking queue which stores audio messages sent toit from SoundEventMonitor.

Three types of messages arrive on the BlockingQueue, with the formats

$w $n $c

Every 0.5 seconds, SoundGen empties the queue of all the messages currently there,storing them in a separate list. Optimizations are applied to the list at run-time (i.e.dynamic analysis), before the messages are converted into either:

a word spoken using the Speaker class (described in section 3.2) a musical note performed by the NotesPlayer class (see section 2)

a WAV audio clip played with the ClipPlayer class (see section 1).The SoundGen constructor initializes audio objects:

/ / gl obal s/ / sound pr ocessi ng obj ect spr i vat e Speaker speaker ;pr i vat e Not esPl ayer not esPl ayer ;pr i vat e Cl i psPl ayer cl i psPl ayer ;

pr i vat e Bl ocki ngQueue queue; / / f or i ncomi ng messages

publ i c SoundGen( Bl ocki ngQueue q){queue = q;


30/34



/ / i ni t i al i ze sound pl ayi ng t ool sspeaker = new Speaker( ) ;notesPl ayer = new NotesPl ayer( ) ;cl i psPl ayer = new Cl i psPl ayer ( ) ;

} / / end of SoundGen( )

The queue emptying, analysis of the resulting list, and message processing is carriedout in the threads run() method:

/ / gl obal spr i vat e st at i c f i nal i nt DELAY = 500;

/ / t i me del ay ( i n ms) between queue exami nat i onspr i vat e Bl ocki ngQueue queue; / / f or i ncomi ng messagespr i vat e bool ean pr ogFi ni shed = f al se;

publ i c voi d r un( )/ / per i odi cal l y dr ai n and pr ocess messages

{Ar r ayLi st msgs = new Ar r ayLi st ( ) ;

/ / t o st ore messages r emoved f r om t he queuetry {

whi l e ( ! pr ogFi ni shed) {msgs. cl ear ( ) ;queue. dr ai nTo( msgs) ; / / the dr ai ni ng may not r etur n anyt hi ngcombineMsgs( msgs) ;f or ( St r i ng msg : msgs)processMsg( msg) ;

wai t ( DELAY) ;}

/ / f i ni sh of fSyst em. out . pr i nt l n( "t hr ead exi t i ng") ;speaker . cl oseDown( ) ;not esPl ayer . cl oseDown( ) ;

}cat ch ( Except i on e){ Syst em. out . pr i nt l n( e) ; }} / / end of r un( )

pr i vat e voi d wai t ( i nt dur at i on){ t r y {

Thread. sl eep( durat i on) ; / / i n ms

}cat ch ( I nt er r upt edExcept i on e){ e. pr i nt St ackTr ace( ) ; }

}

Queue draining occurs every DELAY milliseconds, and the messages are moved overto a temporary list.

4.7. Analyzing the Messages

There are many ways of analyzing the messages, but SoundGen only looks for

repeats, replacing them with a single message.


31/34



For example, assume there are five C4 note messages in the middle of the list:

. . . "$s push" "$n C4" "$n C4" "$n C4" "$n C4" "$n C4" "$c zap 500". . .

The five messages are reduced to one, with the number of repeats appended to it:

. . . "$s push" "$n C4 5" "$c zap 500". . .

This optimization is carried out frequently because of the way that note messagesdenote keywords, including iterative keywords such as "while" and "for". If a loop isrepeatedly executing, then the queue will quickly become lengthy.

There's no reliable way to simplify the repetition information by static analysis sincethe number of loop iterations will often depend on run-time values, such as user input;message reduction must be carried out dynamically.

combineMsgs() looks at each message in the list, and removes any duplicates thatfollow it.

pr i vat e voi d combi neMsgs( Ar r ayLi st msgs){

i nt t ot al Removed = 0;f or ( i nt i =0; i < msgs. si ze( ) ; i ++)

t ot al Removed += r emoveDups( i , msgs) ; / / msgs l i st may get smal l erSyst em. out . pr i nt l n( "Tot al r emoved: " + t otal Removed) ;

}

pr i vat e i nt r emoveDups( i nt i , Ar r ayLi st msgs)/ * Remove any dupl i cat es of t he message at posi t i on i , st ar t i ng

f r om posi t i on i +1 i n t he l i st . Add t he number of r epet i t i onst o t he message. */

{St r i ng cur r Msg = msgs. get ( i ) ;i nt pos = i +1;

i nt numDups = 0;whi l e ( ( pos+numDups) < msgs. si ze( ) ) {

i f ( ! cur r Msg. equal s( msgs. get ( pos+numDups) ) )br eak;

numDups++;}

i f ( numDups > 0) {Syst em. out . pr i nt l n( "Removi ng " + numDups + " dupl i cates" ) ;i nt count = 0;

whi l e ( count ! = numDups) {msgs. r emove( pos) ;count ++;

}

/ / add no of dupl i cat es to end of cur r ent message at imsgs. set ( i , curr Msg + ( " " + ( numDups+1)) ) ;

}

r et urn numDups;} / / end of r emoveDups( )


32/34



A basic drawback of this approach is that the optimizations are time-dependent. Thelist contents depends on how often the message queue is drained, and this affects thenumber of duplicated messages detected and removed..

removeDups() checks if a single message appears multiple times, one after another. Amore complicated optimization would be to search for repeating sequencesofmessages. For instance, assume there's a repeating sequence containing a "push", note

playing, and a 'zap' clip:

. . "$s push" " $n C4 5" "$c zap 500" "$s push" " $n C4 5" "$c zap 500". .

The repeating three-message sequence could be reduced to something like:

. . " [ " " $s push" " $n C4 5" " $c zap 500" " ] 2" . .

The "[" and "]2" denotes that the sequence inside the brackets appeared twice in theoriginal list. This requires more complicated pattern matching and parsing, which Ihaven't implemented.

4.8. Processing a Message

Processing a message is a three-way branch which converts a message to a spokenword, a musical note, or a sound clip.

/ / gl obalpr i vat e Speaker speaker ;

pr i vat e voi d pr ocessMsg( St r i ng msg)

{System. out . pr i nt l n( "- - > " + msg) ;i f ( msg. st ar t sWi t h( "$w ") ) / / speak a wor dspeaker.say( msg. subst r i ng( 3) ) ;

el se i f ( msg. st ar t sWi t h( "$n ") ) / / pl ay a not eprocessNote( msg. subst r i ng( 3) ) ;

el se i f ( msg. start sWi t h( "$c ") ) / / pl ay a cl i pprocessClip( msg. subst r i ng( 3) ) ;

el seSyst em. out . pr i nt l n( "Coul d not under st and " + msg) ;

} / / end of pr ocessMsg( )

The string in the "$w" message is passed to the Speaker object to be spoken.

Processing a note involves the pulling apart of the "$n" data, which takes the form" [ < number of repeats > ]" (e.g. "C4 2"), where the number ofrepeats is optional.

/ / gl obal spr i vat e st at i c f i nal i nt NOTE_DURATI ON = 800;

/ / f i xed dur at i on f or pl ayi ng a not epr i vat e st at i c f i nal i nt MAX_REPEATS = 3;

/ / max mul t i pl e f or pl ayi ng repeat ed notes

pr i vat e Not esPl ayer not esPl ayer ;


33/34



pr i vat e voi d pr ocessNot e( St r i ng msg){

St r i ng[ ] t oks = msg. spl i t ( "\ \ s+") ;i nt numTi mes = 0;

i f ( t oks. l engt h == 2) { / / t her e i s a numbertry {numTi mes = I nt eger . par seI nt ( t oks[ 1] ) ;

}cat ch ( Except i on e){ Syst em. out . pr i nt l n( "Number i s i ncor r ect f or " + t oks[1]) ; }

}

i f ( numTi mes != 0) { / / pl ay a note r epeat edl y, and more qui ckl yi f ( numTi mes > MAX_REPEATS)

numTi mes = MAX_REPEATS;f or ( i nt i =0; i < numTi mes; i ++)notesPlayer.play( t oks[ 0] , NOTE_DURATI ON/ 2) ;

}el se / / pl ay a si ngl e not enotesPlayer.play( t oks[ 0] , NOTE_DURATI ON) ;

} / / end of pr ocessNot e( )

It's irritating if a note is played too many times in succession. So, I've fixed it that anote can only be played at most MAX_REPEATS times, and at increased speed (byhalving its duration).

Processing a clip involves a similar sort of parsing as for a note. "$c" data has theform " [ < number of repeats >]" (e.g. "zap 500 2"), where the number of repetitions is optional.

/ / gl obal spr i vat e st at i c f i nal i nt CLI P_DURATI ON = 2000;

/ / f i xed dur at i on f or pl ayi ng a cl i p

pr i vat e bool ean pr ogFi ni shed = f al se;pr i vat e Cl i psPl ayer cl i psPl ayer ;

pr i vat e voi d pr ocessCl i p( St r i ng msg){

St r i ng[ ] t oks = msg. spl i t ( "\ \ s+") ;

i f ( t oks[0] . equal s(" keyDr op") )/ * denot es end of mai n( ) i n the t r aci ng pr ogr am, whi ch means t he

end of t r aci ng, and of sound gener at i on */pr ogFi ni shed = t r ue;

/ / get dur at i oni nt dur at i on = 0;i f ( t oks. l engt h >= 2) { / / t her e i s a dur at i on

try {dur at i on = I nt eger . par seI nt ( t oks[ 1] ) ;

}

cat ch ( Except i on e){ Syst em. out . pr i nt l n( "Dur at i on i s i ncor r ect f or " + t oks[ 1] ) ; }


34/34


}i f ( dur at i on == 0) / / probl em wi t h dur at i on, so use a guess

dur at i on = CLI P_DURATI ON;

/ / get number of t i mes

i nt numTi mes = 1;i f ( t oks. l engt h == 3) { / / t her e i s a numbertry {

numTi mes = I nt eger . par seI nt ( t oks[ 2] ) ;}cat ch ( Except i on e){ Syst em. out . pr i nt l n( "Repeat s i s i ncor r ect f or " + t oks[ 2] ) ; }

}

/ / pl ay a cl i p r epeat edl yf or ( i nt i =0; i < numTi mes; i ++)clipsPlayer.play( t oks[0] , durat i on) ;

} / / end of pr ocessCl i p( )

If a clip's duration is incorrectly parsed, then it's assumed to be 2 seconds long.

When the "keyDrop" clip turns up, it means that the application's main() method isexiting, and so the global boolean progFinished is set to true. This will cause the

processing loop in run() to terminate, allowing the thread to finish.

son if ication

Documents