the caitlin system musical program auralisation & external representations of pascal

Musical Program Auralisation 1

The cAitlIn systemMusical Program Auralisation

& External Representations of Pascal

http://computing.unn.ac.uk/staff/cgpv1/caitlin

Dr Paul VickersSchool of Informatics, Northumbria University

Professor J.L. AltyDean of Faculty of Science, Loughborough

University


Introduction Major features Background to the project Sound in HCI A case for music The cAitlIn system Demonstration Future work


Major features With cAitlIn we have:

– A musical equivalent for various syntactic components of a programming language = musical grammar

– Extended the earcon technique (q.v.)– An audio program structure diagram

(execution path)


Major features– A MPAS usable by non-musicians– Shown that music can be used to

communicate information about program state and structure

– In turn, this info can assist bug location Empirical evaluation Organising principles Opened new line of enquiry &

researchAuditory External Representations


Background Interests in music, computing, &

programming Hunch that music could be used to

assist in the programming process Develop and test a musical framework

for communication of program information


Sound in HCI Increasing research focus since mid ‘80s. A belief that non-speech audio useful as

interaction medium A nascent field called auditory display

spawns techniques like:– Sonification– Audification– Auralisation


Examples Auditory icons Earcons Data visualisation (sonification &

audification) Assistive technologies


Program auralisation Mapping of program data and events to

sound Main examples

– LogoMedia– Sonnet– ADSL– LISTEN

Largely untested


Why music? Gives multiple audio streams in parallel Structured and organised according to

specific rules Time-ordered. Visual techniques are

spatial. Can music’s grammars be used to carry

real information?


cAitlIn system Computer Audio Interface to Locate

Incorrect Nonsense Uses musical motifs to render Pascal

language constructs at run time Motif design is hierarchic as is

organisation of constructs


Hierarchic motif designConstruct type

Selection Iteration

IFs CASEs

IF...ELSEIF... CASE CASE...ELSE

Determin-ate

FOR

Indeterm-inate

WHILE REPEAT

Abstract auralizationEntry/Body/Exit leitmotifs

Genericselection leitmotif

Genericiteration leitmotif

GenericIF leitmotif

GenericCASE leitmotif


Musical grammars


Musical grammars We map existing grammar (e.g.

Pascal syntax, English, etc.) onto a musical grammar

Transliteration or translation? We may get emergent properties

leading to a new musical grammar


cAitlIn musical grammar Uses motif classes and variations to

render Pascal constructs Allows

– abstraction– nesting (tonality & harmony preserved

across motifs) Uses grammatical rules (e.g. a tonal

music system)


cAitlIn musical grammar cont Renders information not directly

communicated by source code or other visualisation techniques

a new grammar, conveys extra meaning


Advantages Over earcons

– do not show state Over Auditory Icons

– symbolic mappings = poor system aural ecology

Syntactic design expect retention Displays both flow and state


Design of Tunes Points of Interest

– Construct Entry– Evaluation of condition– Execution of body– Exit from construct

Boolean True=major, False=minor Drones


Selection Tunes Metaphorically asking & answering a

question. Tune rises on entry and descends on

exit


Iteration Tunes Major/Minor chord to show when

condition evaluated Heard at start of WHILE loops and at

end of REPEAT loops FOR loops increase/decrease the pitch

each time round the loop


Variations All selections use same basic tune,

modified for individual constructs All iterations use another basic tune,

again modified


Auditory Parentheses As () bracket a thought in a sentence,

so the auditory parentheses serve to open and close constructs


Other Statements All other statements are currently

represented by a drum beat, one per statement.


Demonstration


Examples Generic selection motif Generic iteration motif


IF (true)a := 10 ;IF a = 10 THEN Writeln (‘A is 10’) ;


IF (false)a := 0 ;IF a = 10 THEN Writeln (‘A is 10’) ;


IF…ELSE (true)a := 10 ;IF a = 10 THEN Writeln (‘A is 10’)ELSE Writeln (‘A is not 10’) ;


IF…ELSE (false)a := 1 ;IF a = 10 THEN Writeln (‘A is 10’)ELSE Writeln (‘A is not 10’) ;


CASE matcha := 3 ;CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ;END ;


CASE no matcha := 100 ;CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ;END ;


CASE…ELSE matcha := 3 ;CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ; ELSE Writeln (‘No match’) ;END ;


CASE…ELSE no matcha := 100 ;CASE a OF 1 : Writeln (‘A is 1’) ; 2 : Writeln (‘A is 2’) ; 3 : Writeln (‘A is 3’) ; ELSE Writeln (‘No match’) ;END ;


WHILE 2 iterscntr := 1;WHILE cntr <= 2 DO cntr := cntr + 1 ;


REPEAT 3 iterationscntr := 1 ;REPEAT cntr := cntr + 1 ;UNTIL cntr > 3 ;


FOR…TO 6 itersFOR cntr := 1 TO 6 DO Writeln (cntr) ;


FOR…DOWNTO 6 itersFOR cntr := 6 DOWNTO 1 DO Writeln (cntr) ;


Sequence & Nesting Clue to nesting is in the drone If you hear a continuous note then you

are inside a construct. Each construct has a drone. For nested constructs you hear multiple

drones simultaneously Drone pitch increases as nesting

deepens


Example programProgram ExemplarVAR counter : Integer ;BEGINcounter := 1 ;WHILE counter <= 2 DO BEGIN IF counter MOD 2 = 0 THEN Writeln (‘Counter is even’) ; counter := counter + 1 ; END ;END.


Scoreiteration open WHILE drone

True

counter := counter + 1;

selection open

FalseIF drone

selection close

WHILE condition

True

IF drone

exit IF

Writeln...sleighbell denotes

terminating conditionfor WHILE

False exit WHILE

iteration close

closed triangle

Auralisation score of aprogram. Notice that

the drones for eachconstruct appear

together on the stafflabelled 'Drones'.

Likewise, all percussiveevents for each

construct appear onthe 'Drums' staff.

Shifts between majorand minor are shown

by changes in keysignature.

1999 Paul VickersAll rights reserved counter := 1

True

counter := counter + 1;


Score


Structure DiagramProgram Sales

Prepare file Process Sales

Sales Record

Valid Sale Invalid Sale

Test sale Read next

Close file

*

o o


Experiment 1 Listening test to assess

comprehensibility of motifs 22 subjects, 60 auralisations Recognition rate better than by

chance and with very little training Hierarchic organisation allowed class

level identification & abstraction


Experiment 2 Debugging exercise 22 subjects, 8 programs, 4

auralised, 4 normal Results suggest technique is

useful, especially for complex programs and where program output gives few clues

Identified avenues for further research


Limitations No comparison with visual task yet Cognitive latency of motifs? No longitudinal results


Future work Follow-up studies Extend the grammar to include

assignment statements and procedure calls– How far? Bugs not manifested in program

flow are v. hard to detect and generally require variable inspection to locate, which is impractical for auralisation techniques


Future work Port cAitlIn to MS-Windows and

conduct a longitudinal study Assistive technologies. How can MPA

aid blind programmers?


Future work Preparing EPSRC bid to investigate

Auditory External Representations (c.f. EPSRC CRUSADE project at COGS

in Sussex - du Boulay et al)


Publications–Journals & Reports Vickers, P. and Alty, J. L. (2002). When Bugs Sing,

Interacting with Computers in press. Vickers, P. and Alty, J. L. (2002). Musical Program

Auralisation: A Structured Approach to Motif Design, Interacting with Computers in press.

Vickers, P. and Alty, J. L. (2002). Using Music to Communicate Computing Information, Interacting with Computers in press.

Vickers, P. (2001). Using the World-Wide Web as a Platform for an Interactive Experiment, Liverpool: Liverpool John Moores University, Technical Report CMS-21, ISBN 1-902560-050.


Publications–Conferences Vickers, P. and Alty, J. L. (2000). Musical Program

Auralisation: Empirical Studies, in Proc. ICAD 2000 Sixth International Conference on Auditory Display (Atlanta, GA, 2-5 April, 2000), International Community for Auditory Display, pp. 157-166 ISBN 0-9670904-1-5

Vickers, P. and Alty, J. L. (1998). Towards some Organising Principles for Musical Program Auralisation, in Proc. ICAD '98 Fifth International Conference on Auditory Display (Glasgow, Nov 1-4, 1998), British Computer Society ISBN 1 902505 05 0

Alty, J. L. and Vickers, P. (1997). The CAITLIN Auralization System: Hierarchical Leitmotif Design as a Clue to Program Comprehension, in Proc. ICAD '97 Fourth International Conference on Auditory Display (Palo Alto, November 3-5, 1997), Xerox PARC, Palo Alto, CA 94304, pp. 89-96


Publications–Conferences cont. Alty, J. L., Vickers, P. and Rigas, D. (1997). Using Music as

a Communication Medium, in Proc. Refereed Demonstrations, CHI97 Conference on Human Factors in Computing Systems (Atlanta, GA, March 22-27, 1997), ACM Press, pp. 30-31 ISBN 0-8979-926-2

Vickers, P. and Alty, J. L. (1996). CAITLIN: A Musical Program Auralisation Tool to Assist Novice Programmers with Debugging, in Proc. ICAD '96 Third International Conference on Auditory Display (Palo Alto, Nov 4-6, 1996), Xerox PARC, Palo Alto, CA 94304, pp. 17-24

the caitlin system musical program auralisation & external representations of pascal

Documents

program state

musical framework

musical grammarextended

musical equivalent

musical grammartransliteration

new grammar

visualisation techniques

real information