Decoding Digital Audio Visualizing and Annotating Linear Time-Based Media

Phil Desenne Center  for  Hellenic  Studies,    

Harvard  University

May 8, 2015

To decode or interpret audio is to explain the meaning

or understanding

of something about it

Relevant for Research, Teaching and Learning across all disciplines

Decode –> Interpret

Curiosity - Discovery - Interpretation - Research

Amateurs - Learners - Educators - Scholars

Each one decodes audio in their own realm

the process transcends realms and roles

}}

Songs

Music

Voice recordings

Field recordings

Lyrics / Transcription / Translation

Notes / Musicology / Ethnomusicology

Oral History / Languages / Speech Therapy

Bioacoustics Research / Anthropology

Audio Visualize and Annotate

{}Listen Decode

In our day-to-day we are constantly, filtering, decoding

and attaching meaning in our brain to daily sound

bites that hit our ears

!

digital audio opens a broader spectrum of

decoding possibilities for

research, teaching and learning

Visualization and Annotation of Audio

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nam in augue sodales nisl pharetra efficitur in in felis. Ut malesuada justo nec libero finibus placerat. Donec vitae enim risus. Nunc eget purus eget nunc bibendum tempus hendrerit vel eros. Praesent mollis diam augue, vel convallis quam interdum eu.

Transcript

we now have digital tools that facilitate and enhance the process of decoding, attaching

meaning and understanding

Audio Annotation & Visualizations in

Research

Acoustics / Bioacoustics / Hydroacoustics

Anthropology / Musicology / Ethnomusicology

Speech Therapy

Education / Teaching

Importance of Visualizing Sound:

Human ear does not hear or discern all sounds

www.cochlea.org739 × 252Search by image Frequency hearing range in man and some common animal

very high-frequency

sounds

very low-frequency

sounds

}visualization –> accessibility

Recording Natural Sounds www.leaps.ms900 × 396Search by image

You can also use Raven Lite to slow down natural recordings so that the full complexity of a song may be heard. Listen to the complexity in the ending trill ...

Bioacoustics: detection and interpretation of sounds in animals

Annotating the visual wave form of audio: Amplitude and Frequency

BIRD SONGS AND CALLS WITH SPECTROGRAMS ( SONOGRAMS ) OF SOUTHERN ... www.birdsongs.it489 × 396Search by image

Fig . 2 shows a 20 second fragment over 1-minute song sequence of a Cirl Bunting (Emberiza cirlus). Fig. 3 blows up one phrase of the same song and shows ...

Sonograms and Spectrograms of Bird Songs

The Mind's Machine - Chapter 15 A Step Further www.mindsmachine.com800 × 619Search by image

(a) These sonograms show the typical adult song patterns of two sparrow species. The songs illustrated in part (b) were produced by males reared ...

Species identification and animal behavior

WarblerWatch: Warbler Guy, where do I learn about "reading ... warblerwatch.blogspot.com650 × 578Search by image

Ergo, you'll quickly have no problems identifying a song sparrow classic song via its sonogram in comparison to a common yellowthroat's, and

so on.

Teaching and Learning with audio annotations

Examples at Harvard

Prof. Tom Kelly, First Nights course, Harvard College

Music Courses: Annotated Interactive Play-throughs

Learners explore music

Foreign Culture Courses: Listening Guides

Prof. Richard Wolfe, Foreign Cultures Course, Harvard College

Foreign Culture Courses: Listening Guides

Prof. Richard Wolfe, Foreign Cultures Course, Harvard College

Table text

XML text

Listening guide player

Harvard iSites CAT tool

Prof. Richard Wolfe, Harvard College

Research Teachingdrives

Research Teachingdrives

Prof. Richard Wolfe

Ethnomusicology research in South, Central and West Asia

Courses in Ethnomusicology at Harvard College

Audio annotations

richardkwolf.com www.music.fas.harvard.edu/faculty/rwolf.html

Repositories

Media and Annotation Metadata

How does it all connect ?

Annotation Meta layer

Visualization layer

Audio layer (URI)

Annotation DB

Media Repositories

ideally under same repository entities

Media Search

Open Annotation Data Model

Client tool !

Interoperability: Tying it all together

Persistent Annotation Meta-layer Open API Access

Stable Digital Repositories URNs resolving to URLs

Ephemeral Tools / Content / Learning Management Systems

Open Annotation Model

Public Archives

Other Institutions Archives

Open Source

Museums

HUL DRS

Research Database

Catalyst, LibraryLab

HILT ...

Course & Student Content

Peer Researchers

Personal Research & Archives

Subject Experts

Incubator Projects

Persistent Annotation Repositories

Individual Repositories

External Repositories

Internal RepositoriesAnn

Ann

Open Annotation Federated Systems

across all media

Future of audio annotation

• Searching: faceted, specific range target searches

• Semantic tagging: machine learning

• Automatic Annotations:

• transcription / translation • acoustic detection, • individual voice recognition • bioacoustics species id • AI -> Pairing crowdsourced data and automatic

annotation using semantic annotated data (OA)

• High definition audio and detailed audio analysis

• Collaboration and crowdsourcing tools

• Cross-referencing media annotations

Thank You ! Questions?

!

desenne[ at ]fas[ dot ]harvard[ dot ]edu