project essay

May 2015

Daniel KirkThe capabilities of current technology in relation to interactive video chat that can be used to promote music performance.

Southern Cross University – Independent Project

Music Performance in Live Interactive Digital MediaMay 13, 2015

IntroductionThis project examines the capabilities of digital media and current technology in relation to live interactive music performance. Its outcomes could lead to the promotion of music performance specified for different artist media desires. This could allow musicians to video call live performances to a fan/producer/ manager/employer in a professional manner. This technology could open up job opportunities for bands and allow stronger communication for global bands. Through the use of modern consumer video chat software this project analyses the technological aspects of music performance.

The aim of this project was to create an amateur and professional audio-visual interactive video stream based around musical performance and interaction. The objectives were to create interactive streams using different audio-visual sources. This project had a stronger focus on the audio aspects than the visual sources, looking at a range of different audio interfaces that would produce different results. The audio-visual synchronisation was another key objective, with the desire to see audio interfaces synchronise with video chat software. Musical performance interaction and Ipad/Iphone interfacing will be studied by looking at similar documents that will help identify its current possibilities and future probabilities.

The method of study was as a qualitative research project, achieved by the personal observation of experimentation, the analysis of online document releases related to this field of study and the use of visual-media recording. This project analysis begins with providing research into the history of video chat technology, motive for choice of video chat software and current technological similarities to the projects aim. The study of different audio interfaces is then presented by investigating different performance factors within the interactive video call.

History of Video ChatVideo conferencing/chat dropped into this world displaying futuristic quality. From the picture phone to then be video was a huge achievement, however most of the world didn’t know about it. Prices for using this technology in the 1980’s were incredibly high. When it was initially brought to the commercial market the product cost $250,000 with lines that cost $1000 per hour. As the technology improved prices dropped down to $20,000 in the 90’s. As Video conferencing developed in the business world, the consumer world still hadn’t seen an appealing product until Samsung released their MPEG-4 streaming 3G video mobile phone. Webcams then became a huge hit intergrading Mac computers and selling at only $99. Since the Internet boom in the 2000’s, video chat has increased in quality exponentially. All technology brands were releasing new and improved webcams, mobile phones were becoming video based and products such as Skype were appearing. The Iphone 4 made the huge hit in the consumer video chat world. With a front-facing camera, big screen and software that makes

2/12


video chat simple. Facetime provided an easy way to chat over video on your phone.

Video Chat SoftwareFor the purposes of this project, I chose to use the video chat software Skype and Facetime. Skype was chosen because of how popular it is. Having been one of the first video chat companies that captivated consumers, it proved to be one of my best choices. I used facetime because of its simplicity with Apple products, which is helpful in this project. In my experimentation, I initially used Facetime and found the video quality to be fairly consistent, however when using audio interfaces I had trouble receiving sufficient audio signal. While facetime had a very simple setup, it didn’t allow many options in changing its audio-visual settings. From there I moved onto Skype and found it to be very helpful with what I was trying to achieve. Skype’s audio-visual options are exceptional and even gave me an audio signal bar so I could see if the audio interface was working properly.

Audio-Visual StreamingOne of the key characteristics of this project is interactive chat featuring music performance. A simple alternative to this is video streaming that lacks interaction. Video streaming is gaining ground in the music industry today. More and more live performances are streamed online, however they lack personal interaction. My project was designed to achieve personal interaction with music performance at the centre.

Musical Interaction And Its ChallengesA current and working technology for interactive musical performance is just via audio. Programs such as Jammr, sofasession, and Jamulus allow real-time audio jamming over the Internet. These programs claim that they have real-time interaction. Musicianlink states that research done at Stanford University have shown that musicians can play in sync as long as the latency is below 25ms. Philip Rosdale from High Fidelity (2013) was attempting to achieve interactive audio synchronisation over Second Life, and also found that latency is required to be roughly around 20ms. This project uncovers why we can’t have musical interaction in an audio-visual environment.

My plan for interaction within this project was not to achieve performance interaction but to explore the research and future possibilities. With the existing knowledge that there is too much latency for a performance interaction within consumer software, I researched its reasoning. In 2012 Skype made a video promoting group chat (2012). To sell the idea they had rock band Scarlet Grey group chat a performance featuring members in London, New York and Los Angeles. After watching the video I deemed it unlikely for it to be possible using Skype and my claim is backed up by an in depth article written by Matt Libera, a digital media specialist (2012). He pointed out three things that would make such a thing possible.

3/12


1. “ Specialized software to minimize latency introduced by encoding / decoding time (LOLA)

2. Powerful computer (with custom cabling to the A/V equipment) to perform the encoding / decoding as quickly as possible – custom-built

3. A high-performance network (Internet2) with enough bandwidth to handle the massive traffic flow between locations.” (2012)

To provide such high quality audio and video Skype would have to make huge changes to their software, unleashing the chains that are set on the bandwidth limits that I’ve experienced within my project experiments. The possibility of this video being true and accessible for society is where my projects future leads. Libera goes on to state that,

“If this were somehow made possible, video quality and audio quality would be so degraded that it wouldn’t even be worth listening to it live”.

Within my own experiments I witness video and audio being degraded because the software and network couldn’t handle HD data. An example of true evidence based interaction is shown through a technology known as LOLA (LOw LAtency) that uses internet2, a new super-fast Internet used by educational and research facilities. Earlier this year the technology was used in Scotland with the Hebrides Ensemble, a collection of world-class musicians. A member of this ensemble James Coutts explains,

"It’s much, much faster than Skype or FaceTime, it’s a high speed network without time delay meaning musicians can play as if there was someone in the room performing right along with them." (2015)

Internet2’s partner research network in Australia named AARNet has performed demonstration in Australia featuring LOLA technology. AARNet’s CEO Chris Hancock said,

“LOLA has been designed to virtually eliminate this latency and enables musicians located potentially thousands of kilometers apart to perform together in a natural way, as if they were in the same room”.

This shows that current technology certainly provides these opportunities, however it’s not at a level or advanced enough where consumer society can use the technology, which is similar to the initial introduction of video chat and the costs of using it. Educational and scientific facilities are however using and improving this technology so that soon global musicians can collaborate in real-time.

The Project

4/12


Each of the project experiments was designed to answer four questions to uncover the capabilities of live interactive video streams. These questions include:

The difficulty of setting up the video chat with music performance What microphones were used and why? What are the capabilities of mixing the audio? What is the quality of sound received?

I identified these questions as to find out current consumer technology limits and how and where future technology may lead towards. To accurately answer these questions I recorded the video chat from the receiving computer.

Internal MicrophoneThe first audio-visual setup examined is the internal mic of an iMac. This is the standard mic that consumers would use for a video call, however I questioned its use in music performance. The internal mic has the easiest setup of all possibilities, because there is no signal routing required, as it’s the default microphone for the computer. The signal flow of this experiment is shown below.

The big question is, is this microphone worthy of high quality recording? Unfortunately Apple doesn’t disclose the technical specs about their built-in microphone. Christopher Breen, a senior Editor of Macworld (2014) stated in that the microphone “was designed with speech (and FaceTime) in mind”. From my own use of the microphone and Breen’s knowledge, the mic would be a small condenser with a frequency range that is centred on the voice. It would be a cheap/low quality microphone because it only needs to pick up a conversation.

Within my experiment I played a guitar and sung to examine its mixing capabilities. As its only one microphone there is no possibility of mixing the audio so that all instruments sound well together. Adjusting the gain of the microphone is simply done in sound preferences but no mixing possibility.

Through this analysis of the built-in microphone it is clear that it doesn’t provide high quality audio within a video chat for music performance. Due to its many limitations it doesn’t reach the audio quality I am looking for that other audio interfaces many achieve. Another difficulty with this setup is that you need to be close to the computer for the audio to come through effectively. Other audio setups will show that the capturing computer can be anywhere and even moving with the audio remaining consistent. This setup however would be great for someone who doesn’t have access to an external audio interface and microphones. It could be used successfully in music lessons.

5/12

Built-in Camera

Built-in Microphone

FacetimeReceiving Computer

Internet


The iMac built-in camera provided the visual component of this setup. All the setups use this camera because I found it to be excellent quality as long as the network is operating efficiently. The difficulty with the visual presence is the Internet speed. If the computer knows that the Internet can’t handle streaming HD video, it drops the quality down automatically that causes sudden changes in quality that commonly happen in consumer video chat programs.

Digidesign 003A setup I tried was using an external audio interface to achieve better audio quality. The better audio quality would come from the addition of have independent microphones, with the capability of mixing the audio signals. For this experiment I used the Digidesign 003 rack interface. This interface offers:

8 analog audio inputs (4 with mic preamps) with A/D converters supporting up to 24-bit, 96kHz audio

48V phantom power on mic preampsThis audio interface is starting to be outdated but still does a fantastic job in recording audio. In the test I used two microphones, one for the acoustic guitar and one for the vocals. The setup was quite simple to use because the audio interface had already been installed, so I could plug and play. To get the interface working with facetime I had to change the sound settings so that the input would be taken from the audio interface. The signal flow of the setup can be seen below.

The microphones used in this study were an AKG C3000 and a Shure sm57. Compared to the built-in microphone on the computer these microphones are a great deal better. A Sound On Sound review (1994) summarises the C3000 as,

“An excellent mid-price studio microphone capable of creating the classic AKG vocal sound”. (1994)

With the audio interface being used I thought this microphone would be an excellent choice price wise. The second microphone I used is considered the industry standard mic for instruments. A review at the Wire Realm (2014) stated,

6/12


“The SM57 is one of Shure’s best dynamic microphones as backed up by numerous years of use by ourselves and others around the net. There’s a reason you see those 4 or 5 star review all over music websites”. (2014)

I first tested the C3000 attempting to get signal flow from the interface to Facetime. From there I used the one mic to pick up both the guitar and vocals. Unfortunately there isn’t a way to mix the audio with one microphone therefore I moved on to use both the C3000 and the 57. Two microphones gave me the opportunity to turn the gain up or down of each microphone according to what was required. Due to it being a video call you cant actually hear your own sound so it is difficult to mix on your own. Obviously the receiving end could tell you what is too loud, or you could use pro tools or another audio recording program and create a mix by adjust your gains, find a good level and use that in the video call.

Through the use of microphones I began to learn that perhaps consumer video chat programs weren’t able to process the quality of audio being given through the microphone and audio interface. The video and audio already gets compressed so it can travel to the receiving end as quickly as possible therefore the program would have a bit rate and kHz limit that cannot be exceeded. The audio quality had improved compared to the built-in microphone, each instrument was a lot more defined, however facetime struggled with sending a clean signal.

Studio A/HD ThunderboltThis next setup was my largest, where I was aiming at using a 4-piece band in a Skype session. This experiment took place in Studio A of SCU’s recording studios. This room contains an Amek Langley Big analogue console, with the I/o’s coming from two AVID HD thunderbolt audio interfaces. The manufacturer applauds this product on its “premium A/D and D/A conversion and advanced circuitry, giving you pristine sound quality and clarity and the lowest possible latency”. (2015) Thunderbolt is the latest technology in high-speed data transfer. Past experiments have used firewire connections, however Thunderbolt 2 runs at over 20x quicker than firewire. This means recordings can be done in a higher quality. Apple describes their creation as “a revolutionary I/O technology that supports high-resolution displays and high-performance data devices through a single, compact port”. (2015) While Thunderbolt provides excellent audio quality, can Skype transfer high-quality audio?

This experiment contained two avenues of study. The first area of study was getting the thunderbolt connection and analogue desk to send signal to Skype. The signal flow is shown below.

7/12


This setup isn’t overly difficult to do since everything is already in place. All that is required is for Skype to select thunderbolt as its input, gain the microphone on the desk and away you go. For this experiment I attempted to DI my guitar and have that signal go to Skype. While the sm57 was working, the DI didn’t. After a number of experiments I came to the conclusion that Skype couldn’t handle anymore than 1 input coming in. This of course doesn’t help with mixing so to solve mixing guitar and vocals, microphone position is crucial. In the one microphone test I used a small diaphragm condenser to pick up both guitar and vocals. This worked quite well and produced a better sound quality than the built-in microphone did. However, still not as pure a signal as I was hoping for, it contributed to evidence that consumer video chat software cannot output high quality audio. Studio A/Pro ToolsThe second and more difficult task was getting the signal into Pro Tools, then from Pro Tools into Skype. If successful, this would allow for each instrument to be equalised, compressed and have other effects added to them to produce a better sounding performance. Below shows my initial plan for the signal flow.

This flow presents an external solution to routing the audio through pro tools. I got the idea from my own Macbook Pro, where its 3.5mm headphone port can be changed for sound input. Unfortunately the new Macs don’t have this option at

8/12


all so I couldn’t input the audio back into the iMac using pro tools. To complete this experiment I had the audio come into my Macbook instead. For this test I had a guitarist plugged in through a DI, and a cajon drum miked with a Beta 52 and an sm57. This setup would truly test Skype’s audio limits through use of the DI which didn’t work in past experiments and the added drum that produces low frequencies.

After much testing and tweaking I was able to have audio stream through Skype with no defect. This result shows that mixing capabilities are extensive. Pro tools plug ins can be added, and a studio engineer and mix the live signal that can then be outputted to the Skype session. The quality of sound for this experiment was certainly improved compared to other experiments. However it still sounded as though Skype is crushing the quality down to a level that can be transferred quickly. Joe Weed, a producer, engineer and musician stated that,

"A faster Internet could allow me to receive better audio that I could monitor in my control room and then make decisions about even subtle musical elements, including fine tonal or spatial differences”. (2015)

Due to technology restrictions of Internet upload and download speed, high-speed transfer of HD audio and video not be possible.

As was investigated earlier, HQ audio interaction is currently impossible with consumer products, however audio-visual interaction requires quicker speeds, and the popular video chat products are focused on increasing video quality, not audio quality.

The other option for using pro tools was an internal signal flow, as shown below.

It was proposed to me by an audio technician that the signal could possibly to go from protools to Skype internally by using a program called Soundflower. Its maker Rogue Amoeba defines Soundflower as,

“A Mac OS X kernel extension that enables audio to be passed between applications. It acts as a virtual audio device, with both inputs and outputs. With it, any audio application can send and receive audio”. (2015)

9/12


With much work, research and testing I was unable to get Soundflower to communicate with pro tools and Skype. I tested using Soundflower on the Studio A thunderbolt connection and on a small 1 input mbox audio interface but both didn’t work. The external connection test however gave me all the information I required to diagnose the capabilities of interaction via video chat software.

Mobile DevicesThe use of mobile devices such as Ipads/tablets or smart phones is becoming more and more popular. Unfortunately there were no experiments done on the use of these products, however from tests already done we can provide results on key factors of using these products. The proposed signal flow for the use of an Ipad is as follows.

Interfacing this setup would require an IOS audio interface such as the IK Multimedia iRig HD Interface. Assuming that facetime would interact with the input, this interface could do an excellent job. This particular interface is described as,

“the first high-quality 24-bit audio interface that fits in your pocket and connects directly to your Lightning, 30-pin and USB devices”. (2015)

The audio would output from an interface and input into the Ipad via the IOS interface. Advantages of using the Ipad include the being able to carry the Ipad around to gain a different feel within the video call. If there is a band playing, the person carrying the Ipad could get close ups of each musician as they perform allowing the viewer to feel part of the performance. A downside of using an Ipad is that it may not be able to process high quality audio. If a Macbook Pro and iMac struggle to process the high bit rate within Skype, there isn’t much hope for an Ipad. But overall it comes back to Skype and Facetime not being designed to process HQ audio.

The camera on the Ipad/Iphone is just as good as computer cameras nowadays. The Ipad Air 2 boasts a 1.2MP sensor that captures 1080p HD video. The issue of using an Ipad is the Internet connectivity. The Ipad would be connecting to the

10/12


Internet via wifi, which could affect the video and audio quality, compared to computers connected by Ethernet. Overall, the Ipad/Iphone being used as a mobile video device with high quality audio could add new elements to a live video musical performance. The only issues at hand are the connectivity of the device and the quality it produces compared to the other audio-visual setups studied.

Concluding RemarksTo conclude, the aims of this project have partly been achieved in the creation of an amateur interactive video stream, however professional quality proved difficult for the chosen consumer software, but its capabilities were analysed through document studies and articles featuring that specific current technology available for education and corporate circumstances. The research methods established that social video chat software is unable to process high quality audio streams and since video streams are made priority for Internet transfer, audio quality is degraded. Research has shown that technology specific for musical interaction is current but limited.

Overall, the examination of different audio interfaces, mobile device possibilities, musical interaction limitations, and related document studies, demonstrates the capabilities of current and future technological possibilities of interactive digital media data streams within the discipline of music performance.

References:

11/12


MusicianLink, jamLink frequently asked questions [Online]Available from:https://www.musicianlink.com/jamlink-frequently-asked-questions

Rosedale. P (2013) How much latency can live musicians tolerate? [Online] Available from: https://highfidelity.com/blog/2013/05/how-much-latency-can-live-musicians-tolerate/

Breen. C (2014) Understanding the limitations of a Mac’s microphone [Online]Available from:http://www.macworld.com/article/2367142/understanding-the-limitations-of-a-macs-microphone.html

White. P (1994) AKG C3000 Review [Online]Available from:http://www.soundonsound.com/sos/1994_articles/jan94/akgc3000.html

Sean (2014) Shure SM57 Dynamic Microphone Review [Online]Available from:http://www.wirerealm.com/reviews/shure-sm57-dynamic-microphone-review

Avid (2015) Pro Tools HD I/O [Online]Available from:http://www.avid.com/US/products/HD-IO

Apple (2015) Thunderbolt: The most advanced I/O ever [Online]Available from:https://www.apple.com/au/thunderbolt/

Store DJ (2015) IK Multimedia iRig Guitar Interface for iPad & iPhone [Online]Available from:https://www.storedj.com.au/products/IK-IRIGHD

Rogue Amoeba (2015) Soundflower (Online)Available from:https://rogueamoeba.com/freebies/soundflower/

Skype (2012) Group video calling with Skype: explore the possibilities [Online]Available from:https://www.youtube.com/watch?v=vQzyUdTn0cQ&feature=youtu.be

Libera. M (2012) 3-way live performing with Skype? Yeah, right. [Online]Available from:http://performingarts.uncg.edu/patech/3-way-live-performing-with-skype-yeah-right/

Piper. L (2015) Groundbreaking ‘LOLA’ technology to bring world orchestras to Scotland [Online]Available from:http://edinburgh.stv.tv/articles/315083-lola-technology-to-bring-worlds-top-orchestras-together-in-scotland/

Krieger. L, San Jose Mercury News (2015) Speed of light: music to our ears [Online]Available from:http://www.columbian.com/news/2015/feb/12/speed-of-light-music-to-our-ears/

12/12

project essay

Documents