voice quality and testing eit-03 alan percy director of business development audiocodes

Voice Quality and TestingEIT-03

Alan PercyDirector of Business

DevelopmentAudioCodes

Who is AudioCodes?• Leading OEM Manufacturer

of Enabling VoIP equipmento Gatewayso Media Serverso Boards, Chips and Modules

• Key IPR holder for G.723.1• 11th Year in Business• $82.8M Revenue for FY04• NASDAQ: AUDC• Over 440 Employees• Worldwide Company:

o Headquarters in Israelo US offices in San Jose, Boston, Chicago, Dallas,

RTP, and Somerset NJo Offices in Mexico, France, UK, China, and Japan

Background

• Customer demand is building

• VoIP carriers are proliferating

• What will differentiate the carriers?

Consumer VoIP Subscribers

0

2

4

6

8

10

'03 '04 '05 '06 '07 '08

Year

in m

illio

ns

Source: In-Stat

New Carriers, New Challenges

A few years ago, we equipment providers thought we had this figured out.

Then:• Private networks• Intra-Enterprise• Carefully controlled

(over-engineered) networks

New Carriers, New Challenges

Today: New Frontier• Depending on existing broadband

o Cableo DSLo Fixed wireless

• New Wireless Serviceso 3Go WiMaxo WiFio SAT

What have we learned?

You can’t always control the networko To make the business model work, you

sometimes need to leverage the existing infrastructure

What have we learned?

The last mile is the hardest mile!

What have we learned?

• Murphy’s law is true!(Poor voice quality will only affect the CEO

during an important investor call)

• Poor voice quality will get you kicked out

• Huge variation in voice quality between equipment vendors(Especially when network conditions are

not “ideal”)

Is voice quality important?

“Voice Quality is the most sensitive issue for business customers” Greg Schreiber, VP of Product Development - Vaspian

What affects voice quality?

Choosing the Right Coder

Vocoder Factors

• Single speaker, no packet loss, no acoustic noise

• Vocoder quality given non speech audio (music, background noise)

• Bit rate • Algorithmic delay = Packet size + look

ahead • Audio bandwidth • Low bit rate algorithm type • Complexity (# MIPS)• Intrinsic robustness to packet loss

Vocoder Factors

Vocoder implementation issues:• Not all vocoders standards have bit exact

test vectors• And test vectors don’t cover every angle …• Little known fact - even long approved and

deployed vocoder standards still have bugs – let alone new ones …

o Audiocodes has found and reported bugs in numerous standard vocoders

• Hence the importance of Speech Quality Testing and deployment experience

Vocoder Tradeoffs

• Complexity is always a constraint• Bit rate and algorithm type affect speech

and non-speech vocoding quality • Larger packet size enables lower bit rate

but increases delay (e.g. G.723.1 vs. G.729A)

• Higher bit rate enables better intrinsic robustness to packet loss

• Higher audio bandwidth requires higher bit rate

Echo Canceller Factors

• Imperfect EC performance could result in:

o Residual echoo Clippingo Distorted speecho Annoying variation in background noise

• Longer round trip delay makes echo cancellation more critical

Echo Canceller Factors

• ECs are challenged most during Double Talk (DT) o Double Talk happen occasionally during

full duplex conversationso During Double Talk

• EC may diverge • The Non Linear Processor (NLP) could clip,

attenuate or distort the near-end speech• Therefore it is crucial to disable the NLP during

Double Talk

o Timely and reliable detection of Double Talk could be tricky

Echo Canceller Factors

• Background noise continuity (level, spectrum) should be maintained

• The variety of real life echo paths poses an algorithmic and testing challengeo field experience crucial

• EC standards (e.g. G.168) specify testing methods - not standard algorithms o proprietary EC algorithms could vary

significantly in qualityo Quality testing is essential

Echo Canceller Tradeoffs

• Attenuation vs. tail length vs. adaptation speed vs. steady state coefficients noise and EC stability

• Echo Attenuation in Single Talk vs. Double Talk performance

• Complexity (# MIPS)

Packet Loss Concealment

• Vocoder type (CELP vs. ADPCM vs. PCM)

• All vocoders (including G.711) have PLC standardized to some extent

• Vendor distinction depends on Jitter Buffer quality and PLC enhancements

• Trade offso Latency vs. Jitter buffer size vs. %

frame erasure rateo Redundancy schemes vs. bit rate

Condition Interaction Effect

Effect:• Customers tell us that G.723.1 many

times sounds better than other’s G.711

Why?• Lower bit rate reduces congestion and

results in fewer dropped packets.• G.723.1 also includes packet loss

concealment, smoothing over the gaps.

Improving VoIP Quality

Techniques vendors use to maximize quality:• Dynamic Jitter Buffer

o Adjusting to ever-changing network conditions

• Packet loss concealmento All coders, including G.711

• Echo cancellation algorithms• VAD and CNG algorithms• Extensive field and lab testing

Speech Quality Testing

• Methods• Events• Results

Speech Quality Testing Methods

• MOSo Traditional subjective method, uses a panel of

listeners, accurate but expensive

• PESQ - Perceptual Evaluation of Speech Quality o objective method based on human hearing model, ITU

P.862 standardo Computer driven

• G.168o ITU standard for echo canceller testing, uses artificial

speech signals• P.501 / P.502

o ITU standard for complex test signals used to test echo cancellers and other modules.

o These signals were designed to provide more information to the designer about the weakness points.

ETSI Speech Quality Test Event

• Sponsored by ETSI - Done by HEAD Acoustics Labs • Has been held several times over the past three

years• VoIP equipment testing in controlled conditions • Receive an objective and comparative analysis • The tests include a range of aspects, including:

o Speech sound quality o Echo measurements o Double Talk performance o Transmission quality in the presence of background

noise

• Anonymous - Only your results are identified

ETSI Test Setup

G a t e w a y

I P

E 1

N I S T N e t

P a c k e t M O N I T O R

E 1

P a c k e t L o s s D e l a y

O u t p u t

H E A D a c o u s t i c s T e s t S y s t e m AA CC QQ UU AA

I n p u t

G a t e w a y I P I P

I S D N D S S 1 I S D N D S S 1

P B X

ETSI results “Pie Chart”• Six conversational and six listening tests

• Ten gateways participated in 3rd SQTE

• The results for each gateway are summarized in a Pie Chart with a slice per each category

• The performance in each category was represented by the radius of the slice and by a color code

o Red – the amount of which the results were under the requirement

o Yellow – result was below the requirement if accompanied with red or OK if no red

o Green – result was above the requirement

ETSI SQTE Test Results

ETSI SQTE Test Results

AudioCodes

How can you use this information?

• Ask for ETSI results when shopping• Read and evaluate performance

needs against test results.• Justify partner selection• Help sell your solution

Where to learn more:

Full ETSI SQTE Test Report:http://www.etsi.org/plugtests/History/2004SQTE.htm

AudioCodes ETSI SQTE Results:http://www.audiocodes.com/Content.aspx?voip=2189

Let’s talk! Stop by Booth #405