vowifi testing challenges

1Spirent Communications PROPRIETARY AND CONFIDENTIAL

VoWi-Fi TestingChallengesMeasuring the Quality of Experience in the field and lab


Agenda

Part 1: VoWi-Fi technology landscape Drivers for Wi-Fi adoption VoWi-Fi QoE Challenges

Part 2: SRG VoWi-Fi field-based study results Wi-Fi study background and motivation Study setup and measurement methodology VoWi-Fi Field test results

Part 3: Evaluating VoWi-Fi performance in the lab Advantages of lab-based testing Evaluating Key Performance Indicators (KPIs) in the lab Future deployment and test considerations Audience Q&A


Drivers for Wi-Fi adoptionWhy Wi-Fi is an attractive choice as a technology


VoWi-Fi QoE Challenges

Offloading Algorithms• When and why?• When to go back?

Assessing Signal Strength and Integrity• RSSI measurements before and

during offload

IP Layer• Best effort delivery• Access Point variations

Authorization• ePDG authentication for

unsecured Wi-Fi access


KPIs used to measure VoWi-Fi Quality of Experience (QoE)

Voice Quality: MOS Score – Uplink and Downlink Packet (Audio) loss percentage Conversational (Ear-to-mouth) delay

Call reliability: Percentage of successful handovers in both directions under “expected” circumstances Percentage of call drop under various ePDG adversarial scenarios (lab-based simulation) Percentage of successful handovers when access point is loaded Time taken for call setup (for Wi-Fi originated calls) Time taken for LTE<-> Wi-Fi handover


Background and rationale for the study

Spirent and SRG have collaborated on all-things-VoLTE for the last 15 months VoWi-Fi is the rage Although operators are rushing to embrace the technology – sometimes in

advance of VoLTE – there isn’t a clear understanding how the technology performs

VoWi-Fi is a natural extension of VoLTE, given the interoperability that should exist between the two voice solutions – also doesn’t hurt that HD Voice is supported across all 3 RAN components of the operator’s network


Study setup and test methodology

Spirent Communications provided Nomad User Experience system Accuver X-Cal M data collection tool and Accuver XCAP post-processing software

for logging all lower layer parameters The testing relied on mobile-to-mobile calls – implies that the uplink MOS of one

device can affect the downlink MOS of the other Testing included VoLTE-VoWi-Fi handovers and vice versa, as well as call quality

with different device configurations (3G only, Wi-Fi- only, etc.) VoWi-Fi voice quality measured under realistic traffic loading at residential and

public Wi-Fi access points


Procedures: Introduced additional background traffic onto same Wi-Fi access point while performing voice quality testObservations:Streaming HD Video (2x) had very little impact on voice quality when the downlink consisted of two video

streams

Field test results: residential Wi-Fi access pointHow does the performance look when a device using VoWi-Fi calling on a residential Wi-Fi access point is loaded on the downlink


Field test results: residential Wi-Fi access pointHow does the performance look when a device using VoWi-Fi calling on a residential Wi-Fi access point is loaded on the uplink

Procedures:Transferred emails with ~10 MB files attached over same Wi-Fi AP being use to support voice callsObservations:Relatively modest amounts of uplink traffic can impact call quality, including no voice detectionAs implemented, VoWi-Fi does not support QoS so all traffic is treated as best effort


Field test results: public Wi-Fi access pointHow does the performance look when two smartphones are using VoWi-Fi in a public Wi-Fi access point

Procedures:Stationary testing at airport with both smartphones using VoWi-Fi and the same SSID

– network seemed to be lightly loadedObservations:Bigger variances in the results and lower scores, despite lightly-loaded network

– VoWi-Fi on both ends increases impactQuality of network has a major influence on call quality


Field test results: public Wi-Fi access pointHow does the performance look when a device using VoWiFi calls another device on VoWiFi on a public Wi-Fi Access point

Procedures:Comparable to airport test but done at San Diego Convention Center – 18,000 attendees for a conferenceObservations: Less variance in the results compared with the airport testQuality of network has a major influence on call quality – in this case a good thing


Field test results: VoWi-Fi to VoLTE handoversHow does the performance look when a device using VoWi-Fi hands over from a residential Wi-Fi Access Point to VoLTE

Procedures:Stationary device, which is placed near Wi-Fi AP, uses VoLTE; Second device was mobile – started on VoWi-FiWalked down the street pushing the “little red wagon” until the call switched to VoLTEReturned back to Wi-Fi AP and waited for handover to Wi-Fi before walking again of Wi-Fi coverageObservations:No impact to MOS Score during handover; also hard to detect handover by listening to the callHandovers back into VoWi-Fi take longer (no surprise) while outbound handovers aren’t predictable


Field test results: VoWi-Fi to VoLTE handoversHow does the network and device activity look when a device switches between VoLTE and VoWi-Fi

Procedures:One device configured for 3G voice only; second device configured for VoLTE / VoWi-FiOriginate call within Wi-Fi AP coverage and then proceed to walk out of coverage with both phones in handContinue until handover occurs and then walk back into coverage; repeat as necessaryObservations: Everything consistent with expectations; lack of QoS with VoWi-Fi is evident


Field test results: VoWi-Fi to VoLTE handovers, IIHow does the network and device activity look when a device switches between VoLTE and VoWi-Fi

Procedures:One device (S5) configured for 3G voice only; second device (S6) configured for VoLTE / VoWi-FiOriginate call within Wi-Fi AP coverage and then proceed to walk out of coverage with both phones in handContinue until handover occurs and then walk back into coverage; repeat as necessaryObservations:S5 used UMTS as intended but S6 didn’t use VoWi-Fi, instead using a combination of UMTS and 2G (NB-AMR)Call drops when we returned to Wi-Fi coverage (repeated with same results)


Key observations

The benefits of VoWi-Fi are clear but it isn’t a panacea

VoWi-Fi is as good or as bad as the underlying Wi-Fi network

VoWi-Fi handovers with the LTE network generally worked but there were some exceptions

Problems, which we attribute to network/IMS registration and device interoperability, exist


Advantages of evaluating VoWi-Fi in the lab

Audio quality issues related to low MOS scores, audio (packet) loss and codec (AMR-NB/AMR-WB/EVS) negotiation failure

SIP registration issues ePDG authentication failure Handover at power levels higher/lower than that mandated by operator test plans Poor QoE resulting from UE toggling between LTE – Wi-Fi networks Does not require a wagon


Advantages of evaluating VoWi-Fi in the lab

VoLTE

VoWi-FieAP AKA

IPSec tunnel

VoWi-Fi

VoWi-Fi


Evaluating VoWi-FI KPIs in the lab

Test scenario Measurable KPIs QoE impact

Voic

e qu

ality

Test voice quality when the Wi-Fi power is in the edge condition of making a decision on handover from LTE-Wi-Fi or vice-versa and is toggling between the two

MOS Score Packet loss percentage Network latency Jitter

Lower MOS Score => lower voice quality

High packet loss => Intermittent drops in speech

High network latency => Variable delays in speechTest voice quality in the

presence of loading on the Wi-Fi access point and tolerance of UE to larger network delays

Test voice quality of two test devices against a reference “soft” client, both on the Uplink and the Downlink


Evaluating VoWi-Fi KPIs in the lab

Test scenario Measurable KPIs QoE impact

Cal

l rel

iabi

lity

Test the reliability of the call through the handover process and time for handover by control of Wi-Fi power over multiple iterations

Percentage of successful call completion

Handover time from LTE – Wi-Fi

Larger handover time => periods of voice discontinuity

Lower percentage of call completion => Increased dropped calls during handover

Improper response to ePDGadversarial scenarios => call drops during handover

Test the reliability of the call when the UE is sent adversarial responses by ePDG

Test the reliability of the call through the handover process and time for handover by control of Wi-Fi power over multiple iterations


Test case 1

Test voice quality between two devices Device A and Device B and measure MOS score on both Uplink and Downlink

Result

Low MOS score

measured on Uplink of

Device A

Test Case 1: Voice Quality

Low MOS score on Downlink of Device B in device-to-device calltraced to low Uplink MOS score on Device A


Test case 2

ePDGadversarial testing scenario

Result

LTE-Wi-Fi Handover

failure

UE fails to handover and drops call when ePDG issues error code

Test Case 2: ePDG responses


Test case 3

Test voice quality when the UE is toggling between Wi-Fi and LTE network

Result

Lower MOS Score

Test Case 3: Handover

Packet loss

Root cause identified: Packet loss as a result of device “toggling” between LTE-Wi-Fi


Test case 4

Call initiation on Wi-Fi originated call

Result

Low MOS score

Test Case 4: Codec negotiation

UE1 supporting AMR-WB UE2 supporting only AMR-NB

Root cause identified: Disagreement of codec between UE1 and UE2


VoWi-Fi test capabilities and requirements


Bringing field study into the labSRG and Spirent partner on evaluating VoWi-Fi QoE in the lab

LTE EPC Configuration IMS registration call flow Wi-Fi authentication ePDG Adversarial test condition IMS test agent simulation


Additional VoWi-Fi QoE considerations

Service request type Wi-Fi / LTE power level Capacity/loading on Wi-Fi radio link Measuring impact on voice quality while introducing fading and noise on radio

interface of Wi-Fi Testing audio quality and call performance with different classes of access points

(Home, Enterprise, public Wi-Fi hotspots) Testing performance in the presence of introducing IP impairments on the IP

backhaul Testing the impact of performance in the presence of interference from other

adjacent bands such as LTE and/or adjacent Wi-Fi hotspots ANDSF, Hotspot 2.0…


Questions?

vowifi testing challenges

Mobile