qoe evaluation: the triangle testbed approach
TRANSCRIPT
Research ArticleQoE Evaluation The TRIANGLE Testbed Approach
Almudena D-az Zayas 1 Laura Panizo 1 Janie Bantildeos 2
Carlos Caacuterdenas 2 and Michael Dieudonne 3
1University of Malaga Spain2DEKRA Spain3Keysight Technologies Belgium
Correspondence should be addressed to Almudena Dıaz Zayas adzumaes
Received 11 October 2018 Accepted 25 November 2018 Published 18 December 2018
Academic Editor Giovanni Stea
Copyright copy 2018 Almudena Dıaz Zayas et al This is an open access article distributed under the Creative Commons AttributionLicense which permits unrestricted use distribution and reproduction in any medium provided the original work is properlycited
This paper presents the TRIANGLE testbed approach to score the Quality of Experience (QoE) of mobile applications basedon measurements extracted from tests performed on an end-to-end network testbed The TRIANGLE project approach is amethodology flexible enough to generalize the computation of the QoE for any mobile application The process produces a finalTRIANGLE mark a quality score which could eventually be used to certify applications
1 Introduction
The success of 5G (the fifth generation of mobile commu-nications) and to some extent that of 4G depends on itsability to seamlessly deliver applications and services withgood Quality of Experience (QoE) Along with the user QoEis important to network operators product manufacturers(both hardware and software) and service providers How-ever there is still no consensus on the definition of QoEand a number of acronyms and related concepts (eg see [1])add confusion to the subject QoE (Quality of Experience)QoS (Quality of Service) QoSD (Quality of Service Deliv-eredachieved by service provider) QoSE (Quality of ServiceExperiencePerceived by customeruser) and so forth Thisis a field in continuous evolution where methodologies andalgorithms are the subject of study of many organisations andstandardization bodies such as the ITU-T
TRIANGLE project has adopted the definition of QoEprovided by the ITU-T in Recommendation P10G100(2006) Amendment 1 ldquoDefinition of Quality of Experience(QoE)rdquo [2]
ldquoThe overall acceptability of an application orservice as perceived subjectively by the end-userrdquo
In [2] the ITU-T emphasizes that the Quality of Experi-ence includes the complete end-to-end system effects client
(app) device network services infrastructure and so onTherefore TRIANGLE brings in a complete end-to-endnetwork testbed and a methodology for the evaluation of theQoE
Consistent with the definition themajority of the work inthis area has been concerned with subjective measurementsof experience Typically users rate the perceived quality ona scale resulting in the typical MOS (Mean Opinion Score)Even in this field the methodology for subjective assessmentis the subject of many studies [3]
However there is a clear need to relate QoE scoresto technical parameters that can be monitored and whoseimprovement or worsening can be altered through changes inthe configurations of the different elements of the end-to-endcommunication channel The E-model [4] which is based onmodelling the results from a large number of subjective testsdone in the past on a wide range of transmission parametersis the best-known example of parametric technique for thecomputation of QoE Also one of the conclusions of theProject P-SERQU conducted by the NGMN (Next Genera-tion Mobile Networks) [5] and focused on the QoE analysisof HTTP Adaptive Streaming (HAS) is that it is less complexand more accurate to measure and predict QoE based ontrafficproperties thanmaking a one-to-onemapping betweengeneric radio and core network QoS to QoETheTRIANGLE
HindawiWireless Communications and Mobile ComputingVolume 2018 Article ID 6202854 12 pageshttpsdoiorg10115520186202854
2 Wireless Communications and Mobile Computing
project follows also a parametric approach to compute theQoE
Conclusions in [5] point out that a large number ofparameters in the model could be cumbersome due tothe difficulty of obtaining the required measurements andbecause it would require significantly more data pointsand radio scenarios to tune the model The TRIANGLEapproach has overcome this limitation through the largevariety of measurements collected the variety of end-to-end network scenarios designed and mostly the degree ofautomation reached which enables the execution of intensivetest campaigns covering all scenarios
Although there are many proposals to calculate thequality of experience in general they are very much orientedto specific services for example voice [6] or video streaming[7 8] This paper introduces a methodology to compute theQoE of any application even if the application supportsmorethan one service
The QoE as perceived by the user depends on manyfactors the network conditions both at the core (CN) and atthe radio access (RAN) the terminal the service servers andhuman factors difficult to control Due to the complexity andthe time needed to run experiments or make measurementsmost of the studies limit the evaluation of theQoE to a limitedset of or even noncontrolled network conditions especiallythose that affect the radio interface (fading interference etc)TRIANGLE presents a methodology and a framework tocompute the QoE out of technical parameters weighting theimpact of the network conditions based on the actual usescases for the specific application As in ITU recommendationG1030 [9] and G1031 [10] the userrsquos influence factors areoutside of the scope of the methodology developed inTRIANGLE
TRIANGLE has developed an end-to-end cellular net-work testbed and a set of test cases to automatically testapplications under multiple changing network conditionsandor terminals and provide a single quality score Thescore is computed weighting the results obtained testingthe different uses cases applicable to the application forthe different aspects relevant to the user (the domains inTRIANGLE) and under the network scenarios relevant forthe application The framework allows specific QoS-to-QoEtranslations to be incorporated into the framework based onthe outcome of subjective experiments on new services
Note that although the TRIANGLE project also providesmeans to test devices and services only the process to testapplications is presented here
The rest of the paper is organized as follows Section 2provides an overview of related work Section 3 presents anoverview of the TRIANGLE testbed Section 4 introducesthe TRIANGLE approach Section 5 describes in detail howthe quality score is obtained in the TRIANGLE frameworkSection 6 provides an example and the outcome of thisapproach applied to the evaluation of a simple App theExoplayer Finally Section 7 summarizes the conclusions
2 State of the Art
Modelling and evaluatingQoE in current and next generationof mobile networks is an important and active research
area [8] Different types of testbeds can be found in the lit-erature ranging from simulated to emulated mobilewirelesstestbeds which are used to obtain subjective or objective QoEmetrics to extract a QoE model or to assess the correctnessof a previously generated QoE model Many of the testbedsreviewed have been developed for a specific research insteadof for a more general purpose such as the TRIANGLEtestbed which can serve a wide range of users (researchersapp developers service providers etc) In this section someQoE-related works that rely on testbeds are reviewed
The QoE Doctor tool [12] is closely related to the TRI-ANGLE testbed since its main purpose is the evaluation ofmobile apps QoE in an accurate systematic and repeatableway However QoE Doctor is just an Android tool that cantake measurements at different layers from the app userinterface (UI) to the network and quantify the factors thatimpact the app QoE It can be used to identify the causesof a degraded QoE but it is not able to control or monitorthe mobile network QoE Doctor uses an UI automationtool to reproduce user behaviour in the terminal (app userflows in TRIANGLE nomenclature) and tomeasure the user-perceived latency by detecting changes on the screen OtherQoE metrics computed by QoE Doctor are the mobile dataconsumption and the network energy consumption of the appby means of an offline analysis of the TCP flows The authorshave used QoE Doctor to evaluate the QoE of popular appssuch as YouTube Facebook or mobile web browsers One ofthe drawbacks of this approach is that most metrics are basedon detecting specific changes on the UI Thus the modulein charge of detecting UI changes has to be adapted for eachspecific app under test
QoE-Lab [13] is a multipurpose testbed that allows theevaluation of QoE in mobile networks One of its purposesis to evaluate the effect of new network scenarios on servicessuch as VoIP video streaming or web applications To thisend QoE-Lab extends BERLIN [14] testbed framework withsupport for next generation mobile networks and some newservices such as VoIP and video streaming The testbedallows the study of the effect of network handovers betweenwireless technologies dynamic migrations and virtualizedresources Similar to TRIANGLE the experiments are exe-cuted in a repeatable and controlled environment Howeverin the experiments presented in [13] the user equipmentwere laptops which usually have better performance andmore resources than smartphones (battery memory andCPU)The experiments also evaluated the impact of differentscenarios on the multimedia streaming services included inthe testbed The main limitations are that it is not possibleto evaluate different mobile apps running in different smart-phones or relate the QoE with the CPU battery usage and soforth
De Moor et al [15] proposed a user-centric methodologyfor the multidimensional evaluation of QoE in a mobile real-life environment The methodology relies on a distributedtestbed that monitors the network QoS and context infor-mation and integrates the subjective user experience basedon real-life settings The main component of the proposedarchitecture is theMobile Agent a component to be installedin the user device that monitors contextual data (location
Wireless Communications and Mobile Computing 3
velocity on-body sensors etc) and QoS parameters (CPUmemory signal strength throughput etc) and provides aninterface to collect user experience feedback A processingentity receives the (device and network) monitored data andanalyzes the incoming data The objective of this testbedinfrastructure is to study the effects of different networkparameters in the QoE in order to define new estimationmodels for QoE
In [16] the authors evaluated routing protocols BATMANand OLSR to support VoIP and video traffic from a QoSand QoE perspective The evaluation took place by runningexperiments in two different testbeds First experimentswere run in the Omnet++ simulator using the InetManetframework Second the same network topology and networkscenarios were deployed in the Emulab test bench a real(emulated) testbed and the same experiments were carriedout Finally the results of both testbeds (simulated andreal-emulated) were statistically compared in order to findinconsistencies The experiments in the simulated and emu-lated environments showed that BATMAN achieves betterthan OLSR and determined the relation between differentprotocol parameters and their performanceThese results canbe applied to implement network nodes that control in-stackprotocol parameters as a function of the observed traffic
In [17] a testbed to automatically extract a QoE modelof encrypted video streaming services was presented Thetestbed includes a software agent to be installed in theuser device which is able to reproduce the user interactionand collect the end-user application-level measurements thenetwork emulator NetEm which changes the link conditionsemulating the radio or core network and a Probe softwarewhich processes all the traffic at different levels computesthe TCPIP metrics and compares the end-user and networklevel measurements This testbed has been used to automat-ically construct the model (and validate the model) of thevideo performance of encrypted YouTube traffic over a Wi-Fi connection
More recently in [18] Solera et al presented a testbedfor evaluating video streaming services in LTE networks Inparticular the QoE of 3D video streaming services over LTEwas evaluated The testbed consists of a streaming serverthe NetEm network emulator and a streaming client Oneof the main contributions of the work is the extension ofNetEm to better model the characteristics of the packet delayin bursty services such as video streaming Previously torunning the experiments in the emulation-based testbedthe authors carried out a simulation campaign with anLTE simulator to obtain the configuration parameters ofNetEm for four different network scenarios These scenarioscombine different positions of the user in the cell anddifferent network loads From the review of these works itbecomes clear that the setup of a simulation or emulationframework for wireless or mobile environments requires inmany cases a deep understanding of the network scenariosTRIANGLE aims to reduce this effort by providing a set ofpreconfigured real network scenarios and the computationof the MOS in order to allow both researchers and appdevelopers to focus on the evaluation of new apps servicesand devices
3 TRIANGLE
The testbed the test methodology and the set of test caseshave been developed within the European funded TRIAN-GLE project Figure 1 shows the main functional blocks thatmake up the TRIANGLE testbed architecture
To facilitate the use of the TRIANGLE testbed fordifferent objectives (testing benchmarking and certifying)to remotely access the testbed and to gather and presentresults a web portal which offers an intuitive interfacehas been implemented It provides access to the testbedhiding unnecessary complexity to App developers Foradvanced users interested in deeper access to configurationparameters of the testbed elements or the test cases thetestbed offers a direct access to the Keysight TAP (TestingAutomation Platform) which is a programmable sequencerof actions with plugins that expose the configuration andcontrol of the instruments and tools integrated into thetestbed
In addition to the testbed itself TRIANGLE has devel-oped a test methodology and has implemented a set oftest cases which are made available through the portal Toachieve full test case automation all the testbed componentsare under the control of the testbed management frame-work which coordinates their configuration and executionprocesses the measurements made in each test case andcomputes QoE scores for the application tested
In addition as part of the testbed management frame-work each testbed component is controlled through a TAPdriver which serves as bridge between the TAP engine andthe actual component interface The configuration of thedifferent elements of the testbed is determined by the testcase to run within the set of test cases provided as partof TRIANGLE or the customized test cases built by usersThe testbed translates the test cases specific configurationssettings and actions into TAP commands that take care ofcommanding each testbed component
TRIANGLE test cases specify the measurements thatshould be collected to compute the KPI (Key PerformanceIndicators) of the feature under test Some measurementsare obtained directly from measurement instruments butothers require specific probes (either software or hardware)to help extract the specific measurements Software probesrunning on the same device (UE LTE User Equipment) thatthe application under test include DEKRA Agents and theTestelDroid [19] tool from UMA TRIANGLE also providesan instrumentation library so that app developers can delivermeasurement outputs which cannot otherwise be extractedand must be provided by the application itself Hardwareprobes include a power analyzer connected to the UE tomeasure power consumption and the radio access emulatorthat among others provides internal logs about the protocolexchange and radio interface low layers metrics
The radio access (LTE RAN) emulator plays a key role inthe TRIANGLE testbed The testbed RAN is provided by anoff-the-shelf E7515A UXM Wireless Test Set from Keysightan emulator that provides state-of-the-art test features Mostimportant the UXM also provides radio channel emulationfor the downlink radio channel
4 Wireless Communications and Mobile Computing
DBs
Interface and visualization (Portal)
Testbed management
Measurements and datacollections
UE
RAN
Transport
AppEPC Local application
serversApp App
TAP
Orcomposutor
Orchestration
Compositor ExecutorDBs
ETLframework
ETL modules
DEKRADriver
WebDriverDriver
AndroidTap Driver
EPCDriver
AppInstrumentat
ion TAPDriver
iOS TAPDriver
hellipTAP Driver
Figure 1 TRIANGLE testbed architecture
In order to provide an end-to-end system the testbedintegrates a commercial EPC (LTE Evolved Packet Core)from Polaris Networks which includes the main elements ofa standard 3GPP compliant LTE core network that is MME(Mobility Management Entity) SGW (Serving Gateway)PGW (Packet Gateway) HSS (Home Subscriber Server) andPCRF (Policy and Charging Rules Function) In additionthis EPC includes the EPDG (Evolved Packet Data Gateway)and ANDSF (Access Network Discovery and Session Func-tion) components for dual connectivity scenarios The RANemulator is connected to the EPC through the standard S1interface The testbed also offers the possibility of integratingartificial impairments in the interfaces between the corenetwork and the application servers
The Quamotion WebDriver another TRIANGLE ele-ment is able to automate user actions on both iOS andAndroid applications whether they are native hybrid offully web-based This tool is also used to prerecord the apprsquosuser flows which are needed to automate the otherwisemanual user actions in the test cases This completes the fullautomation operation
Finally the testbed also incorporates commercial mobiledevices (UEs) The devices are physically connected to thetestbed In order to preserve the radio conditions configuredat the radio access emulator the RAN emulator is cable con-ducted to the mobile device antenna connector To accuratelymeasure the power consumption theN6705B power analyzerdirectly powers the device Other measurement instrumentsmay be added in the future
4 TRIANGLE Approach
TheTRIANGLE testbed is an end-to-end framework devotedto testing and benchmarking mobile applications servicesand devices The idea behind the testing approach adoptedin the TRIANGLE testbed is to generalize QoE computationand provide a programmatic way of computing it Withthis approach the TRIANGLE testbed can accommodate thecomputation of the QoE for any application
The basic concept in TRIANGLErsquos approach to QoEevaluation is that the quality perceived by the user dependson many aspects (herein called domains) and that thisperception depends on its targeted use case For examplebattery life is critical for patient monitoring applications butless important in live streaming ones
To define the different 5G uses cases TRIANGLE basedits work in the Next Generation Mobile Network (NGMN)Alliance foundational White Paper which specifies theexpected services and network performance in future 5Gnetworks [20] More precisely the TRIANGLE project hasadopted a modular approach subdividing the so-calledldquoNGMN Use-Casesrdquo into blocks The name Use Case waskept in the TRIANGLE approach for describing the appli-cation service or vertical using the network services Thediversification of services expected in 5G requires a concretecategorization to have a sharp picture of what the user will beexpected to interact with This is essential for understandingwhich aspect of the QoE evaluation needs to be addressedThe final use cases categorization was defined in [11] andencompasses both the services normally accessible via mobile
Wireless Communications and Mobile Computing 5
Synthetic MOS
Test case
Scenario 1
Iteration 1
Meas 1
Meas
Meas P
Iteration
Meas 1
Meas
Meas P
Iteration N
Meas 1
Meas
Meas P
Scenario K
Iteration 1
Meas 1
Meas
Meas P
Iteration
Meas 1
Meas
Meas P
Iteration N
Meas 1
Meas
Meas P
KPI 1 Synthetic MOS 1
KPI Synthetic MOS
KPI R Synthetic MOS R
Aggregation Synthetic MOS Scenario 1
KPI 1 Synthetic MOS 1
KPI Synthetic MOS
KPI R Synthetic MOS R
Aggregation Synthetic MOS Scenario K
Figure 2 The process to obtain the synthetic-MOS score in a TRIANGLE test case
Table 1 Uses cases defined in the TRIANGLE project
Identifier Use CaseVR Virtual RealityGA GamingAR Augmented RealityCS Content Distribution Streaming ServicesLS Live Streaming ServicesSN Social NetworkingHS High Speed InternetPM Patient MonitoringES Emergency ServicesSM Smart MeteringSG Smart GridsCV Connected Vehicles
phones (UEs) and the ones that can be integrated in forexample gaming consoles advanced VR gear car units orIoT systems
The TRIANGLE domains group different aspects thatcan affect the final QoE perceived by the users The cur-rent testbed implementation supports three of the severaldomains that have been identified Apps User Experience(AUE) Apps Energy consumption (AEC) and ApplicationsDevice Resources Usage (RES)
Table 1 provides the use cases and Table 2 lists thedomains initially considered in TRIANGLE
To produce data to evaluate the QoE a series of testcases have been designed developed and implemented tobe run on the TRIANGLE testbed Obviously not all testcases are applicable to all applications under test becausenot all applications need or are designed to support all thefunctionalities that can be tested in the testbed In orderto automatically determine the test cases that are applicableto an application under test a questionnaire (identified as
features questionnaire in the portal) equivalent to the classi-cal conformance testing ICS (Implementation ConformanceStatement) has been developed and is accessible through theportal After filling the questionnaire the applicable test planthat is the test campaign with the list of applicable test casesis automatically generated
The sequence of user actions (type swipe tap etc) a userneeds to perform in the terminal (UE) to complete a task (egplay a video) is called the ldquoapp user flowrdquo In order to be ableto automatically run a test case the actual application userflow with the user actions a user would need to perform onthe phone to complete certain tasks defined in the test casealso has to be provided
Each test case univocally defines the conditions of execu-tion the sequence of actions the user would perform (ie theapp user flow) the sequence of actions that the elements ofthe testbed must perform the traffic injected the collectionof measurements to take and so forth In order to obtainstatistical significance each test case includes a numberof executions (iterations) under certain network conditions(herein called scenarios) Out of the various measurementsmade in the different iterations under any specific networkconditions (scenario) a number of KPIs (Key PerformanceIndicators) are computed The KPIs are normalized into astandard 1-to-5 scale as typically used in MOS scores andreferred to as synthetic-MOS a terminology that has beenadopted from previous works [7 21] The synthetic-MOSvalues are aggregated across network scenarios to produce anumber of intermediate synthetic-MOS scores which finallyare aggregated to obtain a synthetic-MOS score in each testcase (see Figure 2)
The process to obtain the final TRIANGLE mark issequential First for each domain a weighted average ofthe synthetic-MOS scores obtained in each test case inthe domain is calculated Next a weighted average of thesynthetic-MOS values in all the domains of a use case iscalculated to provide a single synthetic-MOS value per use
6 Wireless Communications and Mobile Computing
Table 2 TRIANGLE domains
Category Identifier Domain
Applications
AUE Apps User experienceAEC Apps Energy consumptionRES Device Resources UsageREL ReliabilityNWR Network Resources
Devices
Mobile Devices
DEC Energy ConsumptionDDP Data PerformanceDRF Radio PerformanceDRA User experience with reference apps
IoT DevicesIDR ReliabilityIDP Data PerformanceIEC Energy consumption
Synthetic MOS Domain A Use Case X
Synthetic MOS Domain B Use Case X
Synthetic MOS Use Case X
Synthetic MOS Use Case Y
TRIANGLE MARK
App
Synthetic MOS Test CaseDomain AUse Case X01
Synthetic MOS Test CaseDomain AUse Case X02
Synthetic MOS Test CaseDomain BUse Case X01
Synthetic MOS Test CaseDomain BUse Case X02
Synthetic MOS Test CaseDomain AUse Case Y01 Synthetic MOS Domain A Use Case Y
Synthetic MOS Test CaseDomain BUse Case Y01 Synthetic MOS Domain B Use Case Y
Figure 3 The process to obtain the TRIANGLE mark
case An application will usually be developed for one specificuse case as those defined in Table 1 but may be designed formore than one use case In the latter case a further weightedaverage is made with the synthetic-MOS scores obtained ineach use case supported by the application These sequentialsteps produce a single TRIANGLE mark an overall qualityscore as shown in Figure 3
This approach provides a common framework for testingapplications for benchmarking applications or even forcertifying disparate applications The overall process for anapp that implements features of different use cases is depictedin Figure 3
5 Details of the TRIANGLE QoE Computation
For each use case identified (see Table 1) and domain (seeTable 2) a number of test cases have been developed withinthe TRIANGLE project Each test case intends to test an
individual feature aspect or behaviour of the applicationunder test as shown in Figure 4
Each test case defines a number of measurements andbecause the results of the measurements depend on manyfactors they are not in general deterministic and thuseach test case has been designed not to perform just onesingle measurement but to run a number of iterations (N)of the same measurement Out of those measurements KPIsare computed For example if the time to load the firstmedia frame is the measurement taken in one specific testcase the average user waiting time KPI can be calculated bycomputing the mean of the values across all iterations Ingeneral different use case-domain pairs have a different set ofKPIsThe reader is encouraged to read [11] for further detailsabout the terminology used in TRIANGLE
Recommendation P10G100 Amendment 1 Definition ofQuality of Experience [2] notes that the overall acceptabilitymay be influenced by user expectations and context Forthe definition of the context technical specifications ITU-T
Wireless Communications and Mobile Computing 7
Feature Non-interactiveplayback
bullTime to load firstmedia frame
bullPlayback Cut-off
bullVideo resolution
bullContent Stall
i) Uses cases
vi) Synthetic MOS iv) Context
Networkscenarios
App User Flow Measurements KPIs MOS
ii) Domains
Apps Energy Consumption
Apps Device Resources
Apps User Experience
Mea
sure
men
ts
v) Test case execution
iii)Test caseAUECS01
Figure 4 QoE computation steps
G1030 ldquoEstimating end-to-end performance in IP networksfor data applicationsrdquo [9] and ITU-T G1031 ldquoQoE factors inweb-browsingrdquo [10] have been considered in TRIANGLEIn particular ITU-T G1031 [10] identifies the following con-text influence factors location (cafeteria office and home)interactivity (high-level interactivity versus low-level inter-activity) task type (business entertainment etc) and taskurgency (urgent versus casual) Userrsquos influence factors arehowever outside of the scope of the ITU recommendation
In the TRIANGLE project the context information hasbeen captured in the networks scenarios defined (Urban -Internet Cafe Off Peak Suburban - Shopping Mall BusyHours Urban ndash Pedestrian Urban ndash Office High speed trainndash Relay etc) and in the test cases specified in [11]
The test cases specify the conditions of the test butalso a sequence of actions that have to be executed by theapplication (app user flows) to test its features For examplethe test case that tests the ldquoPlay and Pauserdquo functionalitydefines the app user flow shown in Figure 5
The transformation of KPIs into QoE scores is the mostchallenging step in the TRIANGLE framework The execu-tion of the test cases will generate a significant amount of rawmeasurements about several aspects of the system SpecificKPIs can then be extracted through statistical analysis meandeviation cumulative distribution function (CDF) or ratio
TheKPIs will be individually interpolated in order to pro-vide a common homogeneous comparison and aggregationspace The interpolation is based on the application of twofunctions named Type I and Type II By using the proposedtwo types of interpolations the vast majority of KPIs can betranslated into normalized MOS-type of metric (synthetic-MOS) easy to be averaged in order to provide a simpleunified evaluation
Type I This function performs a linear interpolation on theoriginal data The variables 119898119894119899
119870119875119868and119898119886119909
119870119875119868are the worst
and best known values of a KPI from a reference case The
Perform login step (ifrequired) and wait for
10 seconds
Start playing avideo of 5
minutes during10 seconds
Pause thereproduction
Resume thereproduction after
2 minutes anduntil the end of
the video
Figure 5 App user flow used in the ldquoAUECS02 Play and Pauserdquotest case
function maps a value v of a KPI to vrsquo (synthetic-MOS) inthe range [1-to-5] by computing the following formula
V1015840 =V minus 119898119894119899
119870119875119868
119898119886119909119870119875119868minus 119898119894119899
119870119875119868
(50 minus 10) + 10 (1)
This function transforms a KPI to a synthetic-MOS value byapplying a simple linear interpolation between the worst andbest expected values from a reference case If a future inputcase falls outside the data range of the KPI the new value will
8 Wireless Communications and Mobile Computing
Table 3 AUECS002 test case description
Identifier AUECS002 (App User ExperienceContent Streaming002)Title Play and pauseObjective Measure the ability of the AUT to pause and the resume a media fileApplicability (ICSG ProductType = Application) AND (ICSG UseCases includes CS) AND ICSA CSPauseInitial Conditions AUT in in [AUT STARTED] mode (Note Defined in D22 [11] Appendix 4)
Steps(1) The Test System commands the AUT to replay the Application User Flow (Application User Flow that
presses first the Play button and later the Pause button)(2) The Test System measures whether pause operation was successful or not
Postamble (i) Execute the Postamble sequence (see section 26 in D22 [11] Appendix 4)
Measurements (Raw)
(i) Playback Cut-off Probability that successfully started stream reproduction is ended by a cause other thanthe intentional termination by the user
(ii) Pause Operation Whether pause operation is successful or not(iii) Time to load first media frame (s) after resuming The time elapsed since the user clicks resume button
until the media reproduction starts(Note For Exoplayer the RESUME button is the PLAY button)
be set to the extreme value minKPI (if it is worse) or maxKPI(if it is better)
Type II This function performs a logarithmic interpolationand is inspired on the opinion model recommended by theITU-T in [9] for a simple web search taskThis function mapsa value v of a KPI to vrsquo (synthetic-MOS) in the range [1-to-5]by computing the following formula
V1015840 =50 minus 10
ln ((119886 lowast 119908119900119903119904119905119870119875119868+ 119887) 119908119900119903119904119905
119870119875119868)
∙ (ln (V) minus ln (119886 lowast 119908119900119903119904119905119870119875119868+ 119886)) + 5
(2)
The default values of 119886 and 119887 correspond to the simple websearch task case (119886 = 0003 and 119887 = 012) [9 22] and theworst value has been extracted from the ITU-T G1030 Ifduring experimentation a future input case falls outside thedata range of the KPI the parameters 119886 and 119887will be updatedaccordingly Likewise if through subjective experimentationother values are considered better adjustments for specificservices the function can be easily updated
Once all KPIs are translated into synthetic-MOS valuesthey can be averaged with suitable weights In the averagingprocess the first step is to average over the network scenariosconsidered relevant for the use case as shown in Figure 2This provides the synthetic-MOS output value for the testcase If there is more than one test case per domain which isgenerally the case a weighted average is calculated in order toprovide one synthetic-MOS value per domain as depicted inFigure 3The final step is to average the synthetic-MOS scoresover all use cases supported by the application (see Figure 3)This provides the final score that is the TRIANGLE mark
6 A Practical Case Exoplayer under Test
For better understanding the complete process of obtainingtheTRIANGLEmark for a specific application the Exoplayer
Table4Measurement points associatedwith test caseAUECS002
Measurements Measurement points
Time to load first media frame Media File Playback - StartMedia File Playback - First Picture
Playback cut-off Media File Playback - StartMedia File Playback - End
Pause Media File Playback - Pause
is described in this section This application only has one usecase content distribution streaming services (CS)
Exoplayer is an application levelmedia player forAndroidpromoted by Google It provides an alternative to AndroidrsquosMediaPlayer API for playing audio and video both locally andover the Internet Exoplayer supports features not currentlysupported by Androidrsquos MediaPlayer API including DASHand SmoothStreaming adaptive playbacks
The TRIANGLE project has concentrated in testing justtwo of the Exoplayer features ldquoNoninteractive Playbackrdquoand ldquoPlay and Pauserdquo These features result in 6 test casesapplicable out of the test cases defined in TRIANGLETheseare test cases AUECS001 and AUECS002 in the App UserExperience domain test casesAECCS001 andAECCS002in the App Energy Consumption domain and test casesRESCS001 and RESCS002 in the Device Resources Usagedomain
The AUECS002 ldquoPlay and Pauserdquo test case descriptionbelonging to the AUE domain is shown in Table 3 The testcase description specifies the test conditions the generic appuser flow and the rawmeasurements which shall be collectedduring the execution of the test
The TRIANGLE project also offers a library that includesthe measurement points that should be inserted in thesource code of the app for enabling the collection of themeasurements specified Table 4 shows the measurementpoints required to compute the measurements specified intest case AUECS002
Wireless Communications and Mobile Computing 9
Table 5 Reference values for interpolation
Feature Domain KPI Synthetic MOS Calculation KPI min KPI maxNon-Interactive Playback AEC Average power consumption Type I 10 W 08 WNon-Interactive Playback AUE Time to load first media frame Type II KPI worst=20 msNon-Interactive Playback AUE Playback cut-off ratio Type I 50 0Non-Interactive Playback AUE Video resolution Type I 240p 720pNon-Interactive Playback RES Average CPU usage Type I 100 16Non-Interactive Playback RES Average memory usage Type I 100 40Play and Pause AEC Average power consumption Type I 10 W 08 WPlay and Pause AUE Pause operation success rate Type I 50 100Play and Pause RES Average CPU usage Type I 100 16Play and Pause RES Average memory usage Type I 100 40
The time to load first media picture measurement isobtained subtracting the timestamp of the measurementpoint ldquoMedia File Playback ndash Startrdquo from the measurementpoint ldquoMedia File Playback ndash First Picturerdquo
As specified in [11] all scenarios defined are applicableto the content streaming use case Therefore test cases inthe three domains currently supported by the testbed areexecuted in all the scenarios
Once the test campaign has finished the raw measure-ment results are processed to obtain the KPIs associated witheach test case average current consumption average time toload first media frame average CPU usage and so forth Theprocesses applied are detailed in Table 5 Based on previousexperiments performed by the authors the behaviour of thetime to load the first media frame KPI resembles the webresponse time KPI (ie the amount of time the user hasto wait for the service) and thus as recommended in theopinionmodel forweb search introduced in [9] a logarithmicinterpolation (type II) has been used for this metric
The results of the initial process that is the KPIs compu-tation are translated into synthetics-MOS values To computethese values reference benchmarking values for each of theKPIs need to be used according to the normalization andinterpolation process described in Section 5 Table 5 showswhat has been currently used by TRIANGLE for the AppUser Experience domain which is also used by NGMN asreference in their precommercial Trials document [23]
For example for the ldquotime to load first media framerdquo KPIshown in Table 5 the type of aggregation applied is averagingand the interpolation formula used is Type II
To achieve stable results each test case is executed 10times (10 iterations) in each network scenario The synthetic-MOS value in each domain is calculated by averaging themeasured synthetic-MOS values in the domain For examplesynthetic-MOS value is the RES domain obtained by aver-aging the synthetic-MOS value of ldquoaverage CPU usagerdquo andldquoaverage memory usagerdquo from the two test cases
Although Exoplayer supports several video streamingprotocols in this work only DASH [24] (Dynamic AdaptiveStreaming over HTTP) has been tested DASH clients shouldseamlessly adapt to changing network conditions by makingdecisions on which video segment to download (videosare encoded at multiple bitrates) The Exoplayerrsquos default
000
000
0
001
728
0
003
456
0
005
184
0
010
912
0
012
640
0
014
368
0
020
096
0
Timestamp
Video Resolution
0
200
400
600
800
1000
1200
Hor
izon
tal R
esol
utio
n
Figure 6 Video Resolution evolution in the Driving Urban Normalscenario
adaptation algorithm is basically throughput-based and someparameters control how often and when switching can occur
During the testing the testbed was configured with thedifferent network scenarios defined in [11] In these scenariosthe network configuration changes dynamically following arandom pattern resulting in different maximum throughputrates The expected behaviour of the application under testis that the video streaming client adapts to the availablethroughput by decreasing or increasing the resolution of thereceived video Figure 6 depicts how the client effectivelyadapts to the channel conditions
However the objective of the testing carried out in theTRIANGE testbed is not just to verify that the video stream-ing client actually adapts to the available maximum through-put but also to check whether this adaptation improves theusersrsquo experience quality
Table 6 shows a summary of the synthetic-MOS valuesobtained per scenario in one test case of each domain Thescores obtained in the RES andAECdomains are always highIn the AUE domain the synthetic MOS associated with theVideo Resolution shows low scores in some of the scenariosbecause the resolution decreases reasonable good scores inthe time to load first media and high scores in the time toplayback cut-off ratio Overall it can be concluded that the
10 Wireless Communications and Mobile Computing
Table 6 Synthetic MOS values per test case and scenario for the feature ldquoNoninteractive Playbackrdquo
AUE domain AEC domain RES domain
Test Case AUECS001 Test CaseAECCS001 Test Case RESCS001
ScenarioTime to loadfirst mediaframe
PlaybackCut-off ratio
VideoResolution
mode
AveragePower
Consumption
Average CPUUsage
AverageRAM Usage
HighSpeed DirectPassenger 21 31 23 47 43 42
Suburban Festival 38 47 31 48 43 41Suburban shopping mallbusy hours 37 37 13 48 44 41
Suburban shopping malloff-peak 36 31 23 48 43 41
Suburban stadium 38 29 21 47 44 41Urban Driving Normal 26 39 28 47 44 4Urban Driving TrafficJam 34 37 16 48 44 4
Urban Internet Cafe BusyHours 38 37 19 48 44 4
Urban Internet Cafe OffPeak 38 31 23 48 43 4
Urban Office 38 47 33 48 45 43Urban Pedestrian 39 26 2 47 44 4
35 36 23 47 44 41
DASH implementation of the video streaming client undertest is able to adapt to the changing conditions of the networkmaintaining an acceptable rate of video cut-off rebufferingtimes and resources usage
The final score in each domain is obtained by averagingthe synthetic-MOS values from all the tested network scenar-ios Figure 7 shows the spider diagram for the three domainstested In the User Experience domain the score obtained islower than the other domains due to the low synthetic-MOSvalues obtained for the video resolution
The final synthetic MOS for the use case Content Dis-tribution Streaming is obtained as a weighted average of thethree domains representing the overall QoE as perceived bythe userThefinal score for the Exoplayer version 1516 and thefeatures tested (Noninteractive Playback and Play and Pause)is 42 which means that the low score obtained in the videoresolution is compensated with the high scores in other KPIs
If an application under test has more than one use casethe next steps in the TRIANGLE mark project approachwould be the aggregation per use case and the aggregationover all use cases The final score the TRIANGLE mark is anestimation of the overall QoE as perceived by the user
In the current TRIANGLE implementation the weightsin all aggregations are the same Further research is neededto appropriately define the weights of each domain and eachuse case in the overall score of the applications
7 Conclusions
The main contribution of the TRIANGLE project is theprovision of a framework that generalizes QoE computation
and enables the execution of extensive and repeatable testcampaigns to obtainmeaningfulQoE scoresTheTRIANGLEproject has also defined amethodology which is based on thetransformation and aggregation of KPIs its transformationinto synthetic-MOS values and its aggregation over thedifferent domains and use cases
The TRIANGLE approach is a methodology flexibleenough to generalize the computation of QoE for any applica-tionservice Themethodology has been validated testing theDASH implementation in the Exoplayer App To confirm thesuitability of theweights used in the averaging process and theinterpolation parameters as well as to verify the correlationof the obtained MOS with that scored by users the authorshave started experiments with real users and initial results areencouraging
The process described produces a final TRIANGLEmarka single quality score which could eventually be used to cer-tify applications after achieving a consensus on the differentvalues of the process (weights limits etc) to use
Data Availability
Themethodology and results used to support the findings ofthis study are included within the article
Conflicts of Interest
The authors declare that they have no conflicts of inter-est
Wireless Communications and Mobile Computing 11
DER
AUE AEC
Device Resource Usage
User Experience Energy Consumption
Figure 7 Exoplayer synthetic-MOS values per domain
Acknowledgments
The TRIANGLE project is funded by the European UnionrsquosHorizon 2020 Research and Innovation Programme (GrantAgreement no 688712)
References
[1] ETSI ldquoHuman factors quality of experience (QoE) require-ments for real-time communication servicesrdquo Tech Rep 102643 2010
[2] ITU-T ldquoP10G100 (2006) amendment 1 (0107) new appendixI - definition of quality of experience (QoE)rdquo 2007
[3] F Kozamernik V Steinmann P Sunna and E WyckensldquoSAMVIQ - A new EBUmethodology for video quality evalua-tions in multimediardquo SMPTE Motion Imaging Journal vol 114no 4 pp 152ndash160 2005
[4] ITU-T ldquoG107 the E-model a computational model for use intransmission planningrdquo 2015
[5] J De Vriendt D De Vleeschauwer and D C Robinson ldquoQoEmodel for video delivered over an LTE network using HTTPadaptive streamingrdquo Bell Labs Technical Journal vol 18 no 4pp 45ndash62 2014
[6] S Jelassi G Rubino H Melvin H Youssef and G PujolleldquoQuality of Experience of VoIP Service A Survey of AssessmentApproaches andOpen Issuesrdquo IEEECommunications Surveys ampTutorials vol 14 no 2 pp 491ndash513 2012
[7] M Li C-L Yeh and S-Y Lu ldquoReal-Time QoE MonitoringSystem forVideo Streaming ServiceswithAdaptiveMedia Play-outrdquo International Journal of Digital Multimedia Broadcastingvol 2018 Article ID 2619438 11 pages 2018
[8] S Barakovic and L Skorin-Kapov ldquoSurvey and Challengesof QoE Management Issues in Wireless Networksrdquo Journal ofComputer Networks and Communications vol 2013 Article ID165146 28 pages 2013
[9] ITU-T ldquoG1030 estimating end-to-end performance in IPnetworks for data applicationsrdquo 2014
[10] ITU-T ldquoG1031 QoE factors in web-browsingrdquo 2014[11] EU H2020 TRIANGLE Project Deliverable D22 Final report
on the formalization of the certification process requirementsanduse cases 2017 httpswwwtriangle-projecteuproject-olddeliverables
[12] Q A Chen H Luo S Rosen et al ldquoQoE doctor diagnosingmobile app QoE with automated UI control and cross-layeranalysisrdquo in Proceedings of the Conference on Internet Mea-surement Conference (IMC rsquo14) pp 151ndash164 ACM VancouverCanada November 2014
[13] M A Mehmood A Wundsam S Uhlig D Levin N Sarrarand A Feldmann ldquoQoE-Lab Towards Evaluating Quality ofExperience for Future Internet Conditionsrdquo in Testbeds andResearch Infrastructure Korakis T Li H Tran-Gia P and HS Park Eds vol 90 of TridentCom 2011 Lnicst pp 286ndash301Springer Development of Networks and Communities BerlinGermany 2012
[14] D Levin A Wundsam A Mehmood and A FeldmannldquoBerlin The Berlin Experimental Router Laboratory for Inno-vative Networkingrdquo in TridentCom 2010 Lnicst T MagedanzA Gavras N H Thanh and J S Chase Eds vol 46 of LectureNotes of the Institute for Computer Sciences Social Informaticsand Telecommunications Engineering pp 602ndash604 SpringerHeidelberg Germany 2011
12 Wireless Communications and Mobile Computing
[15] K De Moor I Ketyko W Joseph et al ldquoProposed frameworkfor evaluating quality of experience in a mobile testbed-oriented living lab settingrdquo Mobile Networks and Applicationsvol 15 no 3 pp 378ndash391 2010
[16] R Sanchez-Iborra M-D Cano J J P C Rodrigues and JGarcia-Haro ldquoAnExperimental QoE Performance Study for theEfficient Transmission of High Demanding Traffic over an AdHoc Network Using BATMANrdquo Mobile Information Systemsvol 2015 Article ID 217106 14 pages 2015
[17] P Oliver-Balsalobre M Toril S Luna-Ramırez and R GarcıaGaraluz ldquoA system testbed for modeling encrypted video-streaming service performance indicators based on TCPIPmetricsrdquo EURASIP Journal on Wireless Communications andNetworking vol 2017 no 1 2017
[18] M Solera M Toril I Palomo G Gomez and J Poncela ldquoATestbed for Evaluating Video Streaming Services in LTErdquoWireless Personal Communications vol 98 no 3 pp 2753ndash27732018
[19] A Alvarez A Dıaz P Merino and F J Rivas ldquoField mea-surements of mobile services with Android smartphonesrdquoin Proceedings of the IEEE Consumer Communications andNetworking Conference (CCNC rsquo12) pp 105ndash109 Las Vegas NevUSA January 2012
[20] NGMN Alliance ldquoNGMN 5G white paperrdquo 2015 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2015NGMN 5G White Paper V1 0pdf
[21] ldquoInfrastructure and Design for Adaptivity and Flexibilityrdquo inMobile Information Systems Springer 2006
[22] J Nielsen ldquoResponse Times The Three Important Limitsrdquo inUsability Engineering 1993
[23] NGMN Alliance ldquoDefinition of the testing framework for theNGMN 5G pre-commercial networks trialsrdquo 2018 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2018180220 NGMN PreCommTrials Framework definition v1 0pdf
[24] 3GPP TS 26246 ldquoTransparent end-to-end Packet-switchedStreaming Services (PSS) Progressive Download and DynamicAdaptive Streaming over HTTP (3GP-DASH)rdquo 2018
International Journal of
AerospaceEngineeringHindawiwwwhindawicom Volume 2018
RoboticsJournal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Active and Passive Electronic Components
VLSI Design
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Shock and Vibration
Hindawiwwwhindawicom Volume 2018
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawiwwwhindawicom
Volume 2018
Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom
The Scientific World Journal
Volume 2018
Control Scienceand Engineering
Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom
Journal ofEngineeringVolume 2018
SensorsJournal of
Hindawiwwwhindawicom Volume 2018
International Journal of
RotatingMachinery
Hindawiwwwhindawicom Volume 2018
Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Navigation and Observation
International Journal of
Hindawi
wwwhindawicom Volume 2018
Advances in
Multimedia
Submit your manuscripts atwwwhindawicom
2 Wireless Communications and Mobile Computing
project follows also a parametric approach to compute theQoE
Conclusions in [5] point out that a large number ofparameters in the model could be cumbersome due tothe difficulty of obtaining the required measurements andbecause it would require significantly more data pointsand radio scenarios to tune the model The TRIANGLEapproach has overcome this limitation through the largevariety of measurements collected the variety of end-to-end network scenarios designed and mostly the degree ofautomation reached which enables the execution of intensivetest campaigns covering all scenarios
Although there are many proposals to calculate thequality of experience in general they are very much orientedto specific services for example voice [6] or video streaming[7 8] This paper introduces a methodology to compute theQoE of any application even if the application supportsmorethan one service
The QoE as perceived by the user depends on manyfactors the network conditions both at the core (CN) and atthe radio access (RAN) the terminal the service servers andhuman factors difficult to control Due to the complexity andthe time needed to run experiments or make measurementsmost of the studies limit the evaluation of theQoE to a limitedset of or even noncontrolled network conditions especiallythose that affect the radio interface (fading interference etc)TRIANGLE presents a methodology and a framework tocompute the QoE out of technical parameters weighting theimpact of the network conditions based on the actual usescases for the specific application As in ITU recommendationG1030 [9] and G1031 [10] the userrsquos influence factors areoutside of the scope of the methodology developed inTRIANGLE
TRIANGLE has developed an end-to-end cellular net-work testbed and a set of test cases to automatically testapplications under multiple changing network conditionsandor terminals and provide a single quality score Thescore is computed weighting the results obtained testingthe different uses cases applicable to the application forthe different aspects relevant to the user (the domains inTRIANGLE) and under the network scenarios relevant forthe application The framework allows specific QoS-to-QoEtranslations to be incorporated into the framework based onthe outcome of subjective experiments on new services
Note that although the TRIANGLE project also providesmeans to test devices and services only the process to testapplications is presented here
The rest of the paper is organized as follows Section 2provides an overview of related work Section 3 presents anoverview of the TRIANGLE testbed Section 4 introducesthe TRIANGLE approach Section 5 describes in detail howthe quality score is obtained in the TRIANGLE frameworkSection 6 provides an example and the outcome of thisapproach applied to the evaluation of a simple App theExoplayer Finally Section 7 summarizes the conclusions
2 State of the Art
Modelling and evaluatingQoE in current and next generationof mobile networks is an important and active research
area [8] Different types of testbeds can be found in the lit-erature ranging from simulated to emulated mobilewirelesstestbeds which are used to obtain subjective or objective QoEmetrics to extract a QoE model or to assess the correctnessof a previously generated QoE model Many of the testbedsreviewed have been developed for a specific research insteadof for a more general purpose such as the TRIANGLEtestbed which can serve a wide range of users (researchersapp developers service providers etc) In this section someQoE-related works that rely on testbeds are reviewed
The QoE Doctor tool [12] is closely related to the TRI-ANGLE testbed since its main purpose is the evaluation ofmobile apps QoE in an accurate systematic and repeatableway However QoE Doctor is just an Android tool that cantake measurements at different layers from the app userinterface (UI) to the network and quantify the factors thatimpact the app QoE It can be used to identify the causesof a degraded QoE but it is not able to control or monitorthe mobile network QoE Doctor uses an UI automationtool to reproduce user behaviour in the terminal (app userflows in TRIANGLE nomenclature) and tomeasure the user-perceived latency by detecting changes on the screen OtherQoE metrics computed by QoE Doctor are the mobile dataconsumption and the network energy consumption of the appby means of an offline analysis of the TCP flows The authorshave used QoE Doctor to evaluate the QoE of popular appssuch as YouTube Facebook or mobile web browsers One ofthe drawbacks of this approach is that most metrics are basedon detecting specific changes on the UI Thus the modulein charge of detecting UI changes has to be adapted for eachspecific app under test
QoE-Lab [13] is a multipurpose testbed that allows theevaluation of QoE in mobile networks One of its purposesis to evaluate the effect of new network scenarios on servicessuch as VoIP video streaming or web applications To thisend QoE-Lab extends BERLIN [14] testbed framework withsupport for next generation mobile networks and some newservices such as VoIP and video streaming The testbedallows the study of the effect of network handovers betweenwireless technologies dynamic migrations and virtualizedresources Similar to TRIANGLE the experiments are exe-cuted in a repeatable and controlled environment Howeverin the experiments presented in [13] the user equipmentwere laptops which usually have better performance andmore resources than smartphones (battery memory andCPU)The experiments also evaluated the impact of differentscenarios on the multimedia streaming services included inthe testbed The main limitations are that it is not possibleto evaluate different mobile apps running in different smart-phones or relate the QoE with the CPU battery usage and soforth
De Moor et al [15] proposed a user-centric methodologyfor the multidimensional evaluation of QoE in a mobile real-life environment The methodology relies on a distributedtestbed that monitors the network QoS and context infor-mation and integrates the subjective user experience basedon real-life settings The main component of the proposedarchitecture is theMobile Agent a component to be installedin the user device that monitors contextual data (location
Wireless Communications and Mobile Computing 3
velocity on-body sensors etc) and QoS parameters (CPUmemory signal strength throughput etc) and provides aninterface to collect user experience feedback A processingentity receives the (device and network) monitored data andanalyzes the incoming data The objective of this testbedinfrastructure is to study the effects of different networkparameters in the QoE in order to define new estimationmodels for QoE
In [16] the authors evaluated routing protocols BATMANand OLSR to support VoIP and video traffic from a QoSand QoE perspective The evaluation took place by runningexperiments in two different testbeds First experimentswere run in the Omnet++ simulator using the InetManetframework Second the same network topology and networkscenarios were deployed in the Emulab test bench a real(emulated) testbed and the same experiments were carriedout Finally the results of both testbeds (simulated andreal-emulated) were statistically compared in order to findinconsistencies The experiments in the simulated and emu-lated environments showed that BATMAN achieves betterthan OLSR and determined the relation between differentprotocol parameters and their performanceThese results canbe applied to implement network nodes that control in-stackprotocol parameters as a function of the observed traffic
In [17] a testbed to automatically extract a QoE modelof encrypted video streaming services was presented Thetestbed includes a software agent to be installed in theuser device which is able to reproduce the user interactionand collect the end-user application-level measurements thenetwork emulator NetEm which changes the link conditionsemulating the radio or core network and a Probe softwarewhich processes all the traffic at different levels computesthe TCPIP metrics and compares the end-user and networklevel measurements This testbed has been used to automat-ically construct the model (and validate the model) of thevideo performance of encrypted YouTube traffic over a Wi-Fi connection
More recently in [18] Solera et al presented a testbedfor evaluating video streaming services in LTE networks Inparticular the QoE of 3D video streaming services over LTEwas evaluated The testbed consists of a streaming serverthe NetEm network emulator and a streaming client Oneof the main contributions of the work is the extension ofNetEm to better model the characteristics of the packet delayin bursty services such as video streaming Previously torunning the experiments in the emulation-based testbedthe authors carried out a simulation campaign with anLTE simulator to obtain the configuration parameters ofNetEm for four different network scenarios These scenarioscombine different positions of the user in the cell anddifferent network loads From the review of these works itbecomes clear that the setup of a simulation or emulationframework for wireless or mobile environments requires inmany cases a deep understanding of the network scenariosTRIANGLE aims to reduce this effort by providing a set ofpreconfigured real network scenarios and the computationof the MOS in order to allow both researchers and appdevelopers to focus on the evaluation of new apps servicesand devices
3 TRIANGLE
The testbed the test methodology and the set of test caseshave been developed within the European funded TRIAN-GLE project Figure 1 shows the main functional blocks thatmake up the TRIANGLE testbed architecture
To facilitate the use of the TRIANGLE testbed fordifferent objectives (testing benchmarking and certifying)to remotely access the testbed and to gather and presentresults a web portal which offers an intuitive interfacehas been implemented It provides access to the testbedhiding unnecessary complexity to App developers Foradvanced users interested in deeper access to configurationparameters of the testbed elements or the test cases thetestbed offers a direct access to the Keysight TAP (TestingAutomation Platform) which is a programmable sequencerof actions with plugins that expose the configuration andcontrol of the instruments and tools integrated into thetestbed
In addition to the testbed itself TRIANGLE has devel-oped a test methodology and has implemented a set oftest cases which are made available through the portal Toachieve full test case automation all the testbed componentsare under the control of the testbed management frame-work which coordinates their configuration and executionprocesses the measurements made in each test case andcomputes QoE scores for the application tested
In addition as part of the testbed management frame-work each testbed component is controlled through a TAPdriver which serves as bridge between the TAP engine andthe actual component interface The configuration of thedifferent elements of the testbed is determined by the testcase to run within the set of test cases provided as partof TRIANGLE or the customized test cases built by usersThe testbed translates the test cases specific configurationssettings and actions into TAP commands that take care ofcommanding each testbed component
TRIANGLE test cases specify the measurements thatshould be collected to compute the KPI (Key PerformanceIndicators) of the feature under test Some measurementsare obtained directly from measurement instruments butothers require specific probes (either software or hardware)to help extract the specific measurements Software probesrunning on the same device (UE LTE User Equipment) thatthe application under test include DEKRA Agents and theTestelDroid [19] tool from UMA TRIANGLE also providesan instrumentation library so that app developers can delivermeasurement outputs which cannot otherwise be extractedand must be provided by the application itself Hardwareprobes include a power analyzer connected to the UE tomeasure power consumption and the radio access emulatorthat among others provides internal logs about the protocolexchange and radio interface low layers metrics
The radio access (LTE RAN) emulator plays a key role inthe TRIANGLE testbed The testbed RAN is provided by anoff-the-shelf E7515A UXM Wireless Test Set from Keysightan emulator that provides state-of-the-art test features Mostimportant the UXM also provides radio channel emulationfor the downlink radio channel
4 Wireless Communications and Mobile Computing
DBs
Interface and visualization (Portal)
Testbed management
Measurements and datacollections
UE
RAN
Transport
AppEPC Local application
serversApp App
TAP
Orcomposutor
Orchestration
Compositor ExecutorDBs
ETLframework
ETL modules
DEKRADriver
WebDriverDriver
AndroidTap Driver
EPCDriver
AppInstrumentat
ion TAPDriver
iOS TAPDriver
hellipTAP Driver
Figure 1 TRIANGLE testbed architecture
In order to provide an end-to-end system the testbedintegrates a commercial EPC (LTE Evolved Packet Core)from Polaris Networks which includes the main elements ofa standard 3GPP compliant LTE core network that is MME(Mobility Management Entity) SGW (Serving Gateway)PGW (Packet Gateway) HSS (Home Subscriber Server) andPCRF (Policy and Charging Rules Function) In additionthis EPC includes the EPDG (Evolved Packet Data Gateway)and ANDSF (Access Network Discovery and Session Func-tion) components for dual connectivity scenarios The RANemulator is connected to the EPC through the standard S1interface The testbed also offers the possibility of integratingartificial impairments in the interfaces between the corenetwork and the application servers
The Quamotion WebDriver another TRIANGLE ele-ment is able to automate user actions on both iOS andAndroid applications whether they are native hybrid offully web-based This tool is also used to prerecord the apprsquosuser flows which are needed to automate the otherwisemanual user actions in the test cases This completes the fullautomation operation
Finally the testbed also incorporates commercial mobiledevices (UEs) The devices are physically connected to thetestbed In order to preserve the radio conditions configuredat the radio access emulator the RAN emulator is cable con-ducted to the mobile device antenna connector To accuratelymeasure the power consumption theN6705B power analyzerdirectly powers the device Other measurement instrumentsmay be added in the future
4 TRIANGLE Approach
TheTRIANGLE testbed is an end-to-end framework devotedto testing and benchmarking mobile applications servicesand devices The idea behind the testing approach adoptedin the TRIANGLE testbed is to generalize QoE computationand provide a programmatic way of computing it Withthis approach the TRIANGLE testbed can accommodate thecomputation of the QoE for any application
The basic concept in TRIANGLErsquos approach to QoEevaluation is that the quality perceived by the user dependson many aspects (herein called domains) and that thisperception depends on its targeted use case For examplebattery life is critical for patient monitoring applications butless important in live streaming ones
To define the different 5G uses cases TRIANGLE basedits work in the Next Generation Mobile Network (NGMN)Alliance foundational White Paper which specifies theexpected services and network performance in future 5Gnetworks [20] More precisely the TRIANGLE project hasadopted a modular approach subdividing the so-calledldquoNGMN Use-Casesrdquo into blocks The name Use Case waskept in the TRIANGLE approach for describing the appli-cation service or vertical using the network services Thediversification of services expected in 5G requires a concretecategorization to have a sharp picture of what the user will beexpected to interact with This is essential for understandingwhich aspect of the QoE evaluation needs to be addressedThe final use cases categorization was defined in [11] andencompasses both the services normally accessible via mobile
Wireless Communications and Mobile Computing 5
Synthetic MOS
Test case
Scenario 1
Iteration 1
Meas 1
Meas
Meas P
Iteration
Meas 1
Meas
Meas P
Iteration N
Meas 1
Meas
Meas P
Scenario K
Iteration 1
Meas 1
Meas
Meas P
Iteration
Meas 1
Meas
Meas P
Iteration N
Meas 1
Meas
Meas P
KPI 1 Synthetic MOS 1
KPI Synthetic MOS
KPI R Synthetic MOS R
Aggregation Synthetic MOS Scenario 1
KPI 1 Synthetic MOS 1
KPI Synthetic MOS
KPI R Synthetic MOS R
Aggregation Synthetic MOS Scenario K
Figure 2 The process to obtain the synthetic-MOS score in a TRIANGLE test case
Table 1 Uses cases defined in the TRIANGLE project
Identifier Use CaseVR Virtual RealityGA GamingAR Augmented RealityCS Content Distribution Streaming ServicesLS Live Streaming ServicesSN Social NetworkingHS High Speed InternetPM Patient MonitoringES Emergency ServicesSM Smart MeteringSG Smart GridsCV Connected Vehicles
phones (UEs) and the ones that can be integrated in forexample gaming consoles advanced VR gear car units orIoT systems
The TRIANGLE domains group different aspects thatcan affect the final QoE perceived by the users The cur-rent testbed implementation supports three of the severaldomains that have been identified Apps User Experience(AUE) Apps Energy consumption (AEC) and ApplicationsDevice Resources Usage (RES)
Table 1 provides the use cases and Table 2 lists thedomains initially considered in TRIANGLE
To produce data to evaluate the QoE a series of testcases have been designed developed and implemented tobe run on the TRIANGLE testbed Obviously not all testcases are applicable to all applications under test becausenot all applications need or are designed to support all thefunctionalities that can be tested in the testbed In orderto automatically determine the test cases that are applicableto an application under test a questionnaire (identified as
features questionnaire in the portal) equivalent to the classi-cal conformance testing ICS (Implementation ConformanceStatement) has been developed and is accessible through theportal After filling the questionnaire the applicable test planthat is the test campaign with the list of applicable test casesis automatically generated
The sequence of user actions (type swipe tap etc) a userneeds to perform in the terminal (UE) to complete a task (egplay a video) is called the ldquoapp user flowrdquo In order to be ableto automatically run a test case the actual application userflow with the user actions a user would need to perform onthe phone to complete certain tasks defined in the test casealso has to be provided
Each test case univocally defines the conditions of execu-tion the sequence of actions the user would perform (ie theapp user flow) the sequence of actions that the elements ofthe testbed must perform the traffic injected the collectionof measurements to take and so forth In order to obtainstatistical significance each test case includes a numberof executions (iterations) under certain network conditions(herein called scenarios) Out of the various measurementsmade in the different iterations under any specific networkconditions (scenario) a number of KPIs (Key PerformanceIndicators) are computed The KPIs are normalized into astandard 1-to-5 scale as typically used in MOS scores andreferred to as synthetic-MOS a terminology that has beenadopted from previous works [7 21] The synthetic-MOSvalues are aggregated across network scenarios to produce anumber of intermediate synthetic-MOS scores which finallyare aggregated to obtain a synthetic-MOS score in each testcase (see Figure 2)
The process to obtain the final TRIANGLE mark issequential First for each domain a weighted average ofthe synthetic-MOS scores obtained in each test case inthe domain is calculated Next a weighted average of thesynthetic-MOS values in all the domains of a use case iscalculated to provide a single synthetic-MOS value per use
6 Wireless Communications and Mobile Computing
Table 2 TRIANGLE domains
Category Identifier Domain
Applications
AUE Apps User experienceAEC Apps Energy consumptionRES Device Resources UsageREL ReliabilityNWR Network Resources
Devices
Mobile Devices
DEC Energy ConsumptionDDP Data PerformanceDRF Radio PerformanceDRA User experience with reference apps
IoT DevicesIDR ReliabilityIDP Data PerformanceIEC Energy consumption
Synthetic MOS Domain A Use Case X
Synthetic MOS Domain B Use Case X
Synthetic MOS Use Case X
Synthetic MOS Use Case Y
TRIANGLE MARK
App
Synthetic MOS Test CaseDomain AUse Case X01
Synthetic MOS Test CaseDomain AUse Case X02
Synthetic MOS Test CaseDomain BUse Case X01
Synthetic MOS Test CaseDomain BUse Case X02
Synthetic MOS Test CaseDomain AUse Case Y01 Synthetic MOS Domain A Use Case Y
Synthetic MOS Test CaseDomain BUse Case Y01 Synthetic MOS Domain B Use Case Y
Figure 3 The process to obtain the TRIANGLE mark
case An application will usually be developed for one specificuse case as those defined in Table 1 but may be designed formore than one use case In the latter case a further weightedaverage is made with the synthetic-MOS scores obtained ineach use case supported by the application These sequentialsteps produce a single TRIANGLE mark an overall qualityscore as shown in Figure 3
This approach provides a common framework for testingapplications for benchmarking applications or even forcertifying disparate applications The overall process for anapp that implements features of different use cases is depictedin Figure 3
5 Details of the TRIANGLE QoE Computation
For each use case identified (see Table 1) and domain (seeTable 2) a number of test cases have been developed withinthe TRIANGLE project Each test case intends to test an
individual feature aspect or behaviour of the applicationunder test as shown in Figure 4
Each test case defines a number of measurements andbecause the results of the measurements depend on manyfactors they are not in general deterministic and thuseach test case has been designed not to perform just onesingle measurement but to run a number of iterations (N)of the same measurement Out of those measurements KPIsare computed For example if the time to load the firstmedia frame is the measurement taken in one specific testcase the average user waiting time KPI can be calculated bycomputing the mean of the values across all iterations Ingeneral different use case-domain pairs have a different set ofKPIsThe reader is encouraged to read [11] for further detailsabout the terminology used in TRIANGLE
Recommendation P10G100 Amendment 1 Definition ofQuality of Experience [2] notes that the overall acceptabilitymay be influenced by user expectations and context Forthe definition of the context technical specifications ITU-T
Wireless Communications and Mobile Computing 7
Feature Non-interactiveplayback
bullTime to load firstmedia frame
bullPlayback Cut-off
bullVideo resolution
bullContent Stall
i) Uses cases
vi) Synthetic MOS iv) Context
Networkscenarios
App User Flow Measurements KPIs MOS
ii) Domains
Apps Energy Consumption
Apps Device Resources
Apps User Experience
Mea
sure
men
ts
v) Test case execution
iii)Test caseAUECS01
Figure 4 QoE computation steps
G1030 ldquoEstimating end-to-end performance in IP networksfor data applicationsrdquo [9] and ITU-T G1031 ldquoQoE factors inweb-browsingrdquo [10] have been considered in TRIANGLEIn particular ITU-T G1031 [10] identifies the following con-text influence factors location (cafeteria office and home)interactivity (high-level interactivity versus low-level inter-activity) task type (business entertainment etc) and taskurgency (urgent versus casual) Userrsquos influence factors arehowever outside of the scope of the ITU recommendation
In the TRIANGLE project the context information hasbeen captured in the networks scenarios defined (Urban -Internet Cafe Off Peak Suburban - Shopping Mall BusyHours Urban ndash Pedestrian Urban ndash Office High speed trainndash Relay etc) and in the test cases specified in [11]
The test cases specify the conditions of the test butalso a sequence of actions that have to be executed by theapplication (app user flows) to test its features For examplethe test case that tests the ldquoPlay and Pauserdquo functionalitydefines the app user flow shown in Figure 5
The transformation of KPIs into QoE scores is the mostchallenging step in the TRIANGLE framework The execu-tion of the test cases will generate a significant amount of rawmeasurements about several aspects of the system SpecificKPIs can then be extracted through statistical analysis meandeviation cumulative distribution function (CDF) or ratio
TheKPIs will be individually interpolated in order to pro-vide a common homogeneous comparison and aggregationspace The interpolation is based on the application of twofunctions named Type I and Type II By using the proposedtwo types of interpolations the vast majority of KPIs can betranslated into normalized MOS-type of metric (synthetic-MOS) easy to be averaged in order to provide a simpleunified evaluation
Type I This function performs a linear interpolation on theoriginal data The variables 119898119894119899
119870119875119868and119898119886119909
119870119875119868are the worst
and best known values of a KPI from a reference case The
Perform login step (ifrequired) and wait for
10 seconds
Start playing avideo of 5
minutes during10 seconds
Pause thereproduction
Resume thereproduction after
2 minutes anduntil the end of
the video
Figure 5 App user flow used in the ldquoAUECS02 Play and Pauserdquotest case
function maps a value v of a KPI to vrsquo (synthetic-MOS) inthe range [1-to-5] by computing the following formula
V1015840 =V minus 119898119894119899
119870119875119868
119898119886119909119870119875119868minus 119898119894119899
119870119875119868
(50 minus 10) + 10 (1)
This function transforms a KPI to a synthetic-MOS value byapplying a simple linear interpolation between the worst andbest expected values from a reference case If a future inputcase falls outside the data range of the KPI the new value will
8 Wireless Communications and Mobile Computing
Table 3 AUECS002 test case description
Identifier AUECS002 (App User ExperienceContent Streaming002)Title Play and pauseObjective Measure the ability of the AUT to pause and the resume a media fileApplicability (ICSG ProductType = Application) AND (ICSG UseCases includes CS) AND ICSA CSPauseInitial Conditions AUT in in [AUT STARTED] mode (Note Defined in D22 [11] Appendix 4)
Steps(1) The Test System commands the AUT to replay the Application User Flow (Application User Flow that
presses first the Play button and later the Pause button)(2) The Test System measures whether pause operation was successful or not
Postamble (i) Execute the Postamble sequence (see section 26 in D22 [11] Appendix 4)
Measurements (Raw)
(i) Playback Cut-off Probability that successfully started stream reproduction is ended by a cause other thanthe intentional termination by the user
(ii) Pause Operation Whether pause operation is successful or not(iii) Time to load first media frame (s) after resuming The time elapsed since the user clicks resume button
until the media reproduction starts(Note For Exoplayer the RESUME button is the PLAY button)
be set to the extreme value minKPI (if it is worse) or maxKPI(if it is better)
Type II This function performs a logarithmic interpolationand is inspired on the opinion model recommended by theITU-T in [9] for a simple web search taskThis function mapsa value v of a KPI to vrsquo (synthetic-MOS) in the range [1-to-5]by computing the following formula
V1015840 =50 minus 10
ln ((119886 lowast 119908119900119903119904119905119870119875119868+ 119887) 119908119900119903119904119905
119870119875119868)
∙ (ln (V) minus ln (119886 lowast 119908119900119903119904119905119870119875119868+ 119886)) + 5
(2)
The default values of 119886 and 119887 correspond to the simple websearch task case (119886 = 0003 and 119887 = 012) [9 22] and theworst value has been extracted from the ITU-T G1030 Ifduring experimentation a future input case falls outside thedata range of the KPI the parameters 119886 and 119887will be updatedaccordingly Likewise if through subjective experimentationother values are considered better adjustments for specificservices the function can be easily updated
Once all KPIs are translated into synthetic-MOS valuesthey can be averaged with suitable weights In the averagingprocess the first step is to average over the network scenariosconsidered relevant for the use case as shown in Figure 2This provides the synthetic-MOS output value for the testcase If there is more than one test case per domain which isgenerally the case a weighted average is calculated in order toprovide one synthetic-MOS value per domain as depicted inFigure 3The final step is to average the synthetic-MOS scoresover all use cases supported by the application (see Figure 3)This provides the final score that is the TRIANGLE mark
6 A Practical Case Exoplayer under Test
For better understanding the complete process of obtainingtheTRIANGLEmark for a specific application the Exoplayer
Table4Measurement points associatedwith test caseAUECS002
Measurements Measurement points
Time to load first media frame Media File Playback - StartMedia File Playback - First Picture
Playback cut-off Media File Playback - StartMedia File Playback - End
Pause Media File Playback - Pause
is described in this section This application only has one usecase content distribution streaming services (CS)
Exoplayer is an application levelmedia player forAndroidpromoted by Google It provides an alternative to AndroidrsquosMediaPlayer API for playing audio and video both locally andover the Internet Exoplayer supports features not currentlysupported by Androidrsquos MediaPlayer API including DASHand SmoothStreaming adaptive playbacks
The TRIANGLE project has concentrated in testing justtwo of the Exoplayer features ldquoNoninteractive Playbackrdquoand ldquoPlay and Pauserdquo These features result in 6 test casesapplicable out of the test cases defined in TRIANGLETheseare test cases AUECS001 and AUECS002 in the App UserExperience domain test casesAECCS001 andAECCS002in the App Energy Consumption domain and test casesRESCS001 and RESCS002 in the Device Resources Usagedomain
The AUECS002 ldquoPlay and Pauserdquo test case descriptionbelonging to the AUE domain is shown in Table 3 The testcase description specifies the test conditions the generic appuser flow and the rawmeasurements which shall be collectedduring the execution of the test
The TRIANGLE project also offers a library that includesthe measurement points that should be inserted in thesource code of the app for enabling the collection of themeasurements specified Table 4 shows the measurementpoints required to compute the measurements specified intest case AUECS002
Wireless Communications and Mobile Computing 9
Table 5 Reference values for interpolation
Feature Domain KPI Synthetic MOS Calculation KPI min KPI maxNon-Interactive Playback AEC Average power consumption Type I 10 W 08 WNon-Interactive Playback AUE Time to load first media frame Type II KPI worst=20 msNon-Interactive Playback AUE Playback cut-off ratio Type I 50 0Non-Interactive Playback AUE Video resolution Type I 240p 720pNon-Interactive Playback RES Average CPU usage Type I 100 16Non-Interactive Playback RES Average memory usage Type I 100 40Play and Pause AEC Average power consumption Type I 10 W 08 WPlay and Pause AUE Pause operation success rate Type I 50 100Play and Pause RES Average CPU usage Type I 100 16Play and Pause RES Average memory usage Type I 100 40
The time to load first media picture measurement isobtained subtracting the timestamp of the measurementpoint ldquoMedia File Playback ndash Startrdquo from the measurementpoint ldquoMedia File Playback ndash First Picturerdquo
As specified in [11] all scenarios defined are applicableto the content streaming use case Therefore test cases inthe three domains currently supported by the testbed areexecuted in all the scenarios
Once the test campaign has finished the raw measure-ment results are processed to obtain the KPIs associated witheach test case average current consumption average time toload first media frame average CPU usage and so forth Theprocesses applied are detailed in Table 5 Based on previousexperiments performed by the authors the behaviour of thetime to load the first media frame KPI resembles the webresponse time KPI (ie the amount of time the user hasto wait for the service) and thus as recommended in theopinionmodel forweb search introduced in [9] a logarithmicinterpolation (type II) has been used for this metric
The results of the initial process that is the KPIs compu-tation are translated into synthetics-MOS values To computethese values reference benchmarking values for each of theKPIs need to be used according to the normalization andinterpolation process described in Section 5 Table 5 showswhat has been currently used by TRIANGLE for the AppUser Experience domain which is also used by NGMN asreference in their precommercial Trials document [23]
For example for the ldquotime to load first media framerdquo KPIshown in Table 5 the type of aggregation applied is averagingand the interpolation formula used is Type II
To achieve stable results each test case is executed 10times (10 iterations) in each network scenario The synthetic-MOS value in each domain is calculated by averaging themeasured synthetic-MOS values in the domain For examplesynthetic-MOS value is the RES domain obtained by aver-aging the synthetic-MOS value of ldquoaverage CPU usagerdquo andldquoaverage memory usagerdquo from the two test cases
Although Exoplayer supports several video streamingprotocols in this work only DASH [24] (Dynamic AdaptiveStreaming over HTTP) has been tested DASH clients shouldseamlessly adapt to changing network conditions by makingdecisions on which video segment to download (videosare encoded at multiple bitrates) The Exoplayerrsquos default
000
000
0
001
728
0
003
456
0
005
184
0
010
912
0
012
640
0
014
368
0
020
096
0
Timestamp
Video Resolution
0
200
400
600
800
1000
1200
Hor
izon
tal R
esol
utio
n
Figure 6 Video Resolution evolution in the Driving Urban Normalscenario
adaptation algorithm is basically throughput-based and someparameters control how often and when switching can occur
During the testing the testbed was configured with thedifferent network scenarios defined in [11] In these scenariosthe network configuration changes dynamically following arandom pattern resulting in different maximum throughputrates The expected behaviour of the application under testis that the video streaming client adapts to the availablethroughput by decreasing or increasing the resolution of thereceived video Figure 6 depicts how the client effectivelyadapts to the channel conditions
However the objective of the testing carried out in theTRIANGE testbed is not just to verify that the video stream-ing client actually adapts to the available maximum through-put but also to check whether this adaptation improves theusersrsquo experience quality
Table 6 shows a summary of the synthetic-MOS valuesobtained per scenario in one test case of each domain Thescores obtained in the RES andAECdomains are always highIn the AUE domain the synthetic MOS associated with theVideo Resolution shows low scores in some of the scenariosbecause the resolution decreases reasonable good scores inthe time to load first media and high scores in the time toplayback cut-off ratio Overall it can be concluded that the
10 Wireless Communications and Mobile Computing
Table 6 Synthetic MOS values per test case and scenario for the feature ldquoNoninteractive Playbackrdquo
AUE domain AEC domain RES domain
Test Case AUECS001 Test CaseAECCS001 Test Case RESCS001
ScenarioTime to loadfirst mediaframe
PlaybackCut-off ratio
VideoResolution
mode
AveragePower
Consumption
Average CPUUsage
AverageRAM Usage
HighSpeed DirectPassenger 21 31 23 47 43 42
Suburban Festival 38 47 31 48 43 41Suburban shopping mallbusy hours 37 37 13 48 44 41
Suburban shopping malloff-peak 36 31 23 48 43 41
Suburban stadium 38 29 21 47 44 41Urban Driving Normal 26 39 28 47 44 4Urban Driving TrafficJam 34 37 16 48 44 4
Urban Internet Cafe BusyHours 38 37 19 48 44 4
Urban Internet Cafe OffPeak 38 31 23 48 43 4
Urban Office 38 47 33 48 45 43Urban Pedestrian 39 26 2 47 44 4
35 36 23 47 44 41
DASH implementation of the video streaming client undertest is able to adapt to the changing conditions of the networkmaintaining an acceptable rate of video cut-off rebufferingtimes and resources usage
The final score in each domain is obtained by averagingthe synthetic-MOS values from all the tested network scenar-ios Figure 7 shows the spider diagram for the three domainstested In the User Experience domain the score obtained islower than the other domains due to the low synthetic-MOSvalues obtained for the video resolution
The final synthetic MOS for the use case Content Dis-tribution Streaming is obtained as a weighted average of thethree domains representing the overall QoE as perceived bythe userThefinal score for the Exoplayer version 1516 and thefeatures tested (Noninteractive Playback and Play and Pause)is 42 which means that the low score obtained in the videoresolution is compensated with the high scores in other KPIs
If an application under test has more than one use casethe next steps in the TRIANGLE mark project approachwould be the aggregation per use case and the aggregationover all use cases The final score the TRIANGLE mark is anestimation of the overall QoE as perceived by the user
In the current TRIANGLE implementation the weightsin all aggregations are the same Further research is neededto appropriately define the weights of each domain and eachuse case in the overall score of the applications
7 Conclusions
The main contribution of the TRIANGLE project is theprovision of a framework that generalizes QoE computation
and enables the execution of extensive and repeatable testcampaigns to obtainmeaningfulQoE scoresTheTRIANGLEproject has also defined amethodology which is based on thetransformation and aggregation of KPIs its transformationinto synthetic-MOS values and its aggregation over thedifferent domains and use cases
The TRIANGLE approach is a methodology flexibleenough to generalize the computation of QoE for any applica-tionservice Themethodology has been validated testing theDASH implementation in the Exoplayer App To confirm thesuitability of theweights used in the averaging process and theinterpolation parameters as well as to verify the correlationof the obtained MOS with that scored by users the authorshave started experiments with real users and initial results areencouraging
The process described produces a final TRIANGLEmarka single quality score which could eventually be used to cer-tify applications after achieving a consensus on the differentvalues of the process (weights limits etc) to use
Data Availability
Themethodology and results used to support the findings ofthis study are included within the article
Conflicts of Interest
The authors declare that they have no conflicts of inter-est
Wireless Communications and Mobile Computing 11
DER
AUE AEC
Device Resource Usage
User Experience Energy Consumption
Figure 7 Exoplayer synthetic-MOS values per domain
Acknowledgments
The TRIANGLE project is funded by the European UnionrsquosHorizon 2020 Research and Innovation Programme (GrantAgreement no 688712)
References
[1] ETSI ldquoHuman factors quality of experience (QoE) require-ments for real-time communication servicesrdquo Tech Rep 102643 2010
[2] ITU-T ldquoP10G100 (2006) amendment 1 (0107) new appendixI - definition of quality of experience (QoE)rdquo 2007
[3] F Kozamernik V Steinmann P Sunna and E WyckensldquoSAMVIQ - A new EBUmethodology for video quality evalua-tions in multimediardquo SMPTE Motion Imaging Journal vol 114no 4 pp 152ndash160 2005
[4] ITU-T ldquoG107 the E-model a computational model for use intransmission planningrdquo 2015
[5] J De Vriendt D De Vleeschauwer and D C Robinson ldquoQoEmodel for video delivered over an LTE network using HTTPadaptive streamingrdquo Bell Labs Technical Journal vol 18 no 4pp 45ndash62 2014
[6] S Jelassi G Rubino H Melvin H Youssef and G PujolleldquoQuality of Experience of VoIP Service A Survey of AssessmentApproaches andOpen Issuesrdquo IEEECommunications Surveys ampTutorials vol 14 no 2 pp 491ndash513 2012
[7] M Li C-L Yeh and S-Y Lu ldquoReal-Time QoE MonitoringSystem forVideo Streaming ServiceswithAdaptiveMedia Play-outrdquo International Journal of Digital Multimedia Broadcastingvol 2018 Article ID 2619438 11 pages 2018
[8] S Barakovic and L Skorin-Kapov ldquoSurvey and Challengesof QoE Management Issues in Wireless Networksrdquo Journal ofComputer Networks and Communications vol 2013 Article ID165146 28 pages 2013
[9] ITU-T ldquoG1030 estimating end-to-end performance in IPnetworks for data applicationsrdquo 2014
[10] ITU-T ldquoG1031 QoE factors in web-browsingrdquo 2014[11] EU H2020 TRIANGLE Project Deliverable D22 Final report
on the formalization of the certification process requirementsanduse cases 2017 httpswwwtriangle-projecteuproject-olddeliverables
[12] Q A Chen H Luo S Rosen et al ldquoQoE doctor diagnosingmobile app QoE with automated UI control and cross-layeranalysisrdquo in Proceedings of the Conference on Internet Mea-surement Conference (IMC rsquo14) pp 151ndash164 ACM VancouverCanada November 2014
[13] M A Mehmood A Wundsam S Uhlig D Levin N Sarrarand A Feldmann ldquoQoE-Lab Towards Evaluating Quality ofExperience for Future Internet Conditionsrdquo in Testbeds andResearch Infrastructure Korakis T Li H Tran-Gia P and HS Park Eds vol 90 of TridentCom 2011 Lnicst pp 286ndash301Springer Development of Networks and Communities BerlinGermany 2012
[14] D Levin A Wundsam A Mehmood and A FeldmannldquoBerlin The Berlin Experimental Router Laboratory for Inno-vative Networkingrdquo in TridentCom 2010 Lnicst T MagedanzA Gavras N H Thanh and J S Chase Eds vol 46 of LectureNotes of the Institute for Computer Sciences Social Informaticsand Telecommunications Engineering pp 602ndash604 SpringerHeidelberg Germany 2011
12 Wireless Communications and Mobile Computing
[15] K De Moor I Ketyko W Joseph et al ldquoProposed frameworkfor evaluating quality of experience in a mobile testbed-oriented living lab settingrdquo Mobile Networks and Applicationsvol 15 no 3 pp 378ndash391 2010
[16] R Sanchez-Iborra M-D Cano J J P C Rodrigues and JGarcia-Haro ldquoAnExperimental QoE Performance Study for theEfficient Transmission of High Demanding Traffic over an AdHoc Network Using BATMANrdquo Mobile Information Systemsvol 2015 Article ID 217106 14 pages 2015
[17] P Oliver-Balsalobre M Toril S Luna-Ramırez and R GarcıaGaraluz ldquoA system testbed for modeling encrypted video-streaming service performance indicators based on TCPIPmetricsrdquo EURASIP Journal on Wireless Communications andNetworking vol 2017 no 1 2017
[18] M Solera M Toril I Palomo G Gomez and J Poncela ldquoATestbed for Evaluating Video Streaming Services in LTErdquoWireless Personal Communications vol 98 no 3 pp 2753ndash27732018
[19] A Alvarez A Dıaz P Merino and F J Rivas ldquoField mea-surements of mobile services with Android smartphonesrdquoin Proceedings of the IEEE Consumer Communications andNetworking Conference (CCNC rsquo12) pp 105ndash109 Las Vegas NevUSA January 2012
[20] NGMN Alliance ldquoNGMN 5G white paperrdquo 2015 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2015NGMN 5G White Paper V1 0pdf
[21] ldquoInfrastructure and Design for Adaptivity and Flexibilityrdquo inMobile Information Systems Springer 2006
[22] J Nielsen ldquoResponse Times The Three Important Limitsrdquo inUsability Engineering 1993
[23] NGMN Alliance ldquoDefinition of the testing framework for theNGMN 5G pre-commercial networks trialsrdquo 2018 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2018180220 NGMN PreCommTrials Framework definition v1 0pdf
[24] 3GPP TS 26246 ldquoTransparent end-to-end Packet-switchedStreaming Services (PSS) Progressive Download and DynamicAdaptive Streaming over HTTP (3GP-DASH)rdquo 2018
International Journal of
AerospaceEngineeringHindawiwwwhindawicom Volume 2018
RoboticsJournal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Active and Passive Electronic Components
VLSI Design
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Shock and Vibration
Hindawiwwwhindawicom Volume 2018
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawiwwwhindawicom
Volume 2018
Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom
The Scientific World Journal
Volume 2018
Control Scienceand Engineering
Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom
Journal ofEngineeringVolume 2018
SensorsJournal of
Hindawiwwwhindawicom Volume 2018
International Journal of
RotatingMachinery
Hindawiwwwhindawicom Volume 2018
Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Navigation and Observation
International Journal of
Hindawi
wwwhindawicom Volume 2018
Advances in
Multimedia
Submit your manuscripts atwwwhindawicom
Wireless Communications and Mobile Computing 3
velocity on-body sensors etc) and QoS parameters (CPUmemory signal strength throughput etc) and provides aninterface to collect user experience feedback A processingentity receives the (device and network) monitored data andanalyzes the incoming data The objective of this testbedinfrastructure is to study the effects of different networkparameters in the QoE in order to define new estimationmodels for QoE
In [16] the authors evaluated routing protocols BATMANand OLSR to support VoIP and video traffic from a QoSand QoE perspective The evaluation took place by runningexperiments in two different testbeds First experimentswere run in the Omnet++ simulator using the InetManetframework Second the same network topology and networkscenarios were deployed in the Emulab test bench a real(emulated) testbed and the same experiments were carriedout Finally the results of both testbeds (simulated andreal-emulated) were statistically compared in order to findinconsistencies The experiments in the simulated and emu-lated environments showed that BATMAN achieves betterthan OLSR and determined the relation between differentprotocol parameters and their performanceThese results canbe applied to implement network nodes that control in-stackprotocol parameters as a function of the observed traffic
In [17] a testbed to automatically extract a QoE modelof encrypted video streaming services was presented Thetestbed includes a software agent to be installed in theuser device which is able to reproduce the user interactionand collect the end-user application-level measurements thenetwork emulator NetEm which changes the link conditionsemulating the radio or core network and a Probe softwarewhich processes all the traffic at different levels computesthe TCPIP metrics and compares the end-user and networklevel measurements This testbed has been used to automat-ically construct the model (and validate the model) of thevideo performance of encrypted YouTube traffic over a Wi-Fi connection
More recently in [18] Solera et al presented a testbedfor evaluating video streaming services in LTE networks Inparticular the QoE of 3D video streaming services over LTEwas evaluated The testbed consists of a streaming serverthe NetEm network emulator and a streaming client Oneof the main contributions of the work is the extension ofNetEm to better model the characteristics of the packet delayin bursty services such as video streaming Previously torunning the experiments in the emulation-based testbedthe authors carried out a simulation campaign with anLTE simulator to obtain the configuration parameters ofNetEm for four different network scenarios These scenarioscombine different positions of the user in the cell anddifferent network loads From the review of these works itbecomes clear that the setup of a simulation or emulationframework for wireless or mobile environments requires inmany cases a deep understanding of the network scenariosTRIANGLE aims to reduce this effort by providing a set ofpreconfigured real network scenarios and the computationof the MOS in order to allow both researchers and appdevelopers to focus on the evaluation of new apps servicesand devices
3 TRIANGLE
The testbed the test methodology and the set of test caseshave been developed within the European funded TRIAN-GLE project Figure 1 shows the main functional blocks thatmake up the TRIANGLE testbed architecture
To facilitate the use of the TRIANGLE testbed fordifferent objectives (testing benchmarking and certifying)to remotely access the testbed and to gather and presentresults a web portal which offers an intuitive interfacehas been implemented It provides access to the testbedhiding unnecessary complexity to App developers Foradvanced users interested in deeper access to configurationparameters of the testbed elements or the test cases thetestbed offers a direct access to the Keysight TAP (TestingAutomation Platform) which is a programmable sequencerof actions with plugins that expose the configuration andcontrol of the instruments and tools integrated into thetestbed
In addition to the testbed itself TRIANGLE has devel-oped a test methodology and has implemented a set oftest cases which are made available through the portal Toachieve full test case automation all the testbed componentsare under the control of the testbed management frame-work which coordinates their configuration and executionprocesses the measurements made in each test case andcomputes QoE scores for the application tested
In addition as part of the testbed management frame-work each testbed component is controlled through a TAPdriver which serves as bridge between the TAP engine andthe actual component interface The configuration of thedifferent elements of the testbed is determined by the testcase to run within the set of test cases provided as partof TRIANGLE or the customized test cases built by usersThe testbed translates the test cases specific configurationssettings and actions into TAP commands that take care ofcommanding each testbed component
TRIANGLE test cases specify the measurements thatshould be collected to compute the KPI (Key PerformanceIndicators) of the feature under test Some measurementsare obtained directly from measurement instruments butothers require specific probes (either software or hardware)to help extract the specific measurements Software probesrunning on the same device (UE LTE User Equipment) thatthe application under test include DEKRA Agents and theTestelDroid [19] tool from UMA TRIANGLE also providesan instrumentation library so that app developers can delivermeasurement outputs which cannot otherwise be extractedand must be provided by the application itself Hardwareprobes include a power analyzer connected to the UE tomeasure power consumption and the radio access emulatorthat among others provides internal logs about the protocolexchange and radio interface low layers metrics
The radio access (LTE RAN) emulator plays a key role inthe TRIANGLE testbed The testbed RAN is provided by anoff-the-shelf E7515A UXM Wireless Test Set from Keysightan emulator that provides state-of-the-art test features Mostimportant the UXM also provides radio channel emulationfor the downlink radio channel
4 Wireless Communications and Mobile Computing
DBs
Interface and visualization (Portal)
Testbed management
Measurements and datacollections
UE
RAN
Transport
AppEPC Local application
serversApp App
TAP
Orcomposutor
Orchestration
Compositor ExecutorDBs
ETLframework
ETL modules
DEKRADriver
WebDriverDriver
AndroidTap Driver
EPCDriver
AppInstrumentat
ion TAPDriver
iOS TAPDriver
hellipTAP Driver
Figure 1 TRIANGLE testbed architecture
In order to provide an end-to-end system the testbedintegrates a commercial EPC (LTE Evolved Packet Core)from Polaris Networks which includes the main elements ofa standard 3GPP compliant LTE core network that is MME(Mobility Management Entity) SGW (Serving Gateway)PGW (Packet Gateway) HSS (Home Subscriber Server) andPCRF (Policy and Charging Rules Function) In additionthis EPC includes the EPDG (Evolved Packet Data Gateway)and ANDSF (Access Network Discovery and Session Func-tion) components for dual connectivity scenarios The RANemulator is connected to the EPC through the standard S1interface The testbed also offers the possibility of integratingartificial impairments in the interfaces between the corenetwork and the application servers
The Quamotion WebDriver another TRIANGLE ele-ment is able to automate user actions on both iOS andAndroid applications whether they are native hybrid offully web-based This tool is also used to prerecord the apprsquosuser flows which are needed to automate the otherwisemanual user actions in the test cases This completes the fullautomation operation
Finally the testbed also incorporates commercial mobiledevices (UEs) The devices are physically connected to thetestbed In order to preserve the radio conditions configuredat the radio access emulator the RAN emulator is cable con-ducted to the mobile device antenna connector To accuratelymeasure the power consumption theN6705B power analyzerdirectly powers the device Other measurement instrumentsmay be added in the future
4 TRIANGLE Approach
TheTRIANGLE testbed is an end-to-end framework devotedto testing and benchmarking mobile applications servicesand devices The idea behind the testing approach adoptedin the TRIANGLE testbed is to generalize QoE computationand provide a programmatic way of computing it Withthis approach the TRIANGLE testbed can accommodate thecomputation of the QoE for any application
The basic concept in TRIANGLErsquos approach to QoEevaluation is that the quality perceived by the user dependson many aspects (herein called domains) and that thisperception depends on its targeted use case For examplebattery life is critical for patient monitoring applications butless important in live streaming ones
To define the different 5G uses cases TRIANGLE basedits work in the Next Generation Mobile Network (NGMN)Alliance foundational White Paper which specifies theexpected services and network performance in future 5Gnetworks [20] More precisely the TRIANGLE project hasadopted a modular approach subdividing the so-calledldquoNGMN Use-Casesrdquo into blocks The name Use Case waskept in the TRIANGLE approach for describing the appli-cation service or vertical using the network services Thediversification of services expected in 5G requires a concretecategorization to have a sharp picture of what the user will beexpected to interact with This is essential for understandingwhich aspect of the QoE evaluation needs to be addressedThe final use cases categorization was defined in [11] andencompasses both the services normally accessible via mobile
Wireless Communications and Mobile Computing 5
Synthetic MOS
Test case
Scenario 1
Iteration 1
Meas 1
Meas
Meas P
Iteration
Meas 1
Meas
Meas P
Iteration N
Meas 1
Meas
Meas P
Scenario K
Iteration 1
Meas 1
Meas
Meas P
Iteration
Meas 1
Meas
Meas P
Iteration N
Meas 1
Meas
Meas P
KPI 1 Synthetic MOS 1
KPI Synthetic MOS
KPI R Synthetic MOS R
Aggregation Synthetic MOS Scenario 1
KPI 1 Synthetic MOS 1
KPI Synthetic MOS
KPI R Synthetic MOS R
Aggregation Synthetic MOS Scenario K
Figure 2 The process to obtain the synthetic-MOS score in a TRIANGLE test case
Table 1 Uses cases defined in the TRIANGLE project
Identifier Use CaseVR Virtual RealityGA GamingAR Augmented RealityCS Content Distribution Streaming ServicesLS Live Streaming ServicesSN Social NetworkingHS High Speed InternetPM Patient MonitoringES Emergency ServicesSM Smart MeteringSG Smart GridsCV Connected Vehicles
phones (UEs) and the ones that can be integrated in forexample gaming consoles advanced VR gear car units orIoT systems
The TRIANGLE domains group different aspects thatcan affect the final QoE perceived by the users The cur-rent testbed implementation supports three of the severaldomains that have been identified Apps User Experience(AUE) Apps Energy consumption (AEC) and ApplicationsDevice Resources Usage (RES)
Table 1 provides the use cases and Table 2 lists thedomains initially considered in TRIANGLE
To produce data to evaluate the QoE a series of testcases have been designed developed and implemented tobe run on the TRIANGLE testbed Obviously not all testcases are applicable to all applications under test becausenot all applications need or are designed to support all thefunctionalities that can be tested in the testbed In orderto automatically determine the test cases that are applicableto an application under test a questionnaire (identified as
features questionnaire in the portal) equivalent to the classi-cal conformance testing ICS (Implementation ConformanceStatement) has been developed and is accessible through theportal After filling the questionnaire the applicable test planthat is the test campaign with the list of applicable test casesis automatically generated
The sequence of user actions (type swipe tap etc) a userneeds to perform in the terminal (UE) to complete a task (egplay a video) is called the ldquoapp user flowrdquo In order to be ableto automatically run a test case the actual application userflow with the user actions a user would need to perform onthe phone to complete certain tasks defined in the test casealso has to be provided
Each test case univocally defines the conditions of execu-tion the sequence of actions the user would perform (ie theapp user flow) the sequence of actions that the elements ofthe testbed must perform the traffic injected the collectionof measurements to take and so forth In order to obtainstatistical significance each test case includes a numberof executions (iterations) under certain network conditions(herein called scenarios) Out of the various measurementsmade in the different iterations under any specific networkconditions (scenario) a number of KPIs (Key PerformanceIndicators) are computed The KPIs are normalized into astandard 1-to-5 scale as typically used in MOS scores andreferred to as synthetic-MOS a terminology that has beenadopted from previous works [7 21] The synthetic-MOSvalues are aggregated across network scenarios to produce anumber of intermediate synthetic-MOS scores which finallyare aggregated to obtain a synthetic-MOS score in each testcase (see Figure 2)
The process to obtain the final TRIANGLE mark issequential First for each domain a weighted average ofthe synthetic-MOS scores obtained in each test case inthe domain is calculated Next a weighted average of thesynthetic-MOS values in all the domains of a use case iscalculated to provide a single synthetic-MOS value per use
6 Wireless Communications and Mobile Computing
Table 2 TRIANGLE domains
Category Identifier Domain
Applications
AUE Apps User experienceAEC Apps Energy consumptionRES Device Resources UsageREL ReliabilityNWR Network Resources
Devices
Mobile Devices
DEC Energy ConsumptionDDP Data PerformanceDRF Radio PerformanceDRA User experience with reference apps
IoT DevicesIDR ReliabilityIDP Data PerformanceIEC Energy consumption
Synthetic MOS Domain A Use Case X
Synthetic MOS Domain B Use Case X
Synthetic MOS Use Case X
Synthetic MOS Use Case Y
TRIANGLE MARK
App
Synthetic MOS Test CaseDomain AUse Case X01
Synthetic MOS Test CaseDomain AUse Case X02
Synthetic MOS Test CaseDomain BUse Case X01
Synthetic MOS Test CaseDomain BUse Case X02
Synthetic MOS Test CaseDomain AUse Case Y01 Synthetic MOS Domain A Use Case Y
Synthetic MOS Test CaseDomain BUse Case Y01 Synthetic MOS Domain B Use Case Y
Figure 3 The process to obtain the TRIANGLE mark
case An application will usually be developed for one specificuse case as those defined in Table 1 but may be designed formore than one use case In the latter case a further weightedaverage is made with the synthetic-MOS scores obtained ineach use case supported by the application These sequentialsteps produce a single TRIANGLE mark an overall qualityscore as shown in Figure 3
This approach provides a common framework for testingapplications for benchmarking applications or even forcertifying disparate applications The overall process for anapp that implements features of different use cases is depictedin Figure 3
5 Details of the TRIANGLE QoE Computation
For each use case identified (see Table 1) and domain (seeTable 2) a number of test cases have been developed withinthe TRIANGLE project Each test case intends to test an
individual feature aspect or behaviour of the applicationunder test as shown in Figure 4
Each test case defines a number of measurements andbecause the results of the measurements depend on manyfactors they are not in general deterministic and thuseach test case has been designed not to perform just onesingle measurement but to run a number of iterations (N)of the same measurement Out of those measurements KPIsare computed For example if the time to load the firstmedia frame is the measurement taken in one specific testcase the average user waiting time KPI can be calculated bycomputing the mean of the values across all iterations Ingeneral different use case-domain pairs have a different set ofKPIsThe reader is encouraged to read [11] for further detailsabout the terminology used in TRIANGLE
Recommendation P10G100 Amendment 1 Definition ofQuality of Experience [2] notes that the overall acceptabilitymay be influenced by user expectations and context Forthe definition of the context technical specifications ITU-T
Wireless Communications and Mobile Computing 7
Feature Non-interactiveplayback
bullTime to load firstmedia frame
bullPlayback Cut-off
bullVideo resolution
bullContent Stall
i) Uses cases
vi) Synthetic MOS iv) Context
Networkscenarios
App User Flow Measurements KPIs MOS
ii) Domains
Apps Energy Consumption
Apps Device Resources
Apps User Experience
Mea
sure
men
ts
v) Test case execution
iii)Test caseAUECS01
Figure 4 QoE computation steps
G1030 ldquoEstimating end-to-end performance in IP networksfor data applicationsrdquo [9] and ITU-T G1031 ldquoQoE factors inweb-browsingrdquo [10] have been considered in TRIANGLEIn particular ITU-T G1031 [10] identifies the following con-text influence factors location (cafeteria office and home)interactivity (high-level interactivity versus low-level inter-activity) task type (business entertainment etc) and taskurgency (urgent versus casual) Userrsquos influence factors arehowever outside of the scope of the ITU recommendation
In the TRIANGLE project the context information hasbeen captured in the networks scenarios defined (Urban -Internet Cafe Off Peak Suburban - Shopping Mall BusyHours Urban ndash Pedestrian Urban ndash Office High speed trainndash Relay etc) and in the test cases specified in [11]
The test cases specify the conditions of the test butalso a sequence of actions that have to be executed by theapplication (app user flows) to test its features For examplethe test case that tests the ldquoPlay and Pauserdquo functionalitydefines the app user flow shown in Figure 5
The transformation of KPIs into QoE scores is the mostchallenging step in the TRIANGLE framework The execu-tion of the test cases will generate a significant amount of rawmeasurements about several aspects of the system SpecificKPIs can then be extracted through statistical analysis meandeviation cumulative distribution function (CDF) or ratio
TheKPIs will be individually interpolated in order to pro-vide a common homogeneous comparison and aggregationspace The interpolation is based on the application of twofunctions named Type I and Type II By using the proposedtwo types of interpolations the vast majority of KPIs can betranslated into normalized MOS-type of metric (synthetic-MOS) easy to be averaged in order to provide a simpleunified evaluation
Type I This function performs a linear interpolation on theoriginal data The variables 119898119894119899
119870119875119868and119898119886119909
119870119875119868are the worst
and best known values of a KPI from a reference case The
Perform login step (ifrequired) and wait for
10 seconds
Start playing avideo of 5
minutes during10 seconds
Pause thereproduction
Resume thereproduction after
2 minutes anduntil the end of
the video
Figure 5 App user flow used in the ldquoAUECS02 Play and Pauserdquotest case
function maps a value v of a KPI to vrsquo (synthetic-MOS) inthe range [1-to-5] by computing the following formula
V1015840 =V minus 119898119894119899
119870119875119868
119898119886119909119870119875119868minus 119898119894119899
119870119875119868
(50 minus 10) + 10 (1)
This function transforms a KPI to a synthetic-MOS value byapplying a simple linear interpolation between the worst andbest expected values from a reference case If a future inputcase falls outside the data range of the KPI the new value will
8 Wireless Communications and Mobile Computing
Table 3 AUECS002 test case description
Identifier AUECS002 (App User ExperienceContent Streaming002)Title Play and pauseObjective Measure the ability of the AUT to pause and the resume a media fileApplicability (ICSG ProductType = Application) AND (ICSG UseCases includes CS) AND ICSA CSPauseInitial Conditions AUT in in [AUT STARTED] mode (Note Defined in D22 [11] Appendix 4)
Steps(1) The Test System commands the AUT to replay the Application User Flow (Application User Flow that
presses first the Play button and later the Pause button)(2) The Test System measures whether pause operation was successful or not
Postamble (i) Execute the Postamble sequence (see section 26 in D22 [11] Appendix 4)
Measurements (Raw)
(i) Playback Cut-off Probability that successfully started stream reproduction is ended by a cause other thanthe intentional termination by the user
(ii) Pause Operation Whether pause operation is successful or not(iii) Time to load first media frame (s) after resuming The time elapsed since the user clicks resume button
until the media reproduction starts(Note For Exoplayer the RESUME button is the PLAY button)
be set to the extreme value minKPI (if it is worse) or maxKPI(if it is better)
Type II This function performs a logarithmic interpolationand is inspired on the opinion model recommended by theITU-T in [9] for a simple web search taskThis function mapsa value v of a KPI to vrsquo (synthetic-MOS) in the range [1-to-5]by computing the following formula
V1015840 =50 minus 10
ln ((119886 lowast 119908119900119903119904119905119870119875119868+ 119887) 119908119900119903119904119905
119870119875119868)
∙ (ln (V) minus ln (119886 lowast 119908119900119903119904119905119870119875119868+ 119886)) + 5
(2)
The default values of 119886 and 119887 correspond to the simple websearch task case (119886 = 0003 and 119887 = 012) [9 22] and theworst value has been extracted from the ITU-T G1030 Ifduring experimentation a future input case falls outside thedata range of the KPI the parameters 119886 and 119887will be updatedaccordingly Likewise if through subjective experimentationother values are considered better adjustments for specificservices the function can be easily updated
Once all KPIs are translated into synthetic-MOS valuesthey can be averaged with suitable weights In the averagingprocess the first step is to average over the network scenariosconsidered relevant for the use case as shown in Figure 2This provides the synthetic-MOS output value for the testcase If there is more than one test case per domain which isgenerally the case a weighted average is calculated in order toprovide one synthetic-MOS value per domain as depicted inFigure 3The final step is to average the synthetic-MOS scoresover all use cases supported by the application (see Figure 3)This provides the final score that is the TRIANGLE mark
6 A Practical Case Exoplayer under Test
For better understanding the complete process of obtainingtheTRIANGLEmark for a specific application the Exoplayer
Table4Measurement points associatedwith test caseAUECS002
Measurements Measurement points
Time to load first media frame Media File Playback - StartMedia File Playback - First Picture
Playback cut-off Media File Playback - StartMedia File Playback - End
Pause Media File Playback - Pause
is described in this section This application only has one usecase content distribution streaming services (CS)
Exoplayer is an application levelmedia player forAndroidpromoted by Google It provides an alternative to AndroidrsquosMediaPlayer API for playing audio and video both locally andover the Internet Exoplayer supports features not currentlysupported by Androidrsquos MediaPlayer API including DASHand SmoothStreaming adaptive playbacks
The TRIANGLE project has concentrated in testing justtwo of the Exoplayer features ldquoNoninteractive Playbackrdquoand ldquoPlay and Pauserdquo These features result in 6 test casesapplicable out of the test cases defined in TRIANGLETheseare test cases AUECS001 and AUECS002 in the App UserExperience domain test casesAECCS001 andAECCS002in the App Energy Consumption domain and test casesRESCS001 and RESCS002 in the Device Resources Usagedomain
The AUECS002 ldquoPlay and Pauserdquo test case descriptionbelonging to the AUE domain is shown in Table 3 The testcase description specifies the test conditions the generic appuser flow and the rawmeasurements which shall be collectedduring the execution of the test
The TRIANGLE project also offers a library that includesthe measurement points that should be inserted in thesource code of the app for enabling the collection of themeasurements specified Table 4 shows the measurementpoints required to compute the measurements specified intest case AUECS002
Wireless Communications and Mobile Computing 9
Table 5 Reference values for interpolation
Feature Domain KPI Synthetic MOS Calculation KPI min KPI maxNon-Interactive Playback AEC Average power consumption Type I 10 W 08 WNon-Interactive Playback AUE Time to load first media frame Type II KPI worst=20 msNon-Interactive Playback AUE Playback cut-off ratio Type I 50 0Non-Interactive Playback AUE Video resolution Type I 240p 720pNon-Interactive Playback RES Average CPU usage Type I 100 16Non-Interactive Playback RES Average memory usage Type I 100 40Play and Pause AEC Average power consumption Type I 10 W 08 WPlay and Pause AUE Pause operation success rate Type I 50 100Play and Pause RES Average CPU usage Type I 100 16Play and Pause RES Average memory usage Type I 100 40
The time to load first media picture measurement isobtained subtracting the timestamp of the measurementpoint ldquoMedia File Playback ndash Startrdquo from the measurementpoint ldquoMedia File Playback ndash First Picturerdquo
As specified in [11] all scenarios defined are applicableto the content streaming use case Therefore test cases inthe three domains currently supported by the testbed areexecuted in all the scenarios
Once the test campaign has finished the raw measure-ment results are processed to obtain the KPIs associated witheach test case average current consumption average time toload first media frame average CPU usage and so forth Theprocesses applied are detailed in Table 5 Based on previousexperiments performed by the authors the behaviour of thetime to load the first media frame KPI resembles the webresponse time KPI (ie the amount of time the user hasto wait for the service) and thus as recommended in theopinionmodel forweb search introduced in [9] a logarithmicinterpolation (type II) has been used for this metric
The results of the initial process that is the KPIs compu-tation are translated into synthetics-MOS values To computethese values reference benchmarking values for each of theKPIs need to be used according to the normalization andinterpolation process described in Section 5 Table 5 showswhat has been currently used by TRIANGLE for the AppUser Experience domain which is also used by NGMN asreference in their precommercial Trials document [23]
For example for the ldquotime to load first media framerdquo KPIshown in Table 5 the type of aggregation applied is averagingand the interpolation formula used is Type II
To achieve stable results each test case is executed 10times (10 iterations) in each network scenario The synthetic-MOS value in each domain is calculated by averaging themeasured synthetic-MOS values in the domain For examplesynthetic-MOS value is the RES domain obtained by aver-aging the synthetic-MOS value of ldquoaverage CPU usagerdquo andldquoaverage memory usagerdquo from the two test cases
Although Exoplayer supports several video streamingprotocols in this work only DASH [24] (Dynamic AdaptiveStreaming over HTTP) has been tested DASH clients shouldseamlessly adapt to changing network conditions by makingdecisions on which video segment to download (videosare encoded at multiple bitrates) The Exoplayerrsquos default
000
000
0
001
728
0
003
456
0
005
184
0
010
912
0
012
640
0
014
368
0
020
096
0
Timestamp
Video Resolution
0
200
400
600
800
1000
1200
Hor
izon
tal R
esol
utio
n
Figure 6 Video Resolution evolution in the Driving Urban Normalscenario
adaptation algorithm is basically throughput-based and someparameters control how often and when switching can occur
During the testing the testbed was configured with thedifferent network scenarios defined in [11] In these scenariosthe network configuration changes dynamically following arandom pattern resulting in different maximum throughputrates The expected behaviour of the application under testis that the video streaming client adapts to the availablethroughput by decreasing or increasing the resolution of thereceived video Figure 6 depicts how the client effectivelyadapts to the channel conditions
However the objective of the testing carried out in theTRIANGE testbed is not just to verify that the video stream-ing client actually adapts to the available maximum through-put but also to check whether this adaptation improves theusersrsquo experience quality
Table 6 shows a summary of the synthetic-MOS valuesobtained per scenario in one test case of each domain Thescores obtained in the RES andAECdomains are always highIn the AUE domain the synthetic MOS associated with theVideo Resolution shows low scores in some of the scenariosbecause the resolution decreases reasonable good scores inthe time to load first media and high scores in the time toplayback cut-off ratio Overall it can be concluded that the
10 Wireless Communications and Mobile Computing
Table 6 Synthetic MOS values per test case and scenario for the feature ldquoNoninteractive Playbackrdquo
AUE domain AEC domain RES domain
Test Case AUECS001 Test CaseAECCS001 Test Case RESCS001
ScenarioTime to loadfirst mediaframe
PlaybackCut-off ratio
VideoResolution
mode
AveragePower
Consumption
Average CPUUsage
AverageRAM Usage
HighSpeed DirectPassenger 21 31 23 47 43 42
Suburban Festival 38 47 31 48 43 41Suburban shopping mallbusy hours 37 37 13 48 44 41
Suburban shopping malloff-peak 36 31 23 48 43 41
Suburban stadium 38 29 21 47 44 41Urban Driving Normal 26 39 28 47 44 4Urban Driving TrafficJam 34 37 16 48 44 4
Urban Internet Cafe BusyHours 38 37 19 48 44 4
Urban Internet Cafe OffPeak 38 31 23 48 43 4
Urban Office 38 47 33 48 45 43Urban Pedestrian 39 26 2 47 44 4
35 36 23 47 44 41
DASH implementation of the video streaming client undertest is able to adapt to the changing conditions of the networkmaintaining an acceptable rate of video cut-off rebufferingtimes and resources usage
The final score in each domain is obtained by averagingthe synthetic-MOS values from all the tested network scenar-ios Figure 7 shows the spider diagram for the three domainstested In the User Experience domain the score obtained islower than the other domains due to the low synthetic-MOSvalues obtained for the video resolution
The final synthetic MOS for the use case Content Dis-tribution Streaming is obtained as a weighted average of thethree domains representing the overall QoE as perceived bythe userThefinal score for the Exoplayer version 1516 and thefeatures tested (Noninteractive Playback and Play and Pause)is 42 which means that the low score obtained in the videoresolution is compensated with the high scores in other KPIs
If an application under test has more than one use casethe next steps in the TRIANGLE mark project approachwould be the aggregation per use case and the aggregationover all use cases The final score the TRIANGLE mark is anestimation of the overall QoE as perceived by the user
In the current TRIANGLE implementation the weightsin all aggregations are the same Further research is neededto appropriately define the weights of each domain and eachuse case in the overall score of the applications
7 Conclusions
The main contribution of the TRIANGLE project is theprovision of a framework that generalizes QoE computation
and enables the execution of extensive and repeatable testcampaigns to obtainmeaningfulQoE scoresTheTRIANGLEproject has also defined amethodology which is based on thetransformation and aggregation of KPIs its transformationinto synthetic-MOS values and its aggregation over thedifferent domains and use cases
The TRIANGLE approach is a methodology flexibleenough to generalize the computation of QoE for any applica-tionservice Themethodology has been validated testing theDASH implementation in the Exoplayer App To confirm thesuitability of theweights used in the averaging process and theinterpolation parameters as well as to verify the correlationof the obtained MOS with that scored by users the authorshave started experiments with real users and initial results areencouraging
The process described produces a final TRIANGLEmarka single quality score which could eventually be used to cer-tify applications after achieving a consensus on the differentvalues of the process (weights limits etc) to use
Data Availability
Themethodology and results used to support the findings ofthis study are included within the article
Conflicts of Interest
The authors declare that they have no conflicts of inter-est
Wireless Communications and Mobile Computing 11
DER
AUE AEC
Device Resource Usage
User Experience Energy Consumption
Figure 7 Exoplayer synthetic-MOS values per domain
Acknowledgments
The TRIANGLE project is funded by the European UnionrsquosHorizon 2020 Research and Innovation Programme (GrantAgreement no 688712)
References
[1] ETSI ldquoHuman factors quality of experience (QoE) require-ments for real-time communication servicesrdquo Tech Rep 102643 2010
[2] ITU-T ldquoP10G100 (2006) amendment 1 (0107) new appendixI - definition of quality of experience (QoE)rdquo 2007
[3] F Kozamernik V Steinmann P Sunna and E WyckensldquoSAMVIQ - A new EBUmethodology for video quality evalua-tions in multimediardquo SMPTE Motion Imaging Journal vol 114no 4 pp 152ndash160 2005
[4] ITU-T ldquoG107 the E-model a computational model for use intransmission planningrdquo 2015
[5] J De Vriendt D De Vleeschauwer and D C Robinson ldquoQoEmodel for video delivered over an LTE network using HTTPadaptive streamingrdquo Bell Labs Technical Journal vol 18 no 4pp 45ndash62 2014
[6] S Jelassi G Rubino H Melvin H Youssef and G PujolleldquoQuality of Experience of VoIP Service A Survey of AssessmentApproaches andOpen Issuesrdquo IEEECommunications Surveys ampTutorials vol 14 no 2 pp 491ndash513 2012
[7] M Li C-L Yeh and S-Y Lu ldquoReal-Time QoE MonitoringSystem forVideo Streaming ServiceswithAdaptiveMedia Play-outrdquo International Journal of Digital Multimedia Broadcastingvol 2018 Article ID 2619438 11 pages 2018
[8] S Barakovic and L Skorin-Kapov ldquoSurvey and Challengesof QoE Management Issues in Wireless Networksrdquo Journal ofComputer Networks and Communications vol 2013 Article ID165146 28 pages 2013
[9] ITU-T ldquoG1030 estimating end-to-end performance in IPnetworks for data applicationsrdquo 2014
[10] ITU-T ldquoG1031 QoE factors in web-browsingrdquo 2014[11] EU H2020 TRIANGLE Project Deliverable D22 Final report
on the formalization of the certification process requirementsanduse cases 2017 httpswwwtriangle-projecteuproject-olddeliverables
[12] Q A Chen H Luo S Rosen et al ldquoQoE doctor diagnosingmobile app QoE with automated UI control and cross-layeranalysisrdquo in Proceedings of the Conference on Internet Mea-surement Conference (IMC rsquo14) pp 151ndash164 ACM VancouverCanada November 2014
[13] M A Mehmood A Wundsam S Uhlig D Levin N Sarrarand A Feldmann ldquoQoE-Lab Towards Evaluating Quality ofExperience for Future Internet Conditionsrdquo in Testbeds andResearch Infrastructure Korakis T Li H Tran-Gia P and HS Park Eds vol 90 of TridentCom 2011 Lnicst pp 286ndash301Springer Development of Networks and Communities BerlinGermany 2012
[14] D Levin A Wundsam A Mehmood and A FeldmannldquoBerlin The Berlin Experimental Router Laboratory for Inno-vative Networkingrdquo in TridentCom 2010 Lnicst T MagedanzA Gavras N H Thanh and J S Chase Eds vol 46 of LectureNotes of the Institute for Computer Sciences Social Informaticsand Telecommunications Engineering pp 602ndash604 SpringerHeidelberg Germany 2011
12 Wireless Communications and Mobile Computing
[15] K De Moor I Ketyko W Joseph et al ldquoProposed frameworkfor evaluating quality of experience in a mobile testbed-oriented living lab settingrdquo Mobile Networks and Applicationsvol 15 no 3 pp 378ndash391 2010
[16] R Sanchez-Iborra M-D Cano J J P C Rodrigues and JGarcia-Haro ldquoAnExperimental QoE Performance Study for theEfficient Transmission of High Demanding Traffic over an AdHoc Network Using BATMANrdquo Mobile Information Systemsvol 2015 Article ID 217106 14 pages 2015
[17] P Oliver-Balsalobre M Toril S Luna-Ramırez and R GarcıaGaraluz ldquoA system testbed for modeling encrypted video-streaming service performance indicators based on TCPIPmetricsrdquo EURASIP Journal on Wireless Communications andNetworking vol 2017 no 1 2017
[18] M Solera M Toril I Palomo G Gomez and J Poncela ldquoATestbed for Evaluating Video Streaming Services in LTErdquoWireless Personal Communications vol 98 no 3 pp 2753ndash27732018
[19] A Alvarez A Dıaz P Merino and F J Rivas ldquoField mea-surements of mobile services with Android smartphonesrdquoin Proceedings of the IEEE Consumer Communications andNetworking Conference (CCNC rsquo12) pp 105ndash109 Las Vegas NevUSA January 2012
[20] NGMN Alliance ldquoNGMN 5G white paperrdquo 2015 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2015NGMN 5G White Paper V1 0pdf
[21] ldquoInfrastructure and Design for Adaptivity and Flexibilityrdquo inMobile Information Systems Springer 2006
[22] J Nielsen ldquoResponse Times The Three Important Limitsrdquo inUsability Engineering 1993
[23] NGMN Alliance ldquoDefinition of the testing framework for theNGMN 5G pre-commercial networks trialsrdquo 2018 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2018180220 NGMN PreCommTrials Framework definition v1 0pdf
[24] 3GPP TS 26246 ldquoTransparent end-to-end Packet-switchedStreaming Services (PSS) Progressive Download and DynamicAdaptive Streaming over HTTP (3GP-DASH)rdquo 2018
International Journal of
AerospaceEngineeringHindawiwwwhindawicom Volume 2018
RoboticsJournal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Active and Passive Electronic Components
VLSI Design
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Shock and Vibration
Hindawiwwwhindawicom Volume 2018
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawiwwwhindawicom
Volume 2018
Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom
The Scientific World Journal
Volume 2018
Control Scienceand Engineering
Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom
Journal ofEngineeringVolume 2018
SensorsJournal of
Hindawiwwwhindawicom Volume 2018
International Journal of
RotatingMachinery
Hindawiwwwhindawicom Volume 2018
Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Navigation and Observation
International Journal of
Hindawi
wwwhindawicom Volume 2018
Advances in
Multimedia
Submit your manuscripts atwwwhindawicom
4 Wireless Communications and Mobile Computing
DBs
Interface and visualization (Portal)
Testbed management
Measurements and datacollections
UE
RAN
Transport
AppEPC Local application
serversApp App
TAP
Orcomposutor
Orchestration
Compositor ExecutorDBs
ETLframework
ETL modules
DEKRADriver
WebDriverDriver
AndroidTap Driver
EPCDriver
AppInstrumentat
ion TAPDriver
iOS TAPDriver
hellipTAP Driver
Figure 1 TRIANGLE testbed architecture
In order to provide an end-to-end system the testbedintegrates a commercial EPC (LTE Evolved Packet Core)from Polaris Networks which includes the main elements ofa standard 3GPP compliant LTE core network that is MME(Mobility Management Entity) SGW (Serving Gateway)PGW (Packet Gateway) HSS (Home Subscriber Server) andPCRF (Policy and Charging Rules Function) In additionthis EPC includes the EPDG (Evolved Packet Data Gateway)and ANDSF (Access Network Discovery and Session Func-tion) components for dual connectivity scenarios The RANemulator is connected to the EPC through the standard S1interface The testbed also offers the possibility of integratingartificial impairments in the interfaces between the corenetwork and the application servers
The Quamotion WebDriver another TRIANGLE ele-ment is able to automate user actions on both iOS andAndroid applications whether they are native hybrid offully web-based This tool is also used to prerecord the apprsquosuser flows which are needed to automate the otherwisemanual user actions in the test cases This completes the fullautomation operation
Finally the testbed also incorporates commercial mobiledevices (UEs) The devices are physically connected to thetestbed In order to preserve the radio conditions configuredat the radio access emulator the RAN emulator is cable con-ducted to the mobile device antenna connector To accuratelymeasure the power consumption theN6705B power analyzerdirectly powers the device Other measurement instrumentsmay be added in the future
4 TRIANGLE Approach
TheTRIANGLE testbed is an end-to-end framework devotedto testing and benchmarking mobile applications servicesand devices The idea behind the testing approach adoptedin the TRIANGLE testbed is to generalize QoE computationand provide a programmatic way of computing it Withthis approach the TRIANGLE testbed can accommodate thecomputation of the QoE for any application
The basic concept in TRIANGLErsquos approach to QoEevaluation is that the quality perceived by the user dependson many aspects (herein called domains) and that thisperception depends on its targeted use case For examplebattery life is critical for patient monitoring applications butless important in live streaming ones
To define the different 5G uses cases TRIANGLE basedits work in the Next Generation Mobile Network (NGMN)Alliance foundational White Paper which specifies theexpected services and network performance in future 5Gnetworks [20] More precisely the TRIANGLE project hasadopted a modular approach subdividing the so-calledldquoNGMN Use-Casesrdquo into blocks The name Use Case waskept in the TRIANGLE approach for describing the appli-cation service or vertical using the network services Thediversification of services expected in 5G requires a concretecategorization to have a sharp picture of what the user will beexpected to interact with This is essential for understandingwhich aspect of the QoE evaluation needs to be addressedThe final use cases categorization was defined in [11] andencompasses both the services normally accessible via mobile
Wireless Communications and Mobile Computing 5
Synthetic MOS
Test case
Scenario 1
Iteration 1
Meas 1
Meas
Meas P
Iteration
Meas 1
Meas
Meas P
Iteration N
Meas 1
Meas
Meas P
Scenario K
Iteration 1
Meas 1
Meas
Meas P
Iteration
Meas 1
Meas
Meas P
Iteration N
Meas 1
Meas
Meas P
KPI 1 Synthetic MOS 1
KPI Synthetic MOS
KPI R Synthetic MOS R
Aggregation Synthetic MOS Scenario 1
KPI 1 Synthetic MOS 1
KPI Synthetic MOS
KPI R Synthetic MOS R
Aggregation Synthetic MOS Scenario K
Figure 2 The process to obtain the synthetic-MOS score in a TRIANGLE test case
Table 1 Uses cases defined in the TRIANGLE project
Identifier Use CaseVR Virtual RealityGA GamingAR Augmented RealityCS Content Distribution Streaming ServicesLS Live Streaming ServicesSN Social NetworkingHS High Speed InternetPM Patient MonitoringES Emergency ServicesSM Smart MeteringSG Smart GridsCV Connected Vehicles
phones (UEs) and the ones that can be integrated in forexample gaming consoles advanced VR gear car units orIoT systems
The TRIANGLE domains group different aspects thatcan affect the final QoE perceived by the users The cur-rent testbed implementation supports three of the severaldomains that have been identified Apps User Experience(AUE) Apps Energy consumption (AEC) and ApplicationsDevice Resources Usage (RES)
Table 1 provides the use cases and Table 2 lists thedomains initially considered in TRIANGLE
To produce data to evaluate the QoE a series of testcases have been designed developed and implemented tobe run on the TRIANGLE testbed Obviously not all testcases are applicable to all applications under test becausenot all applications need or are designed to support all thefunctionalities that can be tested in the testbed In orderto automatically determine the test cases that are applicableto an application under test a questionnaire (identified as
features questionnaire in the portal) equivalent to the classi-cal conformance testing ICS (Implementation ConformanceStatement) has been developed and is accessible through theportal After filling the questionnaire the applicable test planthat is the test campaign with the list of applicable test casesis automatically generated
The sequence of user actions (type swipe tap etc) a userneeds to perform in the terminal (UE) to complete a task (egplay a video) is called the ldquoapp user flowrdquo In order to be ableto automatically run a test case the actual application userflow with the user actions a user would need to perform onthe phone to complete certain tasks defined in the test casealso has to be provided
Each test case univocally defines the conditions of execu-tion the sequence of actions the user would perform (ie theapp user flow) the sequence of actions that the elements ofthe testbed must perform the traffic injected the collectionof measurements to take and so forth In order to obtainstatistical significance each test case includes a numberof executions (iterations) under certain network conditions(herein called scenarios) Out of the various measurementsmade in the different iterations under any specific networkconditions (scenario) a number of KPIs (Key PerformanceIndicators) are computed The KPIs are normalized into astandard 1-to-5 scale as typically used in MOS scores andreferred to as synthetic-MOS a terminology that has beenadopted from previous works [7 21] The synthetic-MOSvalues are aggregated across network scenarios to produce anumber of intermediate synthetic-MOS scores which finallyare aggregated to obtain a synthetic-MOS score in each testcase (see Figure 2)
The process to obtain the final TRIANGLE mark issequential First for each domain a weighted average ofthe synthetic-MOS scores obtained in each test case inthe domain is calculated Next a weighted average of thesynthetic-MOS values in all the domains of a use case iscalculated to provide a single synthetic-MOS value per use
6 Wireless Communications and Mobile Computing
Table 2 TRIANGLE domains
Category Identifier Domain
Applications
AUE Apps User experienceAEC Apps Energy consumptionRES Device Resources UsageREL ReliabilityNWR Network Resources
Devices
Mobile Devices
DEC Energy ConsumptionDDP Data PerformanceDRF Radio PerformanceDRA User experience with reference apps
IoT DevicesIDR ReliabilityIDP Data PerformanceIEC Energy consumption
Synthetic MOS Domain A Use Case X
Synthetic MOS Domain B Use Case X
Synthetic MOS Use Case X
Synthetic MOS Use Case Y
TRIANGLE MARK
App
Synthetic MOS Test CaseDomain AUse Case X01
Synthetic MOS Test CaseDomain AUse Case X02
Synthetic MOS Test CaseDomain BUse Case X01
Synthetic MOS Test CaseDomain BUse Case X02
Synthetic MOS Test CaseDomain AUse Case Y01 Synthetic MOS Domain A Use Case Y
Synthetic MOS Test CaseDomain BUse Case Y01 Synthetic MOS Domain B Use Case Y
Figure 3 The process to obtain the TRIANGLE mark
case An application will usually be developed for one specificuse case as those defined in Table 1 but may be designed formore than one use case In the latter case a further weightedaverage is made with the synthetic-MOS scores obtained ineach use case supported by the application These sequentialsteps produce a single TRIANGLE mark an overall qualityscore as shown in Figure 3
This approach provides a common framework for testingapplications for benchmarking applications or even forcertifying disparate applications The overall process for anapp that implements features of different use cases is depictedin Figure 3
5 Details of the TRIANGLE QoE Computation
For each use case identified (see Table 1) and domain (seeTable 2) a number of test cases have been developed withinthe TRIANGLE project Each test case intends to test an
individual feature aspect or behaviour of the applicationunder test as shown in Figure 4
Each test case defines a number of measurements andbecause the results of the measurements depend on manyfactors they are not in general deterministic and thuseach test case has been designed not to perform just onesingle measurement but to run a number of iterations (N)of the same measurement Out of those measurements KPIsare computed For example if the time to load the firstmedia frame is the measurement taken in one specific testcase the average user waiting time KPI can be calculated bycomputing the mean of the values across all iterations Ingeneral different use case-domain pairs have a different set ofKPIsThe reader is encouraged to read [11] for further detailsabout the terminology used in TRIANGLE
Recommendation P10G100 Amendment 1 Definition ofQuality of Experience [2] notes that the overall acceptabilitymay be influenced by user expectations and context Forthe definition of the context technical specifications ITU-T
Wireless Communications and Mobile Computing 7
Feature Non-interactiveplayback
bullTime to load firstmedia frame
bullPlayback Cut-off
bullVideo resolution
bullContent Stall
i) Uses cases
vi) Synthetic MOS iv) Context
Networkscenarios
App User Flow Measurements KPIs MOS
ii) Domains
Apps Energy Consumption
Apps Device Resources
Apps User Experience
Mea
sure
men
ts
v) Test case execution
iii)Test caseAUECS01
Figure 4 QoE computation steps
G1030 ldquoEstimating end-to-end performance in IP networksfor data applicationsrdquo [9] and ITU-T G1031 ldquoQoE factors inweb-browsingrdquo [10] have been considered in TRIANGLEIn particular ITU-T G1031 [10] identifies the following con-text influence factors location (cafeteria office and home)interactivity (high-level interactivity versus low-level inter-activity) task type (business entertainment etc) and taskurgency (urgent versus casual) Userrsquos influence factors arehowever outside of the scope of the ITU recommendation
In the TRIANGLE project the context information hasbeen captured in the networks scenarios defined (Urban -Internet Cafe Off Peak Suburban - Shopping Mall BusyHours Urban ndash Pedestrian Urban ndash Office High speed trainndash Relay etc) and in the test cases specified in [11]
The test cases specify the conditions of the test butalso a sequence of actions that have to be executed by theapplication (app user flows) to test its features For examplethe test case that tests the ldquoPlay and Pauserdquo functionalitydefines the app user flow shown in Figure 5
The transformation of KPIs into QoE scores is the mostchallenging step in the TRIANGLE framework The execu-tion of the test cases will generate a significant amount of rawmeasurements about several aspects of the system SpecificKPIs can then be extracted through statistical analysis meandeviation cumulative distribution function (CDF) or ratio
TheKPIs will be individually interpolated in order to pro-vide a common homogeneous comparison and aggregationspace The interpolation is based on the application of twofunctions named Type I and Type II By using the proposedtwo types of interpolations the vast majority of KPIs can betranslated into normalized MOS-type of metric (synthetic-MOS) easy to be averaged in order to provide a simpleunified evaluation
Type I This function performs a linear interpolation on theoriginal data The variables 119898119894119899
119870119875119868and119898119886119909
119870119875119868are the worst
and best known values of a KPI from a reference case The
Perform login step (ifrequired) and wait for
10 seconds
Start playing avideo of 5
minutes during10 seconds
Pause thereproduction
Resume thereproduction after
2 minutes anduntil the end of
the video
Figure 5 App user flow used in the ldquoAUECS02 Play and Pauserdquotest case
function maps a value v of a KPI to vrsquo (synthetic-MOS) inthe range [1-to-5] by computing the following formula
V1015840 =V minus 119898119894119899
119870119875119868
119898119886119909119870119875119868minus 119898119894119899
119870119875119868
(50 minus 10) + 10 (1)
This function transforms a KPI to a synthetic-MOS value byapplying a simple linear interpolation between the worst andbest expected values from a reference case If a future inputcase falls outside the data range of the KPI the new value will
8 Wireless Communications and Mobile Computing
Table 3 AUECS002 test case description
Identifier AUECS002 (App User ExperienceContent Streaming002)Title Play and pauseObjective Measure the ability of the AUT to pause and the resume a media fileApplicability (ICSG ProductType = Application) AND (ICSG UseCases includes CS) AND ICSA CSPauseInitial Conditions AUT in in [AUT STARTED] mode (Note Defined in D22 [11] Appendix 4)
Steps(1) The Test System commands the AUT to replay the Application User Flow (Application User Flow that
presses first the Play button and later the Pause button)(2) The Test System measures whether pause operation was successful or not
Postamble (i) Execute the Postamble sequence (see section 26 in D22 [11] Appendix 4)
Measurements (Raw)
(i) Playback Cut-off Probability that successfully started stream reproduction is ended by a cause other thanthe intentional termination by the user
(ii) Pause Operation Whether pause operation is successful or not(iii) Time to load first media frame (s) after resuming The time elapsed since the user clicks resume button
until the media reproduction starts(Note For Exoplayer the RESUME button is the PLAY button)
be set to the extreme value minKPI (if it is worse) or maxKPI(if it is better)
Type II This function performs a logarithmic interpolationand is inspired on the opinion model recommended by theITU-T in [9] for a simple web search taskThis function mapsa value v of a KPI to vrsquo (synthetic-MOS) in the range [1-to-5]by computing the following formula
V1015840 =50 minus 10
ln ((119886 lowast 119908119900119903119904119905119870119875119868+ 119887) 119908119900119903119904119905
119870119875119868)
∙ (ln (V) minus ln (119886 lowast 119908119900119903119904119905119870119875119868+ 119886)) + 5
(2)
The default values of 119886 and 119887 correspond to the simple websearch task case (119886 = 0003 and 119887 = 012) [9 22] and theworst value has been extracted from the ITU-T G1030 Ifduring experimentation a future input case falls outside thedata range of the KPI the parameters 119886 and 119887will be updatedaccordingly Likewise if through subjective experimentationother values are considered better adjustments for specificservices the function can be easily updated
Once all KPIs are translated into synthetic-MOS valuesthey can be averaged with suitable weights In the averagingprocess the first step is to average over the network scenariosconsidered relevant for the use case as shown in Figure 2This provides the synthetic-MOS output value for the testcase If there is more than one test case per domain which isgenerally the case a weighted average is calculated in order toprovide one synthetic-MOS value per domain as depicted inFigure 3The final step is to average the synthetic-MOS scoresover all use cases supported by the application (see Figure 3)This provides the final score that is the TRIANGLE mark
6 A Practical Case Exoplayer under Test
For better understanding the complete process of obtainingtheTRIANGLEmark for a specific application the Exoplayer
Table4Measurement points associatedwith test caseAUECS002
Measurements Measurement points
Time to load first media frame Media File Playback - StartMedia File Playback - First Picture
Playback cut-off Media File Playback - StartMedia File Playback - End
Pause Media File Playback - Pause
is described in this section This application only has one usecase content distribution streaming services (CS)
Exoplayer is an application levelmedia player forAndroidpromoted by Google It provides an alternative to AndroidrsquosMediaPlayer API for playing audio and video both locally andover the Internet Exoplayer supports features not currentlysupported by Androidrsquos MediaPlayer API including DASHand SmoothStreaming adaptive playbacks
The TRIANGLE project has concentrated in testing justtwo of the Exoplayer features ldquoNoninteractive Playbackrdquoand ldquoPlay and Pauserdquo These features result in 6 test casesapplicable out of the test cases defined in TRIANGLETheseare test cases AUECS001 and AUECS002 in the App UserExperience domain test casesAECCS001 andAECCS002in the App Energy Consumption domain and test casesRESCS001 and RESCS002 in the Device Resources Usagedomain
The AUECS002 ldquoPlay and Pauserdquo test case descriptionbelonging to the AUE domain is shown in Table 3 The testcase description specifies the test conditions the generic appuser flow and the rawmeasurements which shall be collectedduring the execution of the test
The TRIANGLE project also offers a library that includesthe measurement points that should be inserted in thesource code of the app for enabling the collection of themeasurements specified Table 4 shows the measurementpoints required to compute the measurements specified intest case AUECS002
Wireless Communications and Mobile Computing 9
Table 5 Reference values for interpolation
Feature Domain KPI Synthetic MOS Calculation KPI min KPI maxNon-Interactive Playback AEC Average power consumption Type I 10 W 08 WNon-Interactive Playback AUE Time to load first media frame Type II KPI worst=20 msNon-Interactive Playback AUE Playback cut-off ratio Type I 50 0Non-Interactive Playback AUE Video resolution Type I 240p 720pNon-Interactive Playback RES Average CPU usage Type I 100 16Non-Interactive Playback RES Average memory usage Type I 100 40Play and Pause AEC Average power consumption Type I 10 W 08 WPlay and Pause AUE Pause operation success rate Type I 50 100Play and Pause RES Average CPU usage Type I 100 16Play and Pause RES Average memory usage Type I 100 40
The time to load first media picture measurement isobtained subtracting the timestamp of the measurementpoint ldquoMedia File Playback ndash Startrdquo from the measurementpoint ldquoMedia File Playback ndash First Picturerdquo
As specified in [11] all scenarios defined are applicableto the content streaming use case Therefore test cases inthe three domains currently supported by the testbed areexecuted in all the scenarios
Once the test campaign has finished the raw measure-ment results are processed to obtain the KPIs associated witheach test case average current consumption average time toload first media frame average CPU usage and so forth Theprocesses applied are detailed in Table 5 Based on previousexperiments performed by the authors the behaviour of thetime to load the first media frame KPI resembles the webresponse time KPI (ie the amount of time the user hasto wait for the service) and thus as recommended in theopinionmodel forweb search introduced in [9] a logarithmicinterpolation (type II) has been used for this metric
The results of the initial process that is the KPIs compu-tation are translated into synthetics-MOS values To computethese values reference benchmarking values for each of theKPIs need to be used according to the normalization andinterpolation process described in Section 5 Table 5 showswhat has been currently used by TRIANGLE for the AppUser Experience domain which is also used by NGMN asreference in their precommercial Trials document [23]
For example for the ldquotime to load first media framerdquo KPIshown in Table 5 the type of aggregation applied is averagingand the interpolation formula used is Type II
To achieve stable results each test case is executed 10times (10 iterations) in each network scenario The synthetic-MOS value in each domain is calculated by averaging themeasured synthetic-MOS values in the domain For examplesynthetic-MOS value is the RES domain obtained by aver-aging the synthetic-MOS value of ldquoaverage CPU usagerdquo andldquoaverage memory usagerdquo from the two test cases
Although Exoplayer supports several video streamingprotocols in this work only DASH [24] (Dynamic AdaptiveStreaming over HTTP) has been tested DASH clients shouldseamlessly adapt to changing network conditions by makingdecisions on which video segment to download (videosare encoded at multiple bitrates) The Exoplayerrsquos default
000
000
0
001
728
0
003
456
0
005
184
0
010
912
0
012
640
0
014
368
0
020
096
0
Timestamp
Video Resolution
0
200
400
600
800
1000
1200
Hor
izon
tal R
esol
utio
n
Figure 6 Video Resolution evolution in the Driving Urban Normalscenario
adaptation algorithm is basically throughput-based and someparameters control how often and when switching can occur
During the testing the testbed was configured with thedifferent network scenarios defined in [11] In these scenariosthe network configuration changes dynamically following arandom pattern resulting in different maximum throughputrates The expected behaviour of the application under testis that the video streaming client adapts to the availablethroughput by decreasing or increasing the resolution of thereceived video Figure 6 depicts how the client effectivelyadapts to the channel conditions
However the objective of the testing carried out in theTRIANGE testbed is not just to verify that the video stream-ing client actually adapts to the available maximum through-put but also to check whether this adaptation improves theusersrsquo experience quality
Table 6 shows a summary of the synthetic-MOS valuesobtained per scenario in one test case of each domain Thescores obtained in the RES andAECdomains are always highIn the AUE domain the synthetic MOS associated with theVideo Resolution shows low scores in some of the scenariosbecause the resolution decreases reasonable good scores inthe time to load first media and high scores in the time toplayback cut-off ratio Overall it can be concluded that the
10 Wireless Communications and Mobile Computing
Table 6 Synthetic MOS values per test case and scenario for the feature ldquoNoninteractive Playbackrdquo
AUE domain AEC domain RES domain
Test Case AUECS001 Test CaseAECCS001 Test Case RESCS001
ScenarioTime to loadfirst mediaframe
PlaybackCut-off ratio
VideoResolution
mode
AveragePower
Consumption
Average CPUUsage
AverageRAM Usage
HighSpeed DirectPassenger 21 31 23 47 43 42
Suburban Festival 38 47 31 48 43 41Suburban shopping mallbusy hours 37 37 13 48 44 41
Suburban shopping malloff-peak 36 31 23 48 43 41
Suburban stadium 38 29 21 47 44 41Urban Driving Normal 26 39 28 47 44 4Urban Driving TrafficJam 34 37 16 48 44 4
Urban Internet Cafe BusyHours 38 37 19 48 44 4
Urban Internet Cafe OffPeak 38 31 23 48 43 4
Urban Office 38 47 33 48 45 43Urban Pedestrian 39 26 2 47 44 4
35 36 23 47 44 41
DASH implementation of the video streaming client undertest is able to adapt to the changing conditions of the networkmaintaining an acceptable rate of video cut-off rebufferingtimes and resources usage
The final score in each domain is obtained by averagingthe synthetic-MOS values from all the tested network scenar-ios Figure 7 shows the spider diagram for the three domainstested In the User Experience domain the score obtained islower than the other domains due to the low synthetic-MOSvalues obtained for the video resolution
The final synthetic MOS for the use case Content Dis-tribution Streaming is obtained as a weighted average of thethree domains representing the overall QoE as perceived bythe userThefinal score for the Exoplayer version 1516 and thefeatures tested (Noninteractive Playback and Play and Pause)is 42 which means that the low score obtained in the videoresolution is compensated with the high scores in other KPIs
If an application under test has more than one use casethe next steps in the TRIANGLE mark project approachwould be the aggregation per use case and the aggregationover all use cases The final score the TRIANGLE mark is anestimation of the overall QoE as perceived by the user
In the current TRIANGLE implementation the weightsin all aggregations are the same Further research is neededto appropriately define the weights of each domain and eachuse case in the overall score of the applications
7 Conclusions
The main contribution of the TRIANGLE project is theprovision of a framework that generalizes QoE computation
and enables the execution of extensive and repeatable testcampaigns to obtainmeaningfulQoE scoresTheTRIANGLEproject has also defined amethodology which is based on thetransformation and aggregation of KPIs its transformationinto synthetic-MOS values and its aggregation over thedifferent domains and use cases
The TRIANGLE approach is a methodology flexibleenough to generalize the computation of QoE for any applica-tionservice Themethodology has been validated testing theDASH implementation in the Exoplayer App To confirm thesuitability of theweights used in the averaging process and theinterpolation parameters as well as to verify the correlationof the obtained MOS with that scored by users the authorshave started experiments with real users and initial results areencouraging
The process described produces a final TRIANGLEmarka single quality score which could eventually be used to cer-tify applications after achieving a consensus on the differentvalues of the process (weights limits etc) to use
Data Availability
Themethodology and results used to support the findings ofthis study are included within the article
Conflicts of Interest
The authors declare that they have no conflicts of inter-est
Wireless Communications and Mobile Computing 11
DER
AUE AEC
Device Resource Usage
User Experience Energy Consumption
Figure 7 Exoplayer synthetic-MOS values per domain
Acknowledgments
The TRIANGLE project is funded by the European UnionrsquosHorizon 2020 Research and Innovation Programme (GrantAgreement no 688712)
References
[1] ETSI ldquoHuman factors quality of experience (QoE) require-ments for real-time communication servicesrdquo Tech Rep 102643 2010
[2] ITU-T ldquoP10G100 (2006) amendment 1 (0107) new appendixI - definition of quality of experience (QoE)rdquo 2007
[3] F Kozamernik V Steinmann P Sunna and E WyckensldquoSAMVIQ - A new EBUmethodology for video quality evalua-tions in multimediardquo SMPTE Motion Imaging Journal vol 114no 4 pp 152ndash160 2005
[4] ITU-T ldquoG107 the E-model a computational model for use intransmission planningrdquo 2015
[5] J De Vriendt D De Vleeschauwer and D C Robinson ldquoQoEmodel for video delivered over an LTE network using HTTPadaptive streamingrdquo Bell Labs Technical Journal vol 18 no 4pp 45ndash62 2014
[6] S Jelassi G Rubino H Melvin H Youssef and G PujolleldquoQuality of Experience of VoIP Service A Survey of AssessmentApproaches andOpen Issuesrdquo IEEECommunications Surveys ampTutorials vol 14 no 2 pp 491ndash513 2012
[7] M Li C-L Yeh and S-Y Lu ldquoReal-Time QoE MonitoringSystem forVideo Streaming ServiceswithAdaptiveMedia Play-outrdquo International Journal of Digital Multimedia Broadcastingvol 2018 Article ID 2619438 11 pages 2018
[8] S Barakovic and L Skorin-Kapov ldquoSurvey and Challengesof QoE Management Issues in Wireless Networksrdquo Journal ofComputer Networks and Communications vol 2013 Article ID165146 28 pages 2013
[9] ITU-T ldquoG1030 estimating end-to-end performance in IPnetworks for data applicationsrdquo 2014
[10] ITU-T ldquoG1031 QoE factors in web-browsingrdquo 2014[11] EU H2020 TRIANGLE Project Deliverable D22 Final report
on the formalization of the certification process requirementsanduse cases 2017 httpswwwtriangle-projecteuproject-olddeliverables
[12] Q A Chen H Luo S Rosen et al ldquoQoE doctor diagnosingmobile app QoE with automated UI control and cross-layeranalysisrdquo in Proceedings of the Conference on Internet Mea-surement Conference (IMC rsquo14) pp 151ndash164 ACM VancouverCanada November 2014
[13] M A Mehmood A Wundsam S Uhlig D Levin N Sarrarand A Feldmann ldquoQoE-Lab Towards Evaluating Quality ofExperience for Future Internet Conditionsrdquo in Testbeds andResearch Infrastructure Korakis T Li H Tran-Gia P and HS Park Eds vol 90 of TridentCom 2011 Lnicst pp 286ndash301Springer Development of Networks and Communities BerlinGermany 2012
[14] D Levin A Wundsam A Mehmood and A FeldmannldquoBerlin The Berlin Experimental Router Laboratory for Inno-vative Networkingrdquo in TridentCom 2010 Lnicst T MagedanzA Gavras N H Thanh and J S Chase Eds vol 46 of LectureNotes of the Institute for Computer Sciences Social Informaticsand Telecommunications Engineering pp 602ndash604 SpringerHeidelberg Germany 2011
12 Wireless Communications and Mobile Computing
[15] K De Moor I Ketyko W Joseph et al ldquoProposed frameworkfor evaluating quality of experience in a mobile testbed-oriented living lab settingrdquo Mobile Networks and Applicationsvol 15 no 3 pp 378ndash391 2010
[16] R Sanchez-Iborra M-D Cano J J P C Rodrigues and JGarcia-Haro ldquoAnExperimental QoE Performance Study for theEfficient Transmission of High Demanding Traffic over an AdHoc Network Using BATMANrdquo Mobile Information Systemsvol 2015 Article ID 217106 14 pages 2015
[17] P Oliver-Balsalobre M Toril S Luna-Ramırez and R GarcıaGaraluz ldquoA system testbed for modeling encrypted video-streaming service performance indicators based on TCPIPmetricsrdquo EURASIP Journal on Wireless Communications andNetworking vol 2017 no 1 2017
[18] M Solera M Toril I Palomo G Gomez and J Poncela ldquoATestbed for Evaluating Video Streaming Services in LTErdquoWireless Personal Communications vol 98 no 3 pp 2753ndash27732018
[19] A Alvarez A Dıaz P Merino and F J Rivas ldquoField mea-surements of mobile services with Android smartphonesrdquoin Proceedings of the IEEE Consumer Communications andNetworking Conference (CCNC rsquo12) pp 105ndash109 Las Vegas NevUSA January 2012
[20] NGMN Alliance ldquoNGMN 5G white paperrdquo 2015 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2015NGMN 5G White Paper V1 0pdf
[21] ldquoInfrastructure and Design for Adaptivity and Flexibilityrdquo inMobile Information Systems Springer 2006
[22] J Nielsen ldquoResponse Times The Three Important Limitsrdquo inUsability Engineering 1993
[23] NGMN Alliance ldquoDefinition of the testing framework for theNGMN 5G pre-commercial networks trialsrdquo 2018 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2018180220 NGMN PreCommTrials Framework definition v1 0pdf
[24] 3GPP TS 26246 ldquoTransparent end-to-end Packet-switchedStreaming Services (PSS) Progressive Download and DynamicAdaptive Streaming over HTTP (3GP-DASH)rdquo 2018
International Journal of
AerospaceEngineeringHindawiwwwhindawicom Volume 2018
RoboticsJournal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Active and Passive Electronic Components
VLSI Design
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Shock and Vibration
Hindawiwwwhindawicom Volume 2018
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawiwwwhindawicom
Volume 2018
Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom
The Scientific World Journal
Volume 2018
Control Scienceand Engineering
Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom
Journal ofEngineeringVolume 2018
SensorsJournal of
Hindawiwwwhindawicom Volume 2018
International Journal of
RotatingMachinery
Hindawiwwwhindawicom Volume 2018
Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Navigation and Observation
International Journal of
Hindawi
wwwhindawicom Volume 2018
Advances in
Multimedia
Submit your manuscripts atwwwhindawicom
Wireless Communications and Mobile Computing 5
Synthetic MOS
Test case
Scenario 1
Iteration 1
Meas 1
Meas
Meas P
Iteration
Meas 1
Meas
Meas P
Iteration N
Meas 1
Meas
Meas P
Scenario K
Iteration 1
Meas 1
Meas
Meas P
Iteration
Meas 1
Meas
Meas P
Iteration N
Meas 1
Meas
Meas P
KPI 1 Synthetic MOS 1
KPI Synthetic MOS
KPI R Synthetic MOS R
Aggregation Synthetic MOS Scenario 1
KPI 1 Synthetic MOS 1
KPI Synthetic MOS
KPI R Synthetic MOS R
Aggregation Synthetic MOS Scenario K
Figure 2 The process to obtain the synthetic-MOS score in a TRIANGLE test case
Table 1 Uses cases defined in the TRIANGLE project
Identifier Use CaseVR Virtual RealityGA GamingAR Augmented RealityCS Content Distribution Streaming ServicesLS Live Streaming ServicesSN Social NetworkingHS High Speed InternetPM Patient MonitoringES Emergency ServicesSM Smart MeteringSG Smart GridsCV Connected Vehicles
phones (UEs) and the ones that can be integrated in forexample gaming consoles advanced VR gear car units orIoT systems
The TRIANGLE domains group different aspects thatcan affect the final QoE perceived by the users The cur-rent testbed implementation supports three of the severaldomains that have been identified Apps User Experience(AUE) Apps Energy consumption (AEC) and ApplicationsDevice Resources Usage (RES)
Table 1 provides the use cases and Table 2 lists thedomains initially considered in TRIANGLE
To produce data to evaluate the QoE a series of testcases have been designed developed and implemented tobe run on the TRIANGLE testbed Obviously not all testcases are applicable to all applications under test becausenot all applications need or are designed to support all thefunctionalities that can be tested in the testbed In orderto automatically determine the test cases that are applicableto an application under test a questionnaire (identified as
features questionnaire in the portal) equivalent to the classi-cal conformance testing ICS (Implementation ConformanceStatement) has been developed and is accessible through theportal After filling the questionnaire the applicable test planthat is the test campaign with the list of applicable test casesis automatically generated
The sequence of user actions (type swipe tap etc) a userneeds to perform in the terminal (UE) to complete a task (egplay a video) is called the ldquoapp user flowrdquo In order to be ableto automatically run a test case the actual application userflow with the user actions a user would need to perform onthe phone to complete certain tasks defined in the test casealso has to be provided
Each test case univocally defines the conditions of execu-tion the sequence of actions the user would perform (ie theapp user flow) the sequence of actions that the elements ofthe testbed must perform the traffic injected the collectionof measurements to take and so forth In order to obtainstatistical significance each test case includes a numberof executions (iterations) under certain network conditions(herein called scenarios) Out of the various measurementsmade in the different iterations under any specific networkconditions (scenario) a number of KPIs (Key PerformanceIndicators) are computed The KPIs are normalized into astandard 1-to-5 scale as typically used in MOS scores andreferred to as synthetic-MOS a terminology that has beenadopted from previous works [7 21] The synthetic-MOSvalues are aggregated across network scenarios to produce anumber of intermediate synthetic-MOS scores which finallyare aggregated to obtain a synthetic-MOS score in each testcase (see Figure 2)
The process to obtain the final TRIANGLE mark issequential First for each domain a weighted average ofthe synthetic-MOS scores obtained in each test case inthe domain is calculated Next a weighted average of thesynthetic-MOS values in all the domains of a use case iscalculated to provide a single synthetic-MOS value per use
6 Wireless Communications and Mobile Computing
Table 2 TRIANGLE domains
Category Identifier Domain
Applications
AUE Apps User experienceAEC Apps Energy consumptionRES Device Resources UsageREL ReliabilityNWR Network Resources
Devices
Mobile Devices
DEC Energy ConsumptionDDP Data PerformanceDRF Radio PerformanceDRA User experience with reference apps
IoT DevicesIDR ReliabilityIDP Data PerformanceIEC Energy consumption
Synthetic MOS Domain A Use Case X
Synthetic MOS Domain B Use Case X
Synthetic MOS Use Case X
Synthetic MOS Use Case Y
TRIANGLE MARK
App
Synthetic MOS Test CaseDomain AUse Case X01
Synthetic MOS Test CaseDomain AUse Case X02
Synthetic MOS Test CaseDomain BUse Case X01
Synthetic MOS Test CaseDomain BUse Case X02
Synthetic MOS Test CaseDomain AUse Case Y01 Synthetic MOS Domain A Use Case Y
Synthetic MOS Test CaseDomain BUse Case Y01 Synthetic MOS Domain B Use Case Y
Figure 3 The process to obtain the TRIANGLE mark
case An application will usually be developed for one specificuse case as those defined in Table 1 but may be designed formore than one use case In the latter case a further weightedaverage is made with the synthetic-MOS scores obtained ineach use case supported by the application These sequentialsteps produce a single TRIANGLE mark an overall qualityscore as shown in Figure 3
This approach provides a common framework for testingapplications for benchmarking applications or even forcertifying disparate applications The overall process for anapp that implements features of different use cases is depictedin Figure 3
5 Details of the TRIANGLE QoE Computation
For each use case identified (see Table 1) and domain (seeTable 2) a number of test cases have been developed withinthe TRIANGLE project Each test case intends to test an
individual feature aspect or behaviour of the applicationunder test as shown in Figure 4
Each test case defines a number of measurements andbecause the results of the measurements depend on manyfactors they are not in general deterministic and thuseach test case has been designed not to perform just onesingle measurement but to run a number of iterations (N)of the same measurement Out of those measurements KPIsare computed For example if the time to load the firstmedia frame is the measurement taken in one specific testcase the average user waiting time KPI can be calculated bycomputing the mean of the values across all iterations Ingeneral different use case-domain pairs have a different set ofKPIsThe reader is encouraged to read [11] for further detailsabout the terminology used in TRIANGLE
Recommendation P10G100 Amendment 1 Definition ofQuality of Experience [2] notes that the overall acceptabilitymay be influenced by user expectations and context Forthe definition of the context technical specifications ITU-T
Wireless Communications and Mobile Computing 7
Feature Non-interactiveplayback
bullTime to load firstmedia frame
bullPlayback Cut-off
bullVideo resolution
bullContent Stall
i) Uses cases
vi) Synthetic MOS iv) Context
Networkscenarios
App User Flow Measurements KPIs MOS
ii) Domains
Apps Energy Consumption
Apps Device Resources
Apps User Experience
Mea
sure
men
ts
v) Test case execution
iii)Test caseAUECS01
Figure 4 QoE computation steps
G1030 ldquoEstimating end-to-end performance in IP networksfor data applicationsrdquo [9] and ITU-T G1031 ldquoQoE factors inweb-browsingrdquo [10] have been considered in TRIANGLEIn particular ITU-T G1031 [10] identifies the following con-text influence factors location (cafeteria office and home)interactivity (high-level interactivity versus low-level inter-activity) task type (business entertainment etc) and taskurgency (urgent versus casual) Userrsquos influence factors arehowever outside of the scope of the ITU recommendation
In the TRIANGLE project the context information hasbeen captured in the networks scenarios defined (Urban -Internet Cafe Off Peak Suburban - Shopping Mall BusyHours Urban ndash Pedestrian Urban ndash Office High speed trainndash Relay etc) and in the test cases specified in [11]
The test cases specify the conditions of the test butalso a sequence of actions that have to be executed by theapplication (app user flows) to test its features For examplethe test case that tests the ldquoPlay and Pauserdquo functionalitydefines the app user flow shown in Figure 5
The transformation of KPIs into QoE scores is the mostchallenging step in the TRIANGLE framework The execu-tion of the test cases will generate a significant amount of rawmeasurements about several aspects of the system SpecificKPIs can then be extracted through statistical analysis meandeviation cumulative distribution function (CDF) or ratio
TheKPIs will be individually interpolated in order to pro-vide a common homogeneous comparison and aggregationspace The interpolation is based on the application of twofunctions named Type I and Type II By using the proposedtwo types of interpolations the vast majority of KPIs can betranslated into normalized MOS-type of metric (synthetic-MOS) easy to be averaged in order to provide a simpleunified evaluation
Type I This function performs a linear interpolation on theoriginal data The variables 119898119894119899
119870119875119868and119898119886119909
119870119875119868are the worst
and best known values of a KPI from a reference case The
Perform login step (ifrequired) and wait for
10 seconds
Start playing avideo of 5
minutes during10 seconds
Pause thereproduction
Resume thereproduction after
2 minutes anduntil the end of
the video
Figure 5 App user flow used in the ldquoAUECS02 Play and Pauserdquotest case
function maps a value v of a KPI to vrsquo (synthetic-MOS) inthe range [1-to-5] by computing the following formula
V1015840 =V minus 119898119894119899
119870119875119868
119898119886119909119870119875119868minus 119898119894119899
119870119875119868
(50 minus 10) + 10 (1)
This function transforms a KPI to a synthetic-MOS value byapplying a simple linear interpolation between the worst andbest expected values from a reference case If a future inputcase falls outside the data range of the KPI the new value will
8 Wireless Communications and Mobile Computing
Table 3 AUECS002 test case description
Identifier AUECS002 (App User ExperienceContent Streaming002)Title Play and pauseObjective Measure the ability of the AUT to pause and the resume a media fileApplicability (ICSG ProductType = Application) AND (ICSG UseCases includes CS) AND ICSA CSPauseInitial Conditions AUT in in [AUT STARTED] mode (Note Defined in D22 [11] Appendix 4)
Steps(1) The Test System commands the AUT to replay the Application User Flow (Application User Flow that
presses first the Play button and later the Pause button)(2) The Test System measures whether pause operation was successful or not
Postamble (i) Execute the Postamble sequence (see section 26 in D22 [11] Appendix 4)
Measurements (Raw)
(i) Playback Cut-off Probability that successfully started stream reproduction is ended by a cause other thanthe intentional termination by the user
(ii) Pause Operation Whether pause operation is successful or not(iii) Time to load first media frame (s) after resuming The time elapsed since the user clicks resume button
until the media reproduction starts(Note For Exoplayer the RESUME button is the PLAY button)
be set to the extreme value minKPI (if it is worse) or maxKPI(if it is better)
Type II This function performs a logarithmic interpolationand is inspired on the opinion model recommended by theITU-T in [9] for a simple web search taskThis function mapsa value v of a KPI to vrsquo (synthetic-MOS) in the range [1-to-5]by computing the following formula
V1015840 =50 minus 10
ln ((119886 lowast 119908119900119903119904119905119870119875119868+ 119887) 119908119900119903119904119905
119870119875119868)
∙ (ln (V) minus ln (119886 lowast 119908119900119903119904119905119870119875119868+ 119886)) + 5
(2)
The default values of 119886 and 119887 correspond to the simple websearch task case (119886 = 0003 and 119887 = 012) [9 22] and theworst value has been extracted from the ITU-T G1030 Ifduring experimentation a future input case falls outside thedata range of the KPI the parameters 119886 and 119887will be updatedaccordingly Likewise if through subjective experimentationother values are considered better adjustments for specificservices the function can be easily updated
Once all KPIs are translated into synthetic-MOS valuesthey can be averaged with suitable weights In the averagingprocess the first step is to average over the network scenariosconsidered relevant for the use case as shown in Figure 2This provides the synthetic-MOS output value for the testcase If there is more than one test case per domain which isgenerally the case a weighted average is calculated in order toprovide one synthetic-MOS value per domain as depicted inFigure 3The final step is to average the synthetic-MOS scoresover all use cases supported by the application (see Figure 3)This provides the final score that is the TRIANGLE mark
6 A Practical Case Exoplayer under Test
For better understanding the complete process of obtainingtheTRIANGLEmark for a specific application the Exoplayer
Table4Measurement points associatedwith test caseAUECS002
Measurements Measurement points
Time to load first media frame Media File Playback - StartMedia File Playback - First Picture
Playback cut-off Media File Playback - StartMedia File Playback - End
Pause Media File Playback - Pause
is described in this section This application only has one usecase content distribution streaming services (CS)
Exoplayer is an application levelmedia player forAndroidpromoted by Google It provides an alternative to AndroidrsquosMediaPlayer API for playing audio and video both locally andover the Internet Exoplayer supports features not currentlysupported by Androidrsquos MediaPlayer API including DASHand SmoothStreaming adaptive playbacks
The TRIANGLE project has concentrated in testing justtwo of the Exoplayer features ldquoNoninteractive Playbackrdquoand ldquoPlay and Pauserdquo These features result in 6 test casesapplicable out of the test cases defined in TRIANGLETheseare test cases AUECS001 and AUECS002 in the App UserExperience domain test casesAECCS001 andAECCS002in the App Energy Consumption domain and test casesRESCS001 and RESCS002 in the Device Resources Usagedomain
The AUECS002 ldquoPlay and Pauserdquo test case descriptionbelonging to the AUE domain is shown in Table 3 The testcase description specifies the test conditions the generic appuser flow and the rawmeasurements which shall be collectedduring the execution of the test
The TRIANGLE project also offers a library that includesthe measurement points that should be inserted in thesource code of the app for enabling the collection of themeasurements specified Table 4 shows the measurementpoints required to compute the measurements specified intest case AUECS002
Wireless Communications and Mobile Computing 9
Table 5 Reference values for interpolation
Feature Domain KPI Synthetic MOS Calculation KPI min KPI maxNon-Interactive Playback AEC Average power consumption Type I 10 W 08 WNon-Interactive Playback AUE Time to load first media frame Type II KPI worst=20 msNon-Interactive Playback AUE Playback cut-off ratio Type I 50 0Non-Interactive Playback AUE Video resolution Type I 240p 720pNon-Interactive Playback RES Average CPU usage Type I 100 16Non-Interactive Playback RES Average memory usage Type I 100 40Play and Pause AEC Average power consumption Type I 10 W 08 WPlay and Pause AUE Pause operation success rate Type I 50 100Play and Pause RES Average CPU usage Type I 100 16Play and Pause RES Average memory usage Type I 100 40
The time to load first media picture measurement isobtained subtracting the timestamp of the measurementpoint ldquoMedia File Playback ndash Startrdquo from the measurementpoint ldquoMedia File Playback ndash First Picturerdquo
As specified in [11] all scenarios defined are applicableto the content streaming use case Therefore test cases inthe three domains currently supported by the testbed areexecuted in all the scenarios
Once the test campaign has finished the raw measure-ment results are processed to obtain the KPIs associated witheach test case average current consumption average time toload first media frame average CPU usage and so forth Theprocesses applied are detailed in Table 5 Based on previousexperiments performed by the authors the behaviour of thetime to load the first media frame KPI resembles the webresponse time KPI (ie the amount of time the user hasto wait for the service) and thus as recommended in theopinionmodel forweb search introduced in [9] a logarithmicinterpolation (type II) has been used for this metric
The results of the initial process that is the KPIs compu-tation are translated into synthetics-MOS values To computethese values reference benchmarking values for each of theKPIs need to be used according to the normalization andinterpolation process described in Section 5 Table 5 showswhat has been currently used by TRIANGLE for the AppUser Experience domain which is also used by NGMN asreference in their precommercial Trials document [23]
For example for the ldquotime to load first media framerdquo KPIshown in Table 5 the type of aggregation applied is averagingand the interpolation formula used is Type II
To achieve stable results each test case is executed 10times (10 iterations) in each network scenario The synthetic-MOS value in each domain is calculated by averaging themeasured synthetic-MOS values in the domain For examplesynthetic-MOS value is the RES domain obtained by aver-aging the synthetic-MOS value of ldquoaverage CPU usagerdquo andldquoaverage memory usagerdquo from the two test cases
Although Exoplayer supports several video streamingprotocols in this work only DASH [24] (Dynamic AdaptiveStreaming over HTTP) has been tested DASH clients shouldseamlessly adapt to changing network conditions by makingdecisions on which video segment to download (videosare encoded at multiple bitrates) The Exoplayerrsquos default
000
000
0
001
728
0
003
456
0
005
184
0
010
912
0
012
640
0
014
368
0
020
096
0
Timestamp
Video Resolution
0
200
400
600
800
1000
1200
Hor
izon
tal R
esol
utio
n
Figure 6 Video Resolution evolution in the Driving Urban Normalscenario
adaptation algorithm is basically throughput-based and someparameters control how often and when switching can occur
During the testing the testbed was configured with thedifferent network scenarios defined in [11] In these scenariosthe network configuration changes dynamically following arandom pattern resulting in different maximum throughputrates The expected behaviour of the application under testis that the video streaming client adapts to the availablethroughput by decreasing or increasing the resolution of thereceived video Figure 6 depicts how the client effectivelyadapts to the channel conditions
However the objective of the testing carried out in theTRIANGE testbed is not just to verify that the video stream-ing client actually adapts to the available maximum through-put but also to check whether this adaptation improves theusersrsquo experience quality
Table 6 shows a summary of the synthetic-MOS valuesobtained per scenario in one test case of each domain Thescores obtained in the RES andAECdomains are always highIn the AUE domain the synthetic MOS associated with theVideo Resolution shows low scores in some of the scenariosbecause the resolution decreases reasonable good scores inthe time to load first media and high scores in the time toplayback cut-off ratio Overall it can be concluded that the
10 Wireless Communications and Mobile Computing
Table 6 Synthetic MOS values per test case and scenario for the feature ldquoNoninteractive Playbackrdquo
AUE domain AEC domain RES domain
Test Case AUECS001 Test CaseAECCS001 Test Case RESCS001
ScenarioTime to loadfirst mediaframe
PlaybackCut-off ratio
VideoResolution
mode
AveragePower
Consumption
Average CPUUsage
AverageRAM Usage
HighSpeed DirectPassenger 21 31 23 47 43 42
Suburban Festival 38 47 31 48 43 41Suburban shopping mallbusy hours 37 37 13 48 44 41
Suburban shopping malloff-peak 36 31 23 48 43 41
Suburban stadium 38 29 21 47 44 41Urban Driving Normal 26 39 28 47 44 4Urban Driving TrafficJam 34 37 16 48 44 4
Urban Internet Cafe BusyHours 38 37 19 48 44 4
Urban Internet Cafe OffPeak 38 31 23 48 43 4
Urban Office 38 47 33 48 45 43Urban Pedestrian 39 26 2 47 44 4
35 36 23 47 44 41
DASH implementation of the video streaming client undertest is able to adapt to the changing conditions of the networkmaintaining an acceptable rate of video cut-off rebufferingtimes and resources usage
The final score in each domain is obtained by averagingthe synthetic-MOS values from all the tested network scenar-ios Figure 7 shows the spider diagram for the three domainstested In the User Experience domain the score obtained islower than the other domains due to the low synthetic-MOSvalues obtained for the video resolution
The final synthetic MOS for the use case Content Dis-tribution Streaming is obtained as a weighted average of thethree domains representing the overall QoE as perceived bythe userThefinal score for the Exoplayer version 1516 and thefeatures tested (Noninteractive Playback and Play and Pause)is 42 which means that the low score obtained in the videoresolution is compensated with the high scores in other KPIs
If an application under test has more than one use casethe next steps in the TRIANGLE mark project approachwould be the aggregation per use case and the aggregationover all use cases The final score the TRIANGLE mark is anestimation of the overall QoE as perceived by the user
In the current TRIANGLE implementation the weightsin all aggregations are the same Further research is neededto appropriately define the weights of each domain and eachuse case in the overall score of the applications
7 Conclusions
The main contribution of the TRIANGLE project is theprovision of a framework that generalizes QoE computation
and enables the execution of extensive and repeatable testcampaigns to obtainmeaningfulQoE scoresTheTRIANGLEproject has also defined amethodology which is based on thetransformation and aggregation of KPIs its transformationinto synthetic-MOS values and its aggregation over thedifferent domains and use cases
The TRIANGLE approach is a methodology flexibleenough to generalize the computation of QoE for any applica-tionservice Themethodology has been validated testing theDASH implementation in the Exoplayer App To confirm thesuitability of theweights used in the averaging process and theinterpolation parameters as well as to verify the correlationof the obtained MOS with that scored by users the authorshave started experiments with real users and initial results areencouraging
The process described produces a final TRIANGLEmarka single quality score which could eventually be used to cer-tify applications after achieving a consensus on the differentvalues of the process (weights limits etc) to use
Data Availability
Themethodology and results used to support the findings ofthis study are included within the article
Conflicts of Interest
The authors declare that they have no conflicts of inter-est
Wireless Communications and Mobile Computing 11
DER
AUE AEC
Device Resource Usage
User Experience Energy Consumption
Figure 7 Exoplayer synthetic-MOS values per domain
Acknowledgments
The TRIANGLE project is funded by the European UnionrsquosHorizon 2020 Research and Innovation Programme (GrantAgreement no 688712)
References
[1] ETSI ldquoHuman factors quality of experience (QoE) require-ments for real-time communication servicesrdquo Tech Rep 102643 2010
[2] ITU-T ldquoP10G100 (2006) amendment 1 (0107) new appendixI - definition of quality of experience (QoE)rdquo 2007
[3] F Kozamernik V Steinmann P Sunna and E WyckensldquoSAMVIQ - A new EBUmethodology for video quality evalua-tions in multimediardquo SMPTE Motion Imaging Journal vol 114no 4 pp 152ndash160 2005
[4] ITU-T ldquoG107 the E-model a computational model for use intransmission planningrdquo 2015
[5] J De Vriendt D De Vleeschauwer and D C Robinson ldquoQoEmodel for video delivered over an LTE network using HTTPadaptive streamingrdquo Bell Labs Technical Journal vol 18 no 4pp 45ndash62 2014
[6] S Jelassi G Rubino H Melvin H Youssef and G PujolleldquoQuality of Experience of VoIP Service A Survey of AssessmentApproaches andOpen Issuesrdquo IEEECommunications Surveys ampTutorials vol 14 no 2 pp 491ndash513 2012
[7] M Li C-L Yeh and S-Y Lu ldquoReal-Time QoE MonitoringSystem forVideo Streaming ServiceswithAdaptiveMedia Play-outrdquo International Journal of Digital Multimedia Broadcastingvol 2018 Article ID 2619438 11 pages 2018
[8] S Barakovic and L Skorin-Kapov ldquoSurvey and Challengesof QoE Management Issues in Wireless Networksrdquo Journal ofComputer Networks and Communications vol 2013 Article ID165146 28 pages 2013
[9] ITU-T ldquoG1030 estimating end-to-end performance in IPnetworks for data applicationsrdquo 2014
[10] ITU-T ldquoG1031 QoE factors in web-browsingrdquo 2014[11] EU H2020 TRIANGLE Project Deliverable D22 Final report
on the formalization of the certification process requirementsanduse cases 2017 httpswwwtriangle-projecteuproject-olddeliverables
[12] Q A Chen H Luo S Rosen et al ldquoQoE doctor diagnosingmobile app QoE with automated UI control and cross-layeranalysisrdquo in Proceedings of the Conference on Internet Mea-surement Conference (IMC rsquo14) pp 151ndash164 ACM VancouverCanada November 2014
[13] M A Mehmood A Wundsam S Uhlig D Levin N Sarrarand A Feldmann ldquoQoE-Lab Towards Evaluating Quality ofExperience for Future Internet Conditionsrdquo in Testbeds andResearch Infrastructure Korakis T Li H Tran-Gia P and HS Park Eds vol 90 of TridentCom 2011 Lnicst pp 286ndash301Springer Development of Networks and Communities BerlinGermany 2012
[14] D Levin A Wundsam A Mehmood and A FeldmannldquoBerlin The Berlin Experimental Router Laboratory for Inno-vative Networkingrdquo in TridentCom 2010 Lnicst T MagedanzA Gavras N H Thanh and J S Chase Eds vol 46 of LectureNotes of the Institute for Computer Sciences Social Informaticsand Telecommunications Engineering pp 602ndash604 SpringerHeidelberg Germany 2011
12 Wireless Communications and Mobile Computing
[15] K De Moor I Ketyko W Joseph et al ldquoProposed frameworkfor evaluating quality of experience in a mobile testbed-oriented living lab settingrdquo Mobile Networks and Applicationsvol 15 no 3 pp 378ndash391 2010
[16] R Sanchez-Iborra M-D Cano J J P C Rodrigues and JGarcia-Haro ldquoAnExperimental QoE Performance Study for theEfficient Transmission of High Demanding Traffic over an AdHoc Network Using BATMANrdquo Mobile Information Systemsvol 2015 Article ID 217106 14 pages 2015
[17] P Oliver-Balsalobre M Toril S Luna-Ramırez and R GarcıaGaraluz ldquoA system testbed for modeling encrypted video-streaming service performance indicators based on TCPIPmetricsrdquo EURASIP Journal on Wireless Communications andNetworking vol 2017 no 1 2017
[18] M Solera M Toril I Palomo G Gomez and J Poncela ldquoATestbed for Evaluating Video Streaming Services in LTErdquoWireless Personal Communications vol 98 no 3 pp 2753ndash27732018
[19] A Alvarez A Dıaz P Merino and F J Rivas ldquoField mea-surements of mobile services with Android smartphonesrdquoin Proceedings of the IEEE Consumer Communications andNetworking Conference (CCNC rsquo12) pp 105ndash109 Las Vegas NevUSA January 2012
[20] NGMN Alliance ldquoNGMN 5G white paperrdquo 2015 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2015NGMN 5G White Paper V1 0pdf
[21] ldquoInfrastructure and Design for Adaptivity and Flexibilityrdquo inMobile Information Systems Springer 2006
[22] J Nielsen ldquoResponse Times The Three Important Limitsrdquo inUsability Engineering 1993
[23] NGMN Alliance ldquoDefinition of the testing framework for theNGMN 5G pre-commercial networks trialsrdquo 2018 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2018180220 NGMN PreCommTrials Framework definition v1 0pdf
[24] 3GPP TS 26246 ldquoTransparent end-to-end Packet-switchedStreaming Services (PSS) Progressive Download and DynamicAdaptive Streaming over HTTP (3GP-DASH)rdquo 2018
International Journal of
AerospaceEngineeringHindawiwwwhindawicom Volume 2018
RoboticsJournal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Active and Passive Electronic Components
VLSI Design
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Shock and Vibration
Hindawiwwwhindawicom Volume 2018
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawiwwwhindawicom
Volume 2018
Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom
The Scientific World Journal
Volume 2018
Control Scienceand Engineering
Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom
Journal ofEngineeringVolume 2018
SensorsJournal of
Hindawiwwwhindawicom Volume 2018
International Journal of
RotatingMachinery
Hindawiwwwhindawicom Volume 2018
Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Navigation and Observation
International Journal of
Hindawi
wwwhindawicom Volume 2018
Advances in
Multimedia
Submit your manuscripts atwwwhindawicom
6 Wireless Communications and Mobile Computing
Table 2 TRIANGLE domains
Category Identifier Domain
Applications
AUE Apps User experienceAEC Apps Energy consumptionRES Device Resources UsageREL ReliabilityNWR Network Resources
Devices
Mobile Devices
DEC Energy ConsumptionDDP Data PerformanceDRF Radio PerformanceDRA User experience with reference apps
IoT DevicesIDR ReliabilityIDP Data PerformanceIEC Energy consumption
Synthetic MOS Domain A Use Case X
Synthetic MOS Domain B Use Case X
Synthetic MOS Use Case X
Synthetic MOS Use Case Y
TRIANGLE MARK
App
Synthetic MOS Test CaseDomain AUse Case X01
Synthetic MOS Test CaseDomain AUse Case X02
Synthetic MOS Test CaseDomain BUse Case X01
Synthetic MOS Test CaseDomain BUse Case X02
Synthetic MOS Test CaseDomain AUse Case Y01 Synthetic MOS Domain A Use Case Y
Synthetic MOS Test CaseDomain BUse Case Y01 Synthetic MOS Domain B Use Case Y
Figure 3 The process to obtain the TRIANGLE mark
case An application will usually be developed for one specificuse case as those defined in Table 1 but may be designed formore than one use case In the latter case a further weightedaverage is made with the synthetic-MOS scores obtained ineach use case supported by the application These sequentialsteps produce a single TRIANGLE mark an overall qualityscore as shown in Figure 3
This approach provides a common framework for testingapplications for benchmarking applications or even forcertifying disparate applications The overall process for anapp that implements features of different use cases is depictedin Figure 3
5 Details of the TRIANGLE QoE Computation
For each use case identified (see Table 1) and domain (seeTable 2) a number of test cases have been developed withinthe TRIANGLE project Each test case intends to test an
individual feature aspect or behaviour of the applicationunder test as shown in Figure 4
Each test case defines a number of measurements andbecause the results of the measurements depend on manyfactors they are not in general deterministic and thuseach test case has been designed not to perform just onesingle measurement but to run a number of iterations (N)of the same measurement Out of those measurements KPIsare computed For example if the time to load the firstmedia frame is the measurement taken in one specific testcase the average user waiting time KPI can be calculated bycomputing the mean of the values across all iterations Ingeneral different use case-domain pairs have a different set ofKPIsThe reader is encouraged to read [11] for further detailsabout the terminology used in TRIANGLE
Recommendation P10G100 Amendment 1 Definition ofQuality of Experience [2] notes that the overall acceptabilitymay be influenced by user expectations and context Forthe definition of the context technical specifications ITU-T
Wireless Communications and Mobile Computing 7
Feature Non-interactiveplayback
bullTime to load firstmedia frame
bullPlayback Cut-off
bullVideo resolution
bullContent Stall
i) Uses cases
vi) Synthetic MOS iv) Context
Networkscenarios
App User Flow Measurements KPIs MOS
ii) Domains
Apps Energy Consumption
Apps Device Resources
Apps User Experience
Mea
sure
men
ts
v) Test case execution
iii)Test caseAUECS01
Figure 4 QoE computation steps
G1030 ldquoEstimating end-to-end performance in IP networksfor data applicationsrdquo [9] and ITU-T G1031 ldquoQoE factors inweb-browsingrdquo [10] have been considered in TRIANGLEIn particular ITU-T G1031 [10] identifies the following con-text influence factors location (cafeteria office and home)interactivity (high-level interactivity versus low-level inter-activity) task type (business entertainment etc) and taskurgency (urgent versus casual) Userrsquos influence factors arehowever outside of the scope of the ITU recommendation
In the TRIANGLE project the context information hasbeen captured in the networks scenarios defined (Urban -Internet Cafe Off Peak Suburban - Shopping Mall BusyHours Urban ndash Pedestrian Urban ndash Office High speed trainndash Relay etc) and in the test cases specified in [11]
The test cases specify the conditions of the test butalso a sequence of actions that have to be executed by theapplication (app user flows) to test its features For examplethe test case that tests the ldquoPlay and Pauserdquo functionalitydefines the app user flow shown in Figure 5
The transformation of KPIs into QoE scores is the mostchallenging step in the TRIANGLE framework The execu-tion of the test cases will generate a significant amount of rawmeasurements about several aspects of the system SpecificKPIs can then be extracted through statistical analysis meandeviation cumulative distribution function (CDF) or ratio
TheKPIs will be individually interpolated in order to pro-vide a common homogeneous comparison and aggregationspace The interpolation is based on the application of twofunctions named Type I and Type II By using the proposedtwo types of interpolations the vast majority of KPIs can betranslated into normalized MOS-type of metric (synthetic-MOS) easy to be averaged in order to provide a simpleunified evaluation
Type I This function performs a linear interpolation on theoriginal data The variables 119898119894119899
119870119875119868and119898119886119909
119870119875119868are the worst
and best known values of a KPI from a reference case The
Perform login step (ifrequired) and wait for
10 seconds
Start playing avideo of 5
minutes during10 seconds
Pause thereproduction
Resume thereproduction after
2 minutes anduntil the end of
the video
Figure 5 App user flow used in the ldquoAUECS02 Play and Pauserdquotest case
function maps a value v of a KPI to vrsquo (synthetic-MOS) inthe range [1-to-5] by computing the following formula
V1015840 =V minus 119898119894119899
119870119875119868
119898119886119909119870119875119868minus 119898119894119899
119870119875119868
(50 minus 10) + 10 (1)
This function transforms a KPI to a synthetic-MOS value byapplying a simple linear interpolation between the worst andbest expected values from a reference case If a future inputcase falls outside the data range of the KPI the new value will
8 Wireless Communications and Mobile Computing
Table 3 AUECS002 test case description
Identifier AUECS002 (App User ExperienceContent Streaming002)Title Play and pauseObjective Measure the ability of the AUT to pause and the resume a media fileApplicability (ICSG ProductType = Application) AND (ICSG UseCases includes CS) AND ICSA CSPauseInitial Conditions AUT in in [AUT STARTED] mode (Note Defined in D22 [11] Appendix 4)
Steps(1) The Test System commands the AUT to replay the Application User Flow (Application User Flow that
presses first the Play button and later the Pause button)(2) The Test System measures whether pause operation was successful or not
Postamble (i) Execute the Postamble sequence (see section 26 in D22 [11] Appendix 4)
Measurements (Raw)
(i) Playback Cut-off Probability that successfully started stream reproduction is ended by a cause other thanthe intentional termination by the user
(ii) Pause Operation Whether pause operation is successful or not(iii) Time to load first media frame (s) after resuming The time elapsed since the user clicks resume button
until the media reproduction starts(Note For Exoplayer the RESUME button is the PLAY button)
be set to the extreme value minKPI (if it is worse) or maxKPI(if it is better)
Type II This function performs a logarithmic interpolationand is inspired on the opinion model recommended by theITU-T in [9] for a simple web search taskThis function mapsa value v of a KPI to vrsquo (synthetic-MOS) in the range [1-to-5]by computing the following formula
V1015840 =50 minus 10
ln ((119886 lowast 119908119900119903119904119905119870119875119868+ 119887) 119908119900119903119904119905
119870119875119868)
∙ (ln (V) minus ln (119886 lowast 119908119900119903119904119905119870119875119868+ 119886)) + 5
(2)
The default values of 119886 and 119887 correspond to the simple websearch task case (119886 = 0003 and 119887 = 012) [9 22] and theworst value has been extracted from the ITU-T G1030 Ifduring experimentation a future input case falls outside thedata range of the KPI the parameters 119886 and 119887will be updatedaccordingly Likewise if through subjective experimentationother values are considered better adjustments for specificservices the function can be easily updated
Once all KPIs are translated into synthetic-MOS valuesthey can be averaged with suitable weights In the averagingprocess the first step is to average over the network scenariosconsidered relevant for the use case as shown in Figure 2This provides the synthetic-MOS output value for the testcase If there is more than one test case per domain which isgenerally the case a weighted average is calculated in order toprovide one synthetic-MOS value per domain as depicted inFigure 3The final step is to average the synthetic-MOS scoresover all use cases supported by the application (see Figure 3)This provides the final score that is the TRIANGLE mark
6 A Practical Case Exoplayer under Test
For better understanding the complete process of obtainingtheTRIANGLEmark for a specific application the Exoplayer
Table4Measurement points associatedwith test caseAUECS002
Measurements Measurement points
Time to load first media frame Media File Playback - StartMedia File Playback - First Picture
Playback cut-off Media File Playback - StartMedia File Playback - End
Pause Media File Playback - Pause
is described in this section This application only has one usecase content distribution streaming services (CS)
Exoplayer is an application levelmedia player forAndroidpromoted by Google It provides an alternative to AndroidrsquosMediaPlayer API for playing audio and video both locally andover the Internet Exoplayer supports features not currentlysupported by Androidrsquos MediaPlayer API including DASHand SmoothStreaming adaptive playbacks
The TRIANGLE project has concentrated in testing justtwo of the Exoplayer features ldquoNoninteractive Playbackrdquoand ldquoPlay and Pauserdquo These features result in 6 test casesapplicable out of the test cases defined in TRIANGLETheseare test cases AUECS001 and AUECS002 in the App UserExperience domain test casesAECCS001 andAECCS002in the App Energy Consumption domain and test casesRESCS001 and RESCS002 in the Device Resources Usagedomain
The AUECS002 ldquoPlay and Pauserdquo test case descriptionbelonging to the AUE domain is shown in Table 3 The testcase description specifies the test conditions the generic appuser flow and the rawmeasurements which shall be collectedduring the execution of the test
The TRIANGLE project also offers a library that includesthe measurement points that should be inserted in thesource code of the app for enabling the collection of themeasurements specified Table 4 shows the measurementpoints required to compute the measurements specified intest case AUECS002
Wireless Communications and Mobile Computing 9
Table 5 Reference values for interpolation
Feature Domain KPI Synthetic MOS Calculation KPI min KPI maxNon-Interactive Playback AEC Average power consumption Type I 10 W 08 WNon-Interactive Playback AUE Time to load first media frame Type II KPI worst=20 msNon-Interactive Playback AUE Playback cut-off ratio Type I 50 0Non-Interactive Playback AUE Video resolution Type I 240p 720pNon-Interactive Playback RES Average CPU usage Type I 100 16Non-Interactive Playback RES Average memory usage Type I 100 40Play and Pause AEC Average power consumption Type I 10 W 08 WPlay and Pause AUE Pause operation success rate Type I 50 100Play and Pause RES Average CPU usage Type I 100 16Play and Pause RES Average memory usage Type I 100 40
The time to load first media picture measurement isobtained subtracting the timestamp of the measurementpoint ldquoMedia File Playback ndash Startrdquo from the measurementpoint ldquoMedia File Playback ndash First Picturerdquo
As specified in [11] all scenarios defined are applicableto the content streaming use case Therefore test cases inthe three domains currently supported by the testbed areexecuted in all the scenarios
Once the test campaign has finished the raw measure-ment results are processed to obtain the KPIs associated witheach test case average current consumption average time toload first media frame average CPU usage and so forth Theprocesses applied are detailed in Table 5 Based on previousexperiments performed by the authors the behaviour of thetime to load the first media frame KPI resembles the webresponse time KPI (ie the amount of time the user hasto wait for the service) and thus as recommended in theopinionmodel forweb search introduced in [9] a logarithmicinterpolation (type II) has been used for this metric
The results of the initial process that is the KPIs compu-tation are translated into synthetics-MOS values To computethese values reference benchmarking values for each of theKPIs need to be used according to the normalization andinterpolation process described in Section 5 Table 5 showswhat has been currently used by TRIANGLE for the AppUser Experience domain which is also used by NGMN asreference in their precommercial Trials document [23]
For example for the ldquotime to load first media framerdquo KPIshown in Table 5 the type of aggregation applied is averagingand the interpolation formula used is Type II
To achieve stable results each test case is executed 10times (10 iterations) in each network scenario The synthetic-MOS value in each domain is calculated by averaging themeasured synthetic-MOS values in the domain For examplesynthetic-MOS value is the RES domain obtained by aver-aging the synthetic-MOS value of ldquoaverage CPU usagerdquo andldquoaverage memory usagerdquo from the two test cases
Although Exoplayer supports several video streamingprotocols in this work only DASH [24] (Dynamic AdaptiveStreaming over HTTP) has been tested DASH clients shouldseamlessly adapt to changing network conditions by makingdecisions on which video segment to download (videosare encoded at multiple bitrates) The Exoplayerrsquos default
000
000
0
001
728
0
003
456
0
005
184
0
010
912
0
012
640
0
014
368
0
020
096
0
Timestamp
Video Resolution
0
200
400
600
800
1000
1200
Hor
izon
tal R
esol
utio
n
Figure 6 Video Resolution evolution in the Driving Urban Normalscenario
adaptation algorithm is basically throughput-based and someparameters control how often and when switching can occur
During the testing the testbed was configured with thedifferent network scenarios defined in [11] In these scenariosthe network configuration changes dynamically following arandom pattern resulting in different maximum throughputrates The expected behaviour of the application under testis that the video streaming client adapts to the availablethroughput by decreasing or increasing the resolution of thereceived video Figure 6 depicts how the client effectivelyadapts to the channel conditions
However the objective of the testing carried out in theTRIANGE testbed is not just to verify that the video stream-ing client actually adapts to the available maximum through-put but also to check whether this adaptation improves theusersrsquo experience quality
Table 6 shows a summary of the synthetic-MOS valuesobtained per scenario in one test case of each domain Thescores obtained in the RES andAECdomains are always highIn the AUE domain the synthetic MOS associated with theVideo Resolution shows low scores in some of the scenariosbecause the resolution decreases reasonable good scores inthe time to load first media and high scores in the time toplayback cut-off ratio Overall it can be concluded that the
10 Wireless Communications and Mobile Computing
Table 6 Synthetic MOS values per test case and scenario for the feature ldquoNoninteractive Playbackrdquo
AUE domain AEC domain RES domain
Test Case AUECS001 Test CaseAECCS001 Test Case RESCS001
ScenarioTime to loadfirst mediaframe
PlaybackCut-off ratio
VideoResolution
mode
AveragePower
Consumption
Average CPUUsage
AverageRAM Usage
HighSpeed DirectPassenger 21 31 23 47 43 42
Suburban Festival 38 47 31 48 43 41Suburban shopping mallbusy hours 37 37 13 48 44 41
Suburban shopping malloff-peak 36 31 23 48 43 41
Suburban stadium 38 29 21 47 44 41Urban Driving Normal 26 39 28 47 44 4Urban Driving TrafficJam 34 37 16 48 44 4
Urban Internet Cafe BusyHours 38 37 19 48 44 4
Urban Internet Cafe OffPeak 38 31 23 48 43 4
Urban Office 38 47 33 48 45 43Urban Pedestrian 39 26 2 47 44 4
35 36 23 47 44 41
DASH implementation of the video streaming client undertest is able to adapt to the changing conditions of the networkmaintaining an acceptable rate of video cut-off rebufferingtimes and resources usage
The final score in each domain is obtained by averagingthe synthetic-MOS values from all the tested network scenar-ios Figure 7 shows the spider diagram for the three domainstested In the User Experience domain the score obtained islower than the other domains due to the low synthetic-MOSvalues obtained for the video resolution
The final synthetic MOS for the use case Content Dis-tribution Streaming is obtained as a weighted average of thethree domains representing the overall QoE as perceived bythe userThefinal score for the Exoplayer version 1516 and thefeatures tested (Noninteractive Playback and Play and Pause)is 42 which means that the low score obtained in the videoresolution is compensated with the high scores in other KPIs
If an application under test has more than one use casethe next steps in the TRIANGLE mark project approachwould be the aggregation per use case and the aggregationover all use cases The final score the TRIANGLE mark is anestimation of the overall QoE as perceived by the user
In the current TRIANGLE implementation the weightsin all aggregations are the same Further research is neededto appropriately define the weights of each domain and eachuse case in the overall score of the applications
7 Conclusions
The main contribution of the TRIANGLE project is theprovision of a framework that generalizes QoE computation
and enables the execution of extensive and repeatable testcampaigns to obtainmeaningfulQoE scoresTheTRIANGLEproject has also defined amethodology which is based on thetransformation and aggregation of KPIs its transformationinto synthetic-MOS values and its aggregation over thedifferent domains and use cases
The TRIANGLE approach is a methodology flexibleenough to generalize the computation of QoE for any applica-tionservice Themethodology has been validated testing theDASH implementation in the Exoplayer App To confirm thesuitability of theweights used in the averaging process and theinterpolation parameters as well as to verify the correlationof the obtained MOS with that scored by users the authorshave started experiments with real users and initial results areencouraging
The process described produces a final TRIANGLEmarka single quality score which could eventually be used to cer-tify applications after achieving a consensus on the differentvalues of the process (weights limits etc) to use
Data Availability
Themethodology and results used to support the findings ofthis study are included within the article
Conflicts of Interest
The authors declare that they have no conflicts of inter-est
Wireless Communications and Mobile Computing 11
DER
AUE AEC
Device Resource Usage
User Experience Energy Consumption
Figure 7 Exoplayer synthetic-MOS values per domain
Acknowledgments
The TRIANGLE project is funded by the European UnionrsquosHorizon 2020 Research and Innovation Programme (GrantAgreement no 688712)
References
[1] ETSI ldquoHuman factors quality of experience (QoE) require-ments for real-time communication servicesrdquo Tech Rep 102643 2010
[2] ITU-T ldquoP10G100 (2006) amendment 1 (0107) new appendixI - definition of quality of experience (QoE)rdquo 2007
[3] F Kozamernik V Steinmann P Sunna and E WyckensldquoSAMVIQ - A new EBUmethodology for video quality evalua-tions in multimediardquo SMPTE Motion Imaging Journal vol 114no 4 pp 152ndash160 2005
[4] ITU-T ldquoG107 the E-model a computational model for use intransmission planningrdquo 2015
[5] J De Vriendt D De Vleeschauwer and D C Robinson ldquoQoEmodel for video delivered over an LTE network using HTTPadaptive streamingrdquo Bell Labs Technical Journal vol 18 no 4pp 45ndash62 2014
[6] S Jelassi G Rubino H Melvin H Youssef and G PujolleldquoQuality of Experience of VoIP Service A Survey of AssessmentApproaches andOpen Issuesrdquo IEEECommunications Surveys ampTutorials vol 14 no 2 pp 491ndash513 2012
[7] M Li C-L Yeh and S-Y Lu ldquoReal-Time QoE MonitoringSystem forVideo Streaming ServiceswithAdaptiveMedia Play-outrdquo International Journal of Digital Multimedia Broadcastingvol 2018 Article ID 2619438 11 pages 2018
[8] S Barakovic and L Skorin-Kapov ldquoSurvey and Challengesof QoE Management Issues in Wireless Networksrdquo Journal ofComputer Networks and Communications vol 2013 Article ID165146 28 pages 2013
[9] ITU-T ldquoG1030 estimating end-to-end performance in IPnetworks for data applicationsrdquo 2014
[10] ITU-T ldquoG1031 QoE factors in web-browsingrdquo 2014[11] EU H2020 TRIANGLE Project Deliverable D22 Final report
on the formalization of the certification process requirementsanduse cases 2017 httpswwwtriangle-projecteuproject-olddeliverables
[12] Q A Chen H Luo S Rosen et al ldquoQoE doctor diagnosingmobile app QoE with automated UI control and cross-layeranalysisrdquo in Proceedings of the Conference on Internet Mea-surement Conference (IMC rsquo14) pp 151ndash164 ACM VancouverCanada November 2014
[13] M A Mehmood A Wundsam S Uhlig D Levin N Sarrarand A Feldmann ldquoQoE-Lab Towards Evaluating Quality ofExperience for Future Internet Conditionsrdquo in Testbeds andResearch Infrastructure Korakis T Li H Tran-Gia P and HS Park Eds vol 90 of TridentCom 2011 Lnicst pp 286ndash301Springer Development of Networks and Communities BerlinGermany 2012
[14] D Levin A Wundsam A Mehmood and A FeldmannldquoBerlin The Berlin Experimental Router Laboratory for Inno-vative Networkingrdquo in TridentCom 2010 Lnicst T MagedanzA Gavras N H Thanh and J S Chase Eds vol 46 of LectureNotes of the Institute for Computer Sciences Social Informaticsand Telecommunications Engineering pp 602ndash604 SpringerHeidelberg Germany 2011
12 Wireless Communications and Mobile Computing
[15] K De Moor I Ketyko W Joseph et al ldquoProposed frameworkfor evaluating quality of experience in a mobile testbed-oriented living lab settingrdquo Mobile Networks and Applicationsvol 15 no 3 pp 378ndash391 2010
[16] R Sanchez-Iborra M-D Cano J J P C Rodrigues and JGarcia-Haro ldquoAnExperimental QoE Performance Study for theEfficient Transmission of High Demanding Traffic over an AdHoc Network Using BATMANrdquo Mobile Information Systemsvol 2015 Article ID 217106 14 pages 2015
[17] P Oliver-Balsalobre M Toril S Luna-Ramırez and R GarcıaGaraluz ldquoA system testbed for modeling encrypted video-streaming service performance indicators based on TCPIPmetricsrdquo EURASIP Journal on Wireless Communications andNetworking vol 2017 no 1 2017
[18] M Solera M Toril I Palomo G Gomez and J Poncela ldquoATestbed for Evaluating Video Streaming Services in LTErdquoWireless Personal Communications vol 98 no 3 pp 2753ndash27732018
[19] A Alvarez A Dıaz P Merino and F J Rivas ldquoField mea-surements of mobile services with Android smartphonesrdquoin Proceedings of the IEEE Consumer Communications andNetworking Conference (CCNC rsquo12) pp 105ndash109 Las Vegas NevUSA January 2012
[20] NGMN Alliance ldquoNGMN 5G white paperrdquo 2015 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2015NGMN 5G White Paper V1 0pdf
[21] ldquoInfrastructure and Design for Adaptivity and Flexibilityrdquo inMobile Information Systems Springer 2006
[22] J Nielsen ldquoResponse Times The Three Important Limitsrdquo inUsability Engineering 1993
[23] NGMN Alliance ldquoDefinition of the testing framework for theNGMN 5G pre-commercial networks trialsrdquo 2018 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2018180220 NGMN PreCommTrials Framework definition v1 0pdf
[24] 3GPP TS 26246 ldquoTransparent end-to-end Packet-switchedStreaming Services (PSS) Progressive Download and DynamicAdaptive Streaming over HTTP (3GP-DASH)rdquo 2018
International Journal of
AerospaceEngineeringHindawiwwwhindawicom Volume 2018
RoboticsJournal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Active and Passive Electronic Components
VLSI Design
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Shock and Vibration
Hindawiwwwhindawicom Volume 2018
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawiwwwhindawicom
Volume 2018
Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom
The Scientific World Journal
Volume 2018
Control Scienceand Engineering
Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom
Journal ofEngineeringVolume 2018
SensorsJournal of
Hindawiwwwhindawicom Volume 2018
International Journal of
RotatingMachinery
Hindawiwwwhindawicom Volume 2018
Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Navigation and Observation
International Journal of
Hindawi
wwwhindawicom Volume 2018
Advances in
Multimedia
Submit your manuscripts atwwwhindawicom
Wireless Communications and Mobile Computing 7
Feature Non-interactiveplayback
bullTime to load firstmedia frame
bullPlayback Cut-off
bullVideo resolution
bullContent Stall
i) Uses cases
vi) Synthetic MOS iv) Context
Networkscenarios
App User Flow Measurements KPIs MOS
ii) Domains
Apps Energy Consumption
Apps Device Resources
Apps User Experience
Mea
sure
men
ts
v) Test case execution
iii)Test caseAUECS01
Figure 4 QoE computation steps
G1030 ldquoEstimating end-to-end performance in IP networksfor data applicationsrdquo [9] and ITU-T G1031 ldquoQoE factors inweb-browsingrdquo [10] have been considered in TRIANGLEIn particular ITU-T G1031 [10] identifies the following con-text influence factors location (cafeteria office and home)interactivity (high-level interactivity versus low-level inter-activity) task type (business entertainment etc) and taskurgency (urgent versus casual) Userrsquos influence factors arehowever outside of the scope of the ITU recommendation
In the TRIANGLE project the context information hasbeen captured in the networks scenarios defined (Urban -Internet Cafe Off Peak Suburban - Shopping Mall BusyHours Urban ndash Pedestrian Urban ndash Office High speed trainndash Relay etc) and in the test cases specified in [11]
The test cases specify the conditions of the test butalso a sequence of actions that have to be executed by theapplication (app user flows) to test its features For examplethe test case that tests the ldquoPlay and Pauserdquo functionalitydefines the app user flow shown in Figure 5
The transformation of KPIs into QoE scores is the mostchallenging step in the TRIANGLE framework The execu-tion of the test cases will generate a significant amount of rawmeasurements about several aspects of the system SpecificKPIs can then be extracted through statistical analysis meandeviation cumulative distribution function (CDF) or ratio
TheKPIs will be individually interpolated in order to pro-vide a common homogeneous comparison and aggregationspace The interpolation is based on the application of twofunctions named Type I and Type II By using the proposedtwo types of interpolations the vast majority of KPIs can betranslated into normalized MOS-type of metric (synthetic-MOS) easy to be averaged in order to provide a simpleunified evaluation
Type I This function performs a linear interpolation on theoriginal data The variables 119898119894119899
119870119875119868and119898119886119909
119870119875119868are the worst
and best known values of a KPI from a reference case The
Perform login step (ifrequired) and wait for
10 seconds
Start playing avideo of 5
minutes during10 seconds
Pause thereproduction
Resume thereproduction after
2 minutes anduntil the end of
the video
Figure 5 App user flow used in the ldquoAUECS02 Play and Pauserdquotest case
function maps a value v of a KPI to vrsquo (synthetic-MOS) inthe range [1-to-5] by computing the following formula
V1015840 =V minus 119898119894119899
119870119875119868
119898119886119909119870119875119868minus 119898119894119899
119870119875119868
(50 minus 10) + 10 (1)
This function transforms a KPI to a synthetic-MOS value byapplying a simple linear interpolation between the worst andbest expected values from a reference case If a future inputcase falls outside the data range of the KPI the new value will
8 Wireless Communications and Mobile Computing
Table 3 AUECS002 test case description
Identifier AUECS002 (App User ExperienceContent Streaming002)Title Play and pauseObjective Measure the ability of the AUT to pause and the resume a media fileApplicability (ICSG ProductType = Application) AND (ICSG UseCases includes CS) AND ICSA CSPauseInitial Conditions AUT in in [AUT STARTED] mode (Note Defined in D22 [11] Appendix 4)
Steps(1) The Test System commands the AUT to replay the Application User Flow (Application User Flow that
presses first the Play button and later the Pause button)(2) The Test System measures whether pause operation was successful or not
Postamble (i) Execute the Postamble sequence (see section 26 in D22 [11] Appendix 4)
Measurements (Raw)
(i) Playback Cut-off Probability that successfully started stream reproduction is ended by a cause other thanthe intentional termination by the user
(ii) Pause Operation Whether pause operation is successful or not(iii) Time to load first media frame (s) after resuming The time elapsed since the user clicks resume button
until the media reproduction starts(Note For Exoplayer the RESUME button is the PLAY button)
be set to the extreme value minKPI (if it is worse) or maxKPI(if it is better)
Type II This function performs a logarithmic interpolationand is inspired on the opinion model recommended by theITU-T in [9] for a simple web search taskThis function mapsa value v of a KPI to vrsquo (synthetic-MOS) in the range [1-to-5]by computing the following formula
V1015840 =50 minus 10
ln ((119886 lowast 119908119900119903119904119905119870119875119868+ 119887) 119908119900119903119904119905
119870119875119868)
∙ (ln (V) minus ln (119886 lowast 119908119900119903119904119905119870119875119868+ 119886)) + 5
(2)
The default values of 119886 and 119887 correspond to the simple websearch task case (119886 = 0003 and 119887 = 012) [9 22] and theworst value has been extracted from the ITU-T G1030 Ifduring experimentation a future input case falls outside thedata range of the KPI the parameters 119886 and 119887will be updatedaccordingly Likewise if through subjective experimentationother values are considered better adjustments for specificservices the function can be easily updated
Once all KPIs are translated into synthetic-MOS valuesthey can be averaged with suitable weights In the averagingprocess the first step is to average over the network scenariosconsidered relevant for the use case as shown in Figure 2This provides the synthetic-MOS output value for the testcase If there is more than one test case per domain which isgenerally the case a weighted average is calculated in order toprovide one synthetic-MOS value per domain as depicted inFigure 3The final step is to average the synthetic-MOS scoresover all use cases supported by the application (see Figure 3)This provides the final score that is the TRIANGLE mark
6 A Practical Case Exoplayer under Test
For better understanding the complete process of obtainingtheTRIANGLEmark for a specific application the Exoplayer
Table4Measurement points associatedwith test caseAUECS002
Measurements Measurement points
Time to load first media frame Media File Playback - StartMedia File Playback - First Picture
Playback cut-off Media File Playback - StartMedia File Playback - End
Pause Media File Playback - Pause
is described in this section This application only has one usecase content distribution streaming services (CS)
Exoplayer is an application levelmedia player forAndroidpromoted by Google It provides an alternative to AndroidrsquosMediaPlayer API for playing audio and video both locally andover the Internet Exoplayer supports features not currentlysupported by Androidrsquos MediaPlayer API including DASHand SmoothStreaming adaptive playbacks
The TRIANGLE project has concentrated in testing justtwo of the Exoplayer features ldquoNoninteractive Playbackrdquoand ldquoPlay and Pauserdquo These features result in 6 test casesapplicable out of the test cases defined in TRIANGLETheseare test cases AUECS001 and AUECS002 in the App UserExperience domain test casesAECCS001 andAECCS002in the App Energy Consumption domain and test casesRESCS001 and RESCS002 in the Device Resources Usagedomain
The AUECS002 ldquoPlay and Pauserdquo test case descriptionbelonging to the AUE domain is shown in Table 3 The testcase description specifies the test conditions the generic appuser flow and the rawmeasurements which shall be collectedduring the execution of the test
The TRIANGLE project also offers a library that includesthe measurement points that should be inserted in thesource code of the app for enabling the collection of themeasurements specified Table 4 shows the measurementpoints required to compute the measurements specified intest case AUECS002
Wireless Communications and Mobile Computing 9
Table 5 Reference values for interpolation
Feature Domain KPI Synthetic MOS Calculation KPI min KPI maxNon-Interactive Playback AEC Average power consumption Type I 10 W 08 WNon-Interactive Playback AUE Time to load first media frame Type II KPI worst=20 msNon-Interactive Playback AUE Playback cut-off ratio Type I 50 0Non-Interactive Playback AUE Video resolution Type I 240p 720pNon-Interactive Playback RES Average CPU usage Type I 100 16Non-Interactive Playback RES Average memory usage Type I 100 40Play and Pause AEC Average power consumption Type I 10 W 08 WPlay and Pause AUE Pause operation success rate Type I 50 100Play and Pause RES Average CPU usage Type I 100 16Play and Pause RES Average memory usage Type I 100 40
The time to load first media picture measurement isobtained subtracting the timestamp of the measurementpoint ldquoMedia File Playback ndash Startrdquo from the measurementpoint ldquoMedia File Playback ndash First Picturerdquo
As specified in [11] all scenarios defined are applicableto the content streaming use case Therefore test cases inthe three domains currently supported by the testbed areexecuted in all the scenarios
Once the test campaign has finished the raw measure-ment results are processed to obtain the KPIs associated witheach test case average current consumption average time toload first media frame average CPU usage and so forth Theprocesses applied are detailed in Table 5 Based on previousexperiments performed by the authors the behaviour of thetime to load the first media frame KPI resembles the webresponse time KPI (ie the amount of time the user hasto wait for the service) and thus as recommended in theopinionmodel forweb search introduced in [9] a logarithmicinterpolation (type II) has been used for this metric
The results of the initial process that is the KPIs compu-tation are translated into synthetics-MOS values To computethese values reference benchmarking values for each of theKPIs need to be used according to the normalization andinterpolation process described in Section 5 Table 5 showswhat has been currently used by TRIANGLE for the AppUser Experience domain which is also used by NGMN asreference in their precommercial Trials document [23]
For example for the ldquotime to load first media framerdquo KPIshown in Table 5 the type of aggregation applied is averagingand the interpolation formula used is Type II
To achieve stable results each test case is executed 10times (10 iterations) in each network scenario The synthetic-MOS value in each domain is calculated by averaging themeasured synthetic-MOS values in the domain For examplesynthetic-MOS value is the RES domain obtained by aver-aging the synthetic-MOS value of ldquoaverage CPU usagerdquo andldquoaverage memory usagerdquo from the two test cases
Although Exoplayer supports several video streamingprotocols in this work only DASH [24] (Dynamic AdaptiveStreaming over HTTP) has been tested DASH clients shouldseamlessly adapt to changing network conditions by makingdecisions on which video segment to download (videosare encoded at multiple bitrates) The Exoplayerrsquos default
000
000
0
001
728
0
003
456
0
005
184
0
010
912
0
012
640
0
014
368
0
020
096
0
Timestamp
Video Resolution
0
200
400
600
800
1000
1200
Hor
izon
tal R
esol
utio
n
Figure 6 Video Resolution evolution in the Driving Urban Normalscenario
adaptation algorithm is basically throughput-based and someparameters control how often and when switching can occur
During the testing the testbed was configured with thedifferent network scenarios defined in [11] In these scenariosthe network configuration changes dynamically following arandom pattern resulting in different maximum throughputrates The expected behaviour of the application under testis that the video streaming client adapts to the availablethroughput by decreasing or increasing the resolution of thereceived video Figure 6 depicts how the client effectivelyadapts to the channel conditions
However the objective of the testing carried out in theTRIANGE testbed is not just to verify that the video stream-ing client actually adapts to the available maximum through-put but also to check whether this adaptation improves theusersrsquo experience quality
Table 6 shows a summary of the synthetic-MOS valuesobtained per scenario in one test case of each domain Thescores obtained in the RES andAECdomains are always highIn the AUE domain the synthetic MOS associated with theVideo Resolution shows low scores in some of the scenariosbecause the resolution decreases reasonable good scores inthe time to load first media and high scores in the time toplayback cut-off ratio Overall it can be concluded that the
10 Wireless Communications and Mobile Computing
Table 6 Synthetic MOS values per test case and scenario for the feature ldquoNoninteractive Playbackrdquo
AUE domain AEC domain RES domain
Test Case AUECS001 Test CaseAECCS001 Test Case RESCS001
ScenarioTime to loadfirst mediaframe
PlaybackCut-off ratio
VideoResolution
mode
AveragePower
Consumption
Average CPUUsage
AverageRAM Usage
HighSpeed DirectPassenger 21 31 23 47 43 42
Suburban Festival 38 47 31 48 43 41Suburban shopping mallbusy hours 37 37 13 48 44 41
Suburban shopping malloff-peak 36 31 23 48 43 41
Suburban stadium 38 29 21 47 44 41Urban Driving Normal 26 39 28 47 44 4Urban Driving TrafficJam 34 37 16 48 44 4
Urban Internet Cafe BusyHours 38 37 19 48 44 4
Urban Internet Cafe OffPeak 38 31 23 48 43 4
Urban Office 38 47 33 48 45 43Urban Pedestrian 39 26 2 47 44 4
35 36 23 47 44 41
DASH implementation of the video streaming client undertest is able to adapt to the changing conditions of the networkmaintaining an acceptable rate of video cut-off rebufferingtimes and resources usage
The final score in each domain is obtained by averagingthe synthetic-MOS values from all the tested network scenar-ios Figure 7 shows the spider diagram for the three domainstested In the User Experience domain the score obtained islower than the other domains due to the low synthetic-MOSvalues obtained for the video resolution
The final synthetic MOS for the use case Content Dis-tribution Streaming is obtained as a weighted average of thethree domains representing the overall QoE as perceived bythe userThefinal score for the Exoplayer version 1516 and thefeatures tested (Noninteractive Playback and Play and Pause)is 42 which means that the low score obtained in the videoresolution is compensated with the high scores in other KPIs
If an application under test has more than one use casethe next steps in the TRIANGLE mark project approachwould be the aggregation per use case and the aggregationover all use cases The final score the TRIANGLE mark is anestimation of the overall QoE as perceived by the user
In the current TRIANGLE implementation the weightsin all aggregations are the same Further research is neededto appropriately define the weights of each domain and eachuse case in the overall score of the applications
7 Conclusions
The main contribution of the TRIANGLE project is theprovision of a framework that generalizes QoE computation
and enables the execution of extensive and repeatable testcampaigns to obtainmeaningfulQoE scoresTheTRIANGLEproject has also defined amethodology which is based on thetransformation and aggregation of KPIs its transformationinto synthetic-MOS values and its aggregation over thedifferent domains and use cases
The TRIANGLE approach is a methodology flexibleenough to generalize the computation of QoE for any applica-tionservice Themethodology has been validated testing theDASH implementation in the Exoplayer App To confirm thesuitability of theweights used in the averaging process and theinterpolation parameters as well as to verify the correlationof the obtained MOS with that scored by users the authorshave started experiments with real users and initial results areencouraging
The process described produces a final TRIANGLEmarka single quality score which could eventually be used to cer-tify applications after achieving a consensus on the differentvalues of the process (weights limits etc) to use
Data Availability
Themethodology and results used to support the findings ofthis study are included within the article
Conflicts of Interest
The authors declare that they have no conflicts of inter-est
Wireless Communications and Mobile Computing 11
DER
AUE AEC
Device Resource Usage
User Experience Energy Consumption
Figure 7 Exoplayer synthetic-MOS values per domain
Acknowledgments
The TRIANGLE project is funded by the European UnionrsquosHorizon 2020 Research and Innovation Programme (GrantAgreement no 688712)
References
[1] ETSI ldquoHuman factors quality of experience (QoE) require-ments for real-time communication servicesrdquo Tech Rep 102643 2010
[2] ITU-T ldquoP10G100 (2006) amendment 1 (0107) new appendixI - definition of quality of experience (QoE)rdquo 2007
[3] F Kozamernik V Steinmann P Sunna and E WyckensldquoSAMVIQ - A new EBUmethodology for video quality evalua-tions in multimediardquo SMPTE Motion Imaging Journal vol 114no 4 pp 152ndash160 2005
[4] ITU-T ldquoG107 the E-model a computational model for use intransmission planningrdquo 2015
[5] J De Vriendt D De Vleeschauwer and D C Robinson ldquoQoEmodel for video delivered over an LTE network using HTTPadaptive streamingrdquo Bell Labs Technical Journal vol 18 no 4pp 45ndash62 2014
[6] S Jelassi G Rubino H Melvin H Youssef and G PujolleldquoQuality of Experience of VoIP Service A Survey of AssessmentApproaches andOpen Issuesrdquo IEEECommunications Surveys ampTutorials vol 14 no 2 pp 491ndash513 2012
[7] M Li C-L Yeh and S-Y Lu ldquoReal-Time QoE MonitoringSystem forVideo Streaming ServiceswithAdaptiveMedia Play-outrdquo International Journal of Digital Multimedia Broadcastingvol 2018 Article ID 2619438 11 pages 2018
[8] S Barakovic and L Skorin-Kapov ldquoSurvey and Challengesof QoE Management Issues in Wireless Networksrdquo Journal ofComputer Networks and Communications vol 2013 Article ID165146 28 pages 2013
[9] ITU-T ldquoG1030 estimating end-to-end performance in IPnetworks for data applicationsrdquo 2014
[10] ITU-T ldquoG1031 QoE factors in web-browsingrdquo 2014[11] EU H2020 TRIANGLE Project Deliverable D22 Final report
on the formalization of the certification process requirementsanduse cases 2017 httpswwwtriangle-projecteuproject-olddeliverables
[12] Q A Chen H Luo S Rosen et al ldquoQoE doctor diagnosingmobile app QoE with automated UI control and cross-layeranalysisrdquo in Proceedings of the Conference on Internet Mea-surement Conference (IMC rsquo14) pp 151ndash164 ACM VancouverCanada November 2014
[13] M A Mehmood A Wundsam S Uhlig D Levin N Sarrarand A Feldmann ldquoQoE-Lab Towards Evaluating Quality ofExperience for Future Internet Conditionsrdquo in Testbeds andResearch Infrastructure Korakis T Li H Tran-Gia P and HS Park Eds vol 90 of TridentCom 2011 Lnicst pp 286ndash301Springer Development of Networks and Communities BerlinGermany 2012
[14] D Levin A Wundsam A Mehmood and A FeldmannldquoBerlin The Berlin Experimental Router Laboratory for Inno-vative Networkingrdquo in TridentCom 2010 Lnicst T MagedanzA Gavras N H Thanh and J S Chase Eds vol 46 of LectureNotes of the Institute for Computer Sciences Social Informaticsand Telecommunications Engineering pp 602ndash604 SpringerHeidelberg Germany 2011
12 Wireless Communications and Mobile Computing
[15] K De Moor I Ketyko W Joseph et al ldquoProposed frameworkfor evaluating quality of experience in a mobile testbed-oriented living lab settingrdquo Mobile Networks and Applicationsvol 15 no 3 pp 378ndash391 2010
[16] R Sanchez-Iborra M-D Cano J J P C Rodrigues and JGarcia-Haro ldquoAnExperimental QoE Performance Study for theEfficient Transmission of High Demanding Traffic over an AdHoc Network Using BATMANrdquo Mobile Information Systemsvol 2015 Article ID 217106 14 pages 2015
[17] P Oliver-Balsalobre M Toril S Luna-Ramırez and R GarcıaGaraluz ldquoA system testbed for modeling encrypted video-streaming service performance indicators based on TCPIPmetricsrdquo EURASIP Journal on Wireless Communications andNetworking vol 2017 no 1 2017
[18] M Solera M Toril I Palomo G Gomez and J Poncela ldquoATestbed for Evaluating Video Streaming Services in LTErdquoWireless Personal Communications vol 98 no 3 pp 2753ndash27732018
[19] A Alvarez A Dıaz P Merino and F J Rivas ldquoField mea-surements of mobile services with Android smartphonesrdquoin Proceedings of the IEEE Consumer Communications andNetworking Conference (CCNC rsquo12) pp 105ndash109 Las Vegas NevUSA January 2012
[20] NGMN Alliance ldquoNGMN 5G white paperrdquo 2015 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2015NGMN 5G White Paper V1 0pdf
[21] ldquoInfrastructure and Design for Adaptivity and Flexibilityrdquo inMobile Information Systems Springer 2006
[22] J Nielsen ldquoResponse Times The Three Important Limitsrdquo inUsability Engineering 1993
[23] NGMN Alliance ldquoDefinition of the testing framework for theNGMN 5G pre-commercial networks trialsrdquo 2018 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2018180220 NGMN PreCommTrials Framework definition v1 0pdf
[24] 3GPP TS 26246 ldquoTransparent end-to-end Packet-switchedStreaming Services (PSS) Progressive Download and DynamicAdaptive Streaming over HTTP (3GP-DASH)rdquo 2018
International Journal of
AerospaceEngineeringHindawiwwwhindawicom Volume 2018
RoboticsJournal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Active and Passive Electronic Components
VLSI Design
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Shock and Vibration
Hindawiwwwhindawicom Volume 2018
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawiwwwhindawicom
Volume 2018
Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom
The Scientific World Journal
Volume 2018
Control Scienceand Engineering
Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom
Journal ofEngineeringVolume 2018
SensorsJournal of
Hindawiwwwhindawicom Volume 2018
International Journal of
RotatingMachinery
Hindawiwwwhindawicom Volume 2018
Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Navigation and Observation
International Journal of
Hindawi
wwwhindawicom Volume 2018
Advances in
Multimedia
Submit your manuscripts atwwwhindawicom
8 Wireless Communications and Mobile Computing
Table 3 AUECS002 test case description
Identifier AUECS002 (App User ExperienceContent Streaming002)Title Play and pauseObjective Measure the ability of the AUT to pause and the resume a media fileApplicability (ICSG ProductType = Application) AND (ICSG UseCases includes CS) AND ICSA CSPauseInitial Conditions AUT in in [AUT STARTED] mode (Note Defined in D22 [11] Appendix 4)
Steps(1) The Test System commands the AUT to replay the Application User Flow (Application User Flow that
presses first the Play button and later the Pause button)(2) The Test System measures whether pause operation was successful or not
Postamble (i) Execute the Postamble sequence (see section 26 in D22 [11] Appendix 4)
Measurements (Raw)
(i) Playback Cut-off Probability that successfully started stream reproduction is ended by a cause other thanthe intentional termination by the user
(ii) Pause Operation Whether pause operation is successful or not(iii) Time to load first media frame (s) after resuming The time elapsed since the user clicks resume button
until the media reproduction starts(Note For Exoplayer the RESUME button is the PLAY button)
be set to the extreme value minKPI (if it is worse) or maxKPI(if it is better)
Type II This function performs a logarithmic interpolationand is inspired on the opinion model recommended by theITU-T in [9] for a simple web search taskThis function mapsa value v of a KPI to vrsquo (synthetic-MOS) in the range [1-to-5]by computing the following formula
V1015840 =50 minus 10
ln ((119886 lowast 119908119900119903119904119905119870119875119868+ 119887) 119908119900119903119904119905
119870119875119868)
∙ (ln (V) minus ln (119886 lowast 119908119900119903119904119905119870119875119868+ 119886)) + 5
(2)
The default values of 119886 and 119887 correspond to the simple websearch task case (119886 = 0003 and 119887 = 012) [9 22] and theworst value has been extracted from the ITU-T G1030 Ifduring experimentation a future input case falls outside thedata range of the KPI the parameters 119886 and 119887will be updatedaccordingly Likewise if through subjective experimentationother values are considered better adjustments for specificservices the function can be easily updated
Once all KPIs are translated into synthetic-MOS valuesthey can be averaged with suitable weights In the averagingprocess the first step is to average over the network scenariosconsidered relevant for the use case as shown in Figure 2This provides the synthetic-MOS output value for the testcase If there is more than one test case per domain which isgenerally the case a weighted average is calculated in order toprovide one synthetic-MOS value per domain as depicted inFigure 3The final step is to average the synthetic-MOS scoresover all use cases supported by the application (see Figure 3)This provides the final score that is the TRIANGLE mark
6 A Practical Case Exoplayer under Test
For better understanding the complete process of obtainingtheTRIANGLEmark for a specific application the Exoplayer
Table4Measurement points associatedwith test caseAUECS002
Measurements Measurement points
Time to load first media frame Media File Playback - StartMedia File Playback - First Picture
Playback cut-off Media File Playback - StartMedia File Playback - End
Pause Media File Playback - Pause
is described in this section This application only has one usecase content distribution streaming services (CS)
Exoplayer is an application levelmedia player forAndroidpromoted by Google It provides an alternative to AndroidrsquosMediaPlayer API for playing audio and video both locally andover the Internet Exoplayer supports features not currentlysupported by Androidrsquos MediaPlayer API including DASHand SmoothStreaming adaptive playbacks
The TRIANGLE project has concentrated in testing justtwo of the Exoplayer features ldquoNoninteractive Playbackrdquoand ldquoPlay and Pauserdquo These features result in 6 test casesapplicable out of the test cases defined in TRIANGLETheseare test cases AUECS001 and AUECS002 in the App UserExperience domain test casesAECCS001 andAECCS002in the App Energy Consumption domain and test casesRESCS001 and RESCS002 in the Device Resources Usagedomain
The AUECS002 ldquoPlay and Pauserdquo test case descriptionbelonging to the AUE domain is shown in Table 3 The testcase description specifies the test conditions the generic appuser flow and the rawmeasurements which shall be collectedduring the execution of the test
The TRIANGLE project also offers a library that includesthe measurement points that should be inserted in thesource code of the app for enabling the collection of themeasurements specified Table 4 shows the measurementpoints required to compute the measurements specified intest case AUECS002
Wireless Communications and Mobile Computing 9
Table 5 Reference values for interpolation
Feature Domain KPI Synthetic MOS Calculation KPI min KPI maxNon-Interactive Playback AEC Average power consumption Type I 10 W 08 WNon-Interactive Playback AUE Time to load first media frame Type II KPI worst=20 msNon-Interactive Playback AUE Playback cut-off ratio Type I 50 0Non-Interactive Playback AUE Video resolution Type I 240p 720pNon-Interactive Playback RES Average CPU usage Type I 100 16Non-Interactive Playback RES Average memory usage Type I 100 40Play and Pause AEC Average power consumption Type I 10 W 08 WPlay and Pause AUE Pause operation success rate Type I 50 100Play and Pause RES Average CPU usage Type I 100 16Play and Pause RES Average memory usage Type I 100 40
The time to load first media picture measurement isobtained subtracting the timestamp of the measurementpoint ldquoMedia File Playback ndash Startrdquo from the measurementpoint ldquoMedia File Playback ndash First Picturerdquo
As specified in [11] all scenarios defined are applicableto the content streaming use case Therefore test cases inthe three domains currently supported by the testbed areexecuted in all the scenarios
Once the test campaign has finished the raw measure-ment results are processed to obtain the KPIs associated witheach test case average current consumption average time toload first media frame average CPU usage and so forth Theprocesses applied are detailed in Table 5 Based on previousexperiments performed by the authors the behaviour of thetime to load the first media frame KPI resembles the webresponse time KPI (ie the amount of time the user hasto wait for the service) and thus as recommended in theopinionmodel forweb search introduced in [9] a logarithmicinterpolation (type II) has been used for this metric
The results of the initial process that is the KPIs compu-tation are translated into synthetics-MOS values To computethese values reference benchmarking values for each of theKPIs need to be used according to the normalization andinterpolation process described in Section 5 Table 5 showswhat has been currently used by TRIANGLE for the AppUser Experience domain which is also used by NGMN asreference in their precommercial Trials document [23]
For example for the ldquotime to load first media framerdquo KPIshown in Table 5 the type of aggregation applied is averagingand the interpolation formula used is Type II
To achieve stable results each test case is executed 10times (10 iterations) in each network scenario The synthetic-MOS value in each domain is calculated by averaging themeasured synthetic-MOS values in the domain For examplesynthetic-MOS value is the RES domain obtained by aver-aging the synthetic-MOS value of ldquoaverage CPU usagerdquo andldquoaverage memory usagerdquo from the two test cases
Although Exoplayer supports several video streamingprotocols in this work only DASH [24] (Dynamic AdaptiveStreaming over HTTP) has been tested DASH clients shouldseamlessly adapt to changing network conditions by makingdecisions on which video segment to download (videosare encoded at multiple bitrates) The Exoplayerrsquos default
000
000
0
001
728
0
003
456
0
005
184
0
010
912
0
012
640
0
014
368
0
020
096
0
Timestamp
Video Resolution
0
200
400
600
800
1000
1200
Hor
izon
tal R
esol
utio
n
Figure 6 Video Resolution evolution in the Driving Urban Normalscenario
adaptation algorithm is basically throughput-based and someparameters control how often and when switching can occur
During the testing the testbed was configured with thedifferent network scenarios defined in [11] In these scenariosthe network configuration changes dynamically following arandom pattern resulting in different maximum throughputrates The expected behaviour of the application under testis that the video streaming client adapts to the availablethroughput by decreasing or increasing the resolution of thereceived video Figure 6 depicts how the client effectivelyadapts to the channel conditions
However the objective of the testing carried out in theTRIANGE testbed is not just to verify that the video stream-ing client actually adapts to the available maximum through-put but also to check whether this adaptation improves theusersrsquo experience quality
Table 6 shows a summary of the synthetic-MOS valuesobtained per scenario in one test case of each domain Thescores obtained in the RES andAECdomains are always highIn the AUE domain the synthetic MOS associated with theVideo Resolution shows low scores in some of the scenariosbecause the resolution decreases reasonable good scores inthe time to load first media and high scores in the time toplayback cut-off ratio Overall it can be concluded that the
10 Wireless Communications and Mobile Computing
Table 6 Synthetic MOS values per test case and scenario for the feature ldquoNoninteractive Playbackrdquo
AUE domain AEC domain RES domain
Test Case AUECS001 Test CaseAECCS001 Test Case RESCS001
ScenarioTime to loadfirst mediaframe
PlaybackCut-off ratio
VideoResolution
mode
AveragePower
Consumption
Average CPUUsage
AverageRAM Usage
HighSpeed DirectPassenger 21 31 23 47 43 42
Suburban Festival 38 47 31 48 43 41Suburban shopping mallbusy hours 37 37 13 48 44 41
Suburban shopping malloff-peak 36 31 23 48 43 41
Suburban stadium 38 29 21 47 44 41Urban Driving Normal 26 39 28 47 44 4Urban Driving TrafficJam 34 37 16 48 44 4
Urban Internet Cafe BusyHours 38 37 19 48 44 4
Urban Internet Cafe OffPeak 38 31 23 48 43 4
Urban Office 38 47 33 48 45 43Urban Pedestrian 39 26 2 47 44 4
35 36 23 47 44 41
DASH implementation of the video streaming client undertest is able to adapt to the changing conditions of the networkmaintaining an acceptable rate of video cut-off rebufferingtimes and resources usage
The final score in each domain is obtained by averagingthe synthetic-MOS values from all the tested network scenar-ios Figure 7 shows the spider diagram for the three domainstested In the User Experience domain the score obtained islower than the other domains due to the low synthetic-MOSvalues obtained for the video resolution
The final synthetic MOS for the use case Content Dis-tribution Streaming is obtained as a weighted average of thethree domains representing the overall QoE as perceived bythe userThefinal score for the Exoplayer version 1516 and thefeatures tested (Noninteractive Playback and Play and Pause)is 42 which means that the low score obtained in the videoresolution is compensated with the high scores in other KPIs
If an application under test has more than one use casethe next steps in the TRIANGLE mark project approachwould be the aggregation per use case and the aggregationover all use cases The final score the TRIANGLE mark is anestimation of the overall QoE as perceived by the user
In the current TRIANGLE implementation the weightsin all aggregations are the same Further research is neededto appropriately define the weights of each domain and eachuse case in the overall score of the applications
7 Conclusions
The main contribution of the TRIANGLE project is theprovision of a framework that generalizes QoE computation
and enables the execution of extensive and repeatable testcampaigns to obtainmeaningfulQoE scoresTheTRIANGLEproject has also defined amethodology which is based on thetransformation and aggregation of KPIs its transformationinto synthetic-MOS values and its aggregation over thedifferent domains and use cases
The TRIANGLE approach is a methodology flexibleenough to generalize the computation of QoE for any applica-tionservice Themethodology has been validated testing theDASH implementation in the Exoplayer App To confirm thesuitability of theweights used in the averaging process and theinterpolation parameters as well as to verify the correlationof the obtained MOS with that scored by users the authorshave started experiments with real users and initial results areencouraging
The process described produces a final TRIANGLEmarka single quality score which could eventually be used to cer-tify applications after achieving a consensus on the differentvalues of the process (weights limits etc) to use
Data Availability
Themethodology and results used to support the findings ofthis study are included within the article
Conflicts of Interest
The authors declare that they have no conflicts of inter-est
Wireless Communications and Mobile Computing 11
DER
AUE AEC
Device Resource Usage
User Experience Energy Consumption
Figure 7 Exoplayer synthetic-MOS values per domain
Acknowledgments
The TRIANGLE project is funded by the European UnionrsquosHorizon 2020 Research and Innovation Programme (GrantAgreement no 688712)
References
[1] ETSI ldquoHuman factors quality of experience (QoE) require-ments for real-time communication servicesrdquo Tech Rep 102643 2010
[2] ITU-T ldquoP10G100 (2006) amendment 1 (0107) new appendixI - definition of quality of experience (QoE)rdquo 2007
[3] F Kozamernik V Steinmann P Sunna and E WyckensldquoSAMVIQ - A new EBUmethodology for video quality evalua-tions in multimediardquo SMPTE Motion Imaging Journal vol 114no 4 pp 152ndash160 2005
[4] ITU-T ldquoG107 the E-model a computational model for use intransmission planningrdquo 2015
[5] J De Vriendt D De Vleeschauwer and D C Robinson ldquoQoEmodel for video delivered over an LTE network using HTTPadaptive streamingrdquo Bell Labs Technical Journal vol 18 no 4pp 45ndash62 2014
[6] S Jelassi G Rubino H Melvin H Youssef and G PujolleldquoQuality of Experience of VoIP Service A Survey of AssessmentApproaches andOpen Issuesrdquo IEEECommunications Surveys ampTutorials vol 14 no 2 pp 491ndash513 2012
[7] M Li C-L Yeh and S-Y Lu ldquoReal-Time QoE MonitoringSystem forVideo Streaming ServiceswithAdaptiveMedia Play-outrdquo International Journal of Digital Multimedia Broadcastingvol 2018 Article ID 2619438 11 pages 2018
[8] S Barakovic and L Skorin-Kapov ldquoSurvey and Challengesof QoE Management Issues in Wireless Networksrdquo Journal ofComputer Networks and Communications vol 2013 Article ID165146 28 pages 2013
[9] ITU-T ldquoG1030 estimating end-to-end performance in IPnetworks for data applicationsrdquo 2014
[10] ITU-T ldquoG1031 QoE factors in web-browsingrdquo 2014[11] EU H2020 TRIANGLE Project Deliverable D22 Final report
on the formalization of the certification process requirementsanduse cases 2017 httpswwwtriangle-projecteuproject-olddeliverables
[12] Q A Chen H Luo S Rosen et al ldquoQoE doctor diagnosingmobile app QoE with automated UI control and cross-layeranalysisrdquo in Proceedings of the Conference on Internet Mea-surement Conference (IMC rsquo14) pp 151ndash164 ACM VancouverCanada November 2014
[13] M A Mehmood A Wundsam S Uhlig D Levin N Sarrarand A Feldmann ldquoQoE-Lab Towards Evaluating Quality ofExperience for Future Internet Conditionsrdquo in Testbeds andResearch Infrastructure Korakis T Li H Tran-Gia P and HS Park Eds vol 90 of TridentCom 2011 Lnicst pp 286ndash301Springer Development of Networks and Communities BerlinGermany 2012
[14] D Levin A Wundsam A Mehmood and A FeldmannldquoBerlin The Berlin Experimental Router Laboratory for Inno-vative Networkingrdquo in TridentCom 2010 Lnicst T MagedanzA Gavras N H Thanh and J S Chase Eds vol 46 of LectureNotes of the Institute for Computer Sciences Social Informaticsand Telecommunications Engineering pp 602ndash604 SpringerHeidelberg Germany 2011
12 Wireless Communications and Mobile Computing
[15] K De Moor I Ketyko W Joseph et al ldquoProposed frameworkfor evaluating quality of experience in a mobile testbed-oriented living lab settingrdquo Mobile Networks and Applicationsvol 15 no 3 pp 378ndash391 2010
[16] R Sanchez-Iborra M-D Cano J J P C Rodrigues and JGarcia-Haro ldquoAnExperimental QoE Performance Study for theEfficient Transmission of High Demanding Traffic over an AdHoc Network Using BATMANrdquo Mobile Information Systemsvol 2015 Article ID 217106 14 pages 2015
[17] P Oliver-Balsalobre M Toril S Luna-Ramırez and R GarcıaGaraluz ldquoA system testbed for modeling encrypted video-streaming service performance indicators based on TCPIPmetricsrdquo EURASIP Journal on Wireless Communications andNetworking vol 2017 no 1 2017
[18] M Solera M Toril I Palomo G Gomez and J Poncela ldquoATestbed for Evaluating Video Streaming Services in LTErdquoWireless Personal Communications vol 98 no 3 pp 2753ndash27732018
[19] A Alvarez A Dıaz P Merino and F J Rivas ldquoField mea-surements of mobile services with Android smartphonesrdquoin Proceedings of the IEEE Consumer Communications andNetworking Conference (CCNC rsquo12) pp 105ndash109 Las Vegas NevUSA January 2012
[20] NGMN Alliance ldquoNGMN 5G white paperrdquo 2015 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2015NGMN 5G White Paper V1 0pdf
[21] ldquoInfrastructure and Design for Adaptivity and Flexibilityrdquo inMobile Information Systems Springer 2006
[22] J Nielsen ldquoResponse Times The Three Important Limitsrdquo inUsability Engineering 1993
[23] NGMN Alliance ldquoDefinition of the testing framework for theNGMN 5G pre-commercial networks trialsrdquo 2018 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2018180220 NGMN PreCommTrials Framework definition v1 0pdf
[24] 3GPP TS 26246 ldquoTransparent end-to-end Packet-switchedStreaming Services (PSS) Progressive Download and DynamicAdaptive Streaming over HTTP (3GP-DASH)rdquo 2018
International Journal of
AerospaceEngineeringHindawiwwwhindawicom Volume 2018
RoboticsJournal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Active and Passive Electronic Components
VLSI Design
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Shock and Vibration
Hindawiwwwhindawicom Volume 2018
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawiwwwhindawicom
Volume 2018
Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom
The Scientific World Journal
Volume 2018
Control Scienceand Engineering
Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom
Journal ofEngineeringVolume 2018
SensorsJournal of
Hindawiwwwhindawicom Volume 2018
International Journal of
RotatingMachinery
Hindawiwwwhindawicom Volume 2018
Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Navigation and Observation
International Journal of
Hindawi
wwwhindawicom Volume 2018
Advances in
Multimedia
Submit your manuscripts atwwwhindawicom
Wireless Communications and Mobile Computing 9
Table 5 Reference values for interpolation
Feature Domain KPI Synthetic MOS Calculation KPI min KPI maxNon-Interactive Playback AEC Average power consumption Type I 10 W 08 WNon-Interactive Playback AUE Time to load first media frame Type II KPI worst=20 msNon-Interactive Playback AUE Playback cut-off ratio Type I 50 0Non-Interactive Playback AUE Video resolution Type I 240p 720pNon-Interactive Playback RES Average CPU usage Type I 100 16Non-Interactive Playback RES Average memory usage Type I 100 40Play and Pause AEC Average power consumption Type I 10 W 08 WPlay and Pause AUE Pause operation success rate Type I 50 100Play and Pause RES Average CPU usage Type I 100 16Play and Pause RES Average memory usage Type I 100 40
The time to load first media picture measurement isobtained subtracting the timestamp of the measurementpoint ldquoMedia File Playback ndash Startrdquo from the measurementpoint ldquoMedia File Playback ndash First Picturerdquo
As specified in [11] all scenarios defined are applicableto the content streaming use case Therefore test cases inthe three domains currently supported by the testbed areexecuted in all the scenarios
Once the test campaign has finished the raw measure-ment results are processed to obtain the KPIs associated witheach test case average current consumption average time toload first media frame average CPU usage and so forth Theprocesses applied are detailed in Table 5 Based on previousexperiments performed by the authors the behaviour of thetime to load the first media frame KPI resembles the webresponse time KPI (ie the amount of time the user hasto wait for the service) and thus as recommended in theopinionmodel forweb search introduced in [9] a logarithmicinterpolation (type II) has been used for this metric
The results of the initial process that is the KPIs compu-tation are translated into synthetics-MOS values To computethese values reference benchmarking values for each of theKPIs need to be used according to the normalization andinterpolation process described in Section 5 Table 5 showswhat has been currently used by TRIANGLE for the AppUser Experience domain which is also used by NGMN asreference in their precommercial Trials document [23]
For example for the ldquotime to load first media framerdquo KPIshown in Table 5 the type of aggregation applied is averagingand the interpolation formula used is Type II
To achieve stable results each test case is executed 10times (10 iterations) in each network scenario The synthetic-MOS value in each domain is calculated by averaging themeasured synthetic-MOS values in the domain For examplesynthetic-MOS value is the RES domain obtained by aver-aging the synthetic-MOS value of ldquoaverage CPU usagerdquo andldquoaverage memory usagerdquo from the two test cases
Although Exoplayer supports several video streamingprotocols in this work only DASH [24] (Dynamic AdaptiveStreaming over HTTP) has been tested DASH clients shouldseamlessly adapt to changing network conditions by makingdecisions on which video segment to download (videosare encoded at multiple bitrates) The Exoplayerrsquos default
000
000
0
001
728
0
003
456
0
005
184
0
010
912
0
012
640
0
014
368
0
020
096
0
Timestamp
Video Resolution
0
200
400
600
800
1000
1200
Hor
izon
tal R
esol
utio
n
Figure 6 Video Resolution evolution in the Driving Urban Normalscenario
adaptation algorithm is basically throughput-based and someparameters control how often and when switching can occur
During the testing the testbed was configured with thedifferent network scenarios defined in [11] In these scenariosthe network configuration changes dynamically following arandom pattern resulting in different maximum throughputrates The expected behaviour of the application under testis that the video streaming client adapts to the availablethroughput by decreasing or increasing the resolution of thereceived video Figure 6 depicts how the client effectivelyadapts to the channel conditions
However the objective of the testing carried out in theTRIANGE testbed is not just to verify that the video stream-ing client actually adapts to the available maximum through-put but also to check whether this adaptation improves theusersrsquo experience quality
Table 6 shows a summary of the synthetic-MOS valuesobtained per scenario in one test case of each domain Thescores obtained in the RES andAECdomains are always highIn the AUE domain the synthetic MOS associated with theVideo Resolution shows low scores in some of the scenariosbecause the resolution decreases reasonable good scores inthe time to load first media and high scores in the time toplayback cut-off ratio Overall it can be concluded that the
10 Wireless Communications and Mobile Computing
Table 6 Synthetic MOS values per test case and scenario for the feature ldquoNoninteractive Playbackrdquo
AUE domain AEC domain RES domain
Test Case AUECS001 Test CaseAECCS001 Test Case RESCS001
ScenarioTime to loadfirst mediaframe
PlaybackCut-off ratio
VideoResolution
mode
AveragePower
Consumption
Average CPUUsage
AverageRAM Usage
HighSpeed DirectPassenger 21 31 23 47 43 42
Suburban Festival 38 47 31 48 43 41Suburban shopping mallbusy hours 37 37 13 48 44 41
Suburban shopping malloff-peak 36 31 23 48 43 41
Suburban stadium 38 29 21 47 44 41Urban Driving Normal 26 39 28 47 44 4Urban Driving TrafficJam 34 37 16 48 44 4
Urban Internet Cafe BusyHours 38 37 19 48 44 4
Urban Internet Cafe OffPeak 38 31 23 48 43 4
Urban Office 38 47 33 48 45 43Urban Pedestrian 39 26 2 47 44 4
35 36 23 47 44 41
DASH implementation of the video streaming client undertest is able to adapt to the changing conditions of the networkmaintaining an acceptable rate of video cut-off rebufferingtimes and resources usage
The final score in each domain is obtained by averagingthe synthetic-MOS values from all the tested network scenar-ios Figure 7 shows the spider diagram for the three domainstested In the User Experience domain the score obtained islower than the other domains due to the low synthetic-MOSvalues obtained for the video resolution
The final synthetic MOS for the use case Content Dis-tribution Streaming is obtained as a weighted average of thethree domains representing the overall QoE as perceived bythe userThefinal score for the Exoplayer version 1516 and thefeatures tested (Noninteractive Playback and Play and Pause)is 42 which means that the low score obtained in the videoresolution is compensated with the high scores in other KPIs
If an application under test has more than one use casethe next steps in the TRIANGLE mark project approachwould be the aggregation per use case and the aggregationover all use cases The final score the TRIANGLE mark is anestimation of the overall QoE as perceived by the user
In the current TRIANGLE implementation the weightsin all aggregations are the same Further research is neededto appropriately define the weights of each domain and eachuse case in the overall score of the applications
7 Conclusions
The main contribution of the TRIANGLE project is theprovision of a framework that generalizes QoE computation
and enables the execution of extensive and repeatable testcampaigns to obtainmeaningfulQoE scoresTheTRIANGLEproject has also defined amethodology which is based on thetransformation and aggregation of KPIs its transformationinto synthetic-MOS values and its aggregation over thedifferent domains and use cases
The TRIANGLE approach is a methodology flexibleenough to generalize the computation of QoE for any applica-tionservice Themethodology has been validated testing theDASH implementation in the Exoplayer App To confirm thesuitability of theweights used in the averaging process and theinterpolation parameters as well as to verify the correlationof the obtained MOS with that scored by users the authorshave started experiments with real users and initial results areencouraging
The process described produces a final TRIANGLEmarka single quality score which could eventually be used to cer-tify applications after achieving a consensus on the differentvalues of the process (weights limits etc) to use
Data Availability
Themethodology and results used to support the findings ofthis study are included within the article
Conflicts of Interest
The authors declare that they have no conflicts of inter-est
Wireless Communications and Mobile Computing 11
DER
AUE AEC
Device Resource Usage
User Experience Energy Consumption
Figure 7 Exoplayer synthetic-MOS values per domain
Acknowledgments
The TRIANGLE project is funded by the European UnionrsquosHorizon 2020 Research and Innovation Programme (GrantAgreement no 688712)
References
[1] ETSI ldquoHuman factors quality of experience (QoE) require-ments for real-time communication servicesrdquo Tech Rep 102643 2010
[2] ITU-T ldquoP10G100 (2006) amendment 1 (0107) new appendixI - definition of quality of experience (QoE)rdquo 2007
[3] F Kozamernik V Steinmann P Sunna and E WyckensldquoSAMVIQ - A new EBUmethodology for video quality evalua-tions in multimediardquo SMPTE Motion Imaging Journal vol 114no 4 pp 152ndash160 2005
[4] ITU-T ldquoG107 the E-model a computational model for use intransmission planningrdquo 2015
[5] J De Vriendt D De Vleeschauwer and D C Robinson ldquoQoEmodel for video delivered over an LTE network using HTTPadaptive streamingrdquo Bell Labs Technical Journal vol 18 no 4pp 45ndash62 2014
[6] S Jelassi G Rubino H Melvin H Youssef and G PujolleldquoQuality of Experience of VoIP Service A Survey of AssessmentApproaches andOpen Issuesrdquo IEEECommunications Surveys ampTutorials vol 14 no 2 pp 491ndash513 2012
[7] M Li C-L Yeh and S-Y Lu ldquoReal-Time QoE MonitoringSystem forVideo Streaming ServiceswithAdaptiveMedia Play-outrdquo International Journal of Digital Multimedia Broadcastingvol 2018 Article ID 2619438 11 pages 2018
[8] S Barakovic and L Skorin-Kapov ldquoSurvey and Challengesof QoE Management Issues in Wireless Networksrdquo Journal ofComputer Networks and Communications vol 2013 Article ID165146 28 pages 2013
[9] ITU-T ldquoG1030 estimating end-to-end performance in IPnetworks for data applicationsrdquo 2014
[10] ITU-T ldquoG1031 QoE factors in web-browsingrdquo 2014[11] EU H2020 TRIANGLE Project Deliverable D22 Final report
on the formalization of the certification process requirementsanduse cases 2017 httpswwwtriangle-projecteuproject-olddeliverables
[12] Q A Chen H Luo S Rosen et al ldquoQoE doctor diagnosingmobile app QoE with automated UI control and cross-layeranalysisrdquo in Proceedings of the Conference on Internet Mea-surement Conference (IMC rsquo14) pp 151ndash164 ACM VancouverCanada November 2014
[13] M A Mehmood A Wundsam S Uhlig D Levin N Sarrarand A Feldmann ldquoQoE-Lab Towards Evaluating Quality ofExperience for Future Internet Conditionsrdquo in Testbeds andResearch Infrastructure Korakis T Li H Tran-Gia P and HS Park Eds vol 90 of TridentCom 2011 Lnicst pp 286ndash301Springer Development of Networks and Communities BerlinGermany 2012
[14] D Levin A Wundsam A Mehmood and A FeldmannldquoBerlin The Berlin Experimental Router Laboratory for Inno-vative Networkingrdquo in TridentCom 2010 Lnicst T MagedanzA Gavras N H Thanh and J S Chase Eds vol 46 of LectureNotes of the Institute for Computer Sciences Social Informaticsand Telecommunications Engineering pp 602ndash604 SpringerHeidelberg Germany 2011
12 Wireless Communications and Mobile Computing
[15] K De Moor I Ketyko W Joseph et al ldquoProposed frameworkfor evaluating quality of experience in a mobile testbed-oriented living lab settingrdquo Mobile Networks and Applicationsvol 15 no 3 pp 378ndash391 2010
[16] R Sanchez-Iborra M-D Cano J J P C Rodrigues and JGarcia-Haro ldquoAnExperimental QoE Performance Study for theEfficient Transmission of High Demanding Traffic over an AdHoc Network Using BATMANrdquo Mobile Information Systemsvol 2015 Article ID 217106 14 pages 2015
[17] P Oliver-Balsalobre M Toril S Luna-Ramırez and R GarcıaGaraluz ldquoA system testbed for modeling encrypted video-streaming service performance indicators based on TCPIPmetricsrdquo EURASIP Journal on Wireless Communications andNetworking vol 2017 no 1 2017
[18] M Solera M Toril I Palomo G Gomez and J Poncela ldquoATestbed for Evaluating Video Streaming Services in LTErdquoWireless Personal Communications vol 98 no 3 pp 2753ndash27732018
[19] A Alvarez A Dıaz P Merino and F J Rivas ldquoField mea-surements of mobile services with Android smartphonesrdquoin Proceedings of the IEEE Consumer Communications andNetworking Conference (CCNC rsquo12) pp 105ndash109 Las Vegas NevUSA January 2012
[20] NGMN Alliance ldquoNGMN 5G white paperrdquo 2015 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2015NGMN 5G White Paper V1 0pdf
[21] ldquoInfrastructure and Design for Adaptivity and Flexibilityrdquo inMobile Information Systems Springer 2006
[22] J Nielsen ldquoResponse Times The Three Important Limitsrdquo inUsability Engineering 1993
[23] NGMN Alliance ldquoDefinition of the testing framework for theNGMN 5G pre-commercial networks trialsrdquo 2018 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2018180220 NGMN PreCommTrials Framework definition v1 0pdf
[24] 3GPP TS 26246 ldquoTransparent end-to-end Packet-switchedStreaming Services (PSS) Progressive Download and DynamicAdaptive Streaming over HTTP (3GP-DASH)rdquo 2018
International Journal of
AerospaceEngineeringHindawiwwwhindawicom Volume 2018
RoboticsJournal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Active and Passive Electronic Components
VLSI Design
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Shock and Vibration
Hindawiwwwhindawicom Volume 2018
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawiwwwhindawicom
Volume 2018
Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom
The Scientific World Journal
Volume 2018
Control Scienceand Engineering
Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom
Journal ofEngineeringVolume 2018
SensorsJournal of
Hindawiwwwhindawicom Volume 2018
International Journal of
RotatingMachinery
Hindawiwwwhindawicom Volume 2018
Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Navigation and Observation
International Journal of
Hindawi
wwwhindawicom Volume 2018
Advances in
Multimedia
Submit your manuscripts atwwwhindawicom
10 Wireless Communications and Mobile Computing
Table 6 Synthetic MOS values per test case and scenario for the feature ldquoNoninteractive Playbackrdquo
AUE domain AEC domain RES domain
Test Case AUECS001 Test CaseAECCS001 Test Case RESCS001
ScenarioTime to loadfirst mediaframe
PlaybackCut-off ratio
VideoResolution
mode
AveragePower
Consumption
Average CPUUsage
AverageRAM Usage
HighSpeed DirectPassenger 21 31 23 47 43 42
Suburban Festival 38 47 31 48 43 41Suburban shopping mallbusy hours 37 37 13 48 44 41
Suburban shopping malloff-peak 36 31 23 48 43 41
Suburban stadium 38 29 21 47 44 41Urban Driving Normal 26 39 28 47 44 4Urban Driving TrafficJam 34 37 16 48 44 4
Urban Internet Cafe BusyHours 38 37 19 48 44 4
Urban Internet Cafe OffPeak 38 31 23 48 43 4
Urban Office 38 47 33 48 45 43Urban Pedestrian 39 26 2 47 44 4
35 36 23 47 44 41
DASH implementation of the video streaming client undertest is able to adapt to the changing conditions of the networkmaintaining an acceptable rate of video cut-off rebufferingtimes and resources usage
The final score in each domain is obtained by averagingthe synthetic-MOS values from all the tested network scenar-ios Figure 7 shows the spider diagram for the three domainstested In the User Experience domain the score obtained islower than the other domains due to the low synthetic-MOSvalues obtained for the video resolution
The final synthetic MOS for the use case Content Dis-tribution Streaming is obtained as a weighted average of thethree domains representing the overall QoE as perceived bythe userThefinal score for the Exoplayer version 1516 and thefeatures tested (Noninteractive Playback and Play and Pause)is 42 which means that the low score obtained in the videoresolution is compensated with the high scores in other KPIs
If an application under test has more than one use casethe next steps in the TRIANGLE mark project approachwould be the aggregation per use case and the aggregationover all use cases The final score the TRIANGLE mark is anestimation of the overall QoE as perceived by the user
In the current TRIANGLE implementation the weightsin all aggregations are the same Further research is neededto appropriately define the weights of each domain and eachuse case in the overall score of the applications
7 Conclusions
The main contribution of the TRIANGLE project is theprovision of a framework that generalizes QoE computation
and enables the execution of extensive and repeatable testcampaigns to obtainmeaningfulQoE scoresTheTRIANGLEproject has also defined amethodology which is based on thetransformation and aggregation of KPIs its transformationinto synthetic-MOS values and its aggregation over thedifferent domains and use cases
The TRIANGLE approach is a methodology flexibleenough to generalize the computation of QoE for any applica-tionservice Themethodology has been validated testing theDASH implementation in the Exoplayer App To confirm thesuitability of theweights used in the averaging process and theinterpolation parameters as well as to verify the correlationof the obtained MOS with that scored by users the authorshave started experiments with real users and initial results areencouraging
The process described produces a final TRIANGLEmarka single quality score which could eventually be used to cer-tify applications after achieving a consensus on the differentvalues of the process (weights limits etc) to use
Data Availability
Themethodology and results used to support the findings ofthis study are included within the article
Conflicts of Interest
The authors declare that they have no conflicts of inter-est
Wireless Communications and Mobile Computing 11
DER
AUE AEC
Device Resource Usage
User Experience Energy Consumption
Figure 7 Exoplayer synthetic-MOS values per domain
Acknowledgments
The TRIANGLE project is funded by the European UnionrsquosHorizon 2020 Research and Innovation Programme (GrantAgreement no 688712)
References
[1] ETSI ldquoHuman factors quality of experience (QoE) require-ments for real-time communication servicesrdquo Tech Rep 102643 2010
[2] ITU-T ldquoP10G100 (2006) amendment 1 (0107) new appendixI - definition of quality of experience (QoE)rdquo 2007
[3] F Kozamernik V Steinmann P Sunna and E WyckensldquoSAMVIQ - A new EBUmethodology for video quality evalua-tions in multimediardquo SMPTE Motion Imaging Journal vol 114no 4 pp 152ndash160 2005
[4] ITU-T ldquoG107 the E-model a computational model for use intransmission planningrdquo 2015
[5] J De Vriendt D De Vleeschauwer and D C Robinson ldquoQoEmodel for video delivered over an LTE network using HTTPadaptive streamingrdquo Bell Labs Technical Journal vol 18 no 4pp 45ndash62 2014
[6] S Jelassi G Rubino H Melvin H Youssef and G PujolleldquoQuality of Experience of VoIP Service A Survey of AssessmentApproaches andOpen Issuesrdquo IEEECommunications Surveys ampTutorials vol 14 no 2 pp 491ndash513 2012
[7] M Li C-L Yeh and S-Y Lu ldquoReal-Time QoE MonitoringSystem forVideo Streaming ServiceswithAdaptiveMedia Play-outrdquo International Journal of Digital Multimedia Broadcastingvol 2018 Article ID 2619438 11 pages 2018
[8] S Barakovic and L Skorin-Kapov ldquoSurvey and Challengesof QoE Management Issues in Wireless Networksrdquo Journal ofComputer Networks and Communications vol 2013 Article ID165146 28 pages 2013
[9] ITU-T ldquoG1030 estimating end-to-end performance in IPnetworks for data applicationsrdquo 2014
[10] ITU-T ldquoG1031 QoE factors in web-browsingrdquo 2014[11] EU H2020 TRIANGLE Project Deliverable D22 Final report
on the formalization of the certification process requirementsanduse cases 2017 httpswwwtriangle-projecteuproject-olddeliverables
[12] Q A Chen H Luo S Rosen et al ldquoQoE doctor diagnosingmobile app QoE with automated UI control and cross-layeranalysisrdquo in Proceedings of the Conference on Internet Mea-surement Conference (IMC rsquo14) pp 151ndash164 ACM VancouverCanada November 2014
[13] M A Mehmood A Wundsam S Uhlig D Levin N Sarrarand A Feldmann ldquoQoE-Lab Towards Evaluating Quality ofExperience for Future Internet Conditionsrdquo in Testbeds andResearch Infrastructure Korakis T Li H Tran-Gia P and HS Park Eds vol 90 of TridentCom 2011 Lnicst pp 286ndash301Springer Development of Networks and Communities BerlinGermany 2012
[14] D Levin A Wundsam A Mehmood and A FeldmannldquoBerlin The Berlin Experimental Router Laboratory for Inno-vative Networkingrdquo in TridentCom 2010 Lnicst T MagedanzA Gavras N H Thanh and J S Chase Eds vol 46 of LectureNotes of the Institute for Computer Sciences Social Informaticsand Telecommunications Engineering pp 602ndash604 SpringerHeidelberg Germany 2011
12 Wireless Communications and Mobile Computing
[15] K De Moor I Ketyko W Joseph et al ldquoProposed frameworkfor evaluating quality of experience in a mobile testbed-oriented living lab settingrdquo Mobile Networks and Applicationsvol 15 no 3 pp 378ndash391 2010
[16] R Sanchez-Iborra M-D Cano J J P C Rodrigues and JGarcia-Haro ldquoAnExperimental QoE Performance Study for theEfficient Transmission of High Demanding Traffic over an AdHoc Network Using BATMANrdquo Mobile Information Systemsvol 2015 Article ID 217106 14 pages 2015
[17] P Oliver-Balsalobre M Toril S Luna-Ramırez and R GarcıaGaraluz ldquoA system testbed for modeling encrypted video-streaming service performance indicators based on TCPIPmetricsrdquo EURASIP Journal on Wireless Communications andNetworking vol 2017 no 1 2017
[18] M Solera M Toril I Palomo G Gomez and J Poncela ldquoATestbed for Evaluating Video Streaming Services in LTErdquoWireless Personal Communications vol 98 no 3 pp 2753ndash27732018
[19] A Alvarez A Dıaz P Merino and F J Rivas ldquoField mea-surements of mobile services with Android smartphonesrdquoin Proceedings of the IEEE Consumer Communications andNetworking Conference (CCNC rsquo12) pp 105ndash109 Las Vegas NevUSA January 2012
[20] NGMN Alliance ldquoNGMN 5G white paperrdquo 2015 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2015NGMN 5G White Paper V1 0pdf
[21] ldquoInfrastructure and Design for Adaptivity and Flexibilityrdquo inMobile Information Systems Springer 2006
[22] J Nielsen ldquoResponse Times The Three Important Limitsrdquo inUsability Engineering 1993
[23] NGMN Alliance ldquoDefinition of the testing framework for theNGMN 5G pre-commercial networks trialsrdquo 2018 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2018180220 NGMN PreCommTrials Framework definition v1 0pdf
[24] 3GPP TS 26246 ldquoTransparent end-to-end Packet-switchedStreaming Services (PSS) Progressive Download and DynamicAdaptive Streaming over HTTP (3GP-DASH)rdquo 2018
International Journal of
AerospaceEngineeringHindawiwwwhindawicom Volume 2018
RoboticsJournal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Active and Passive Electronic Components
VLSI Design
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Shock and Vibration
Hindawiwwwhindawicom Volume 2018
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawiwwwhindawicom
Volume 2018
Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom
The Scientific World Journal
Volume 2018
Control Scienceand Engineering
Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom
Journal ofEngineeringVolume 2018
SensorsJournal of
Hindawiwwwhindawicom Volume 2018
International Journal of
RotatingMachinery
Hindawiwwwhindawicom Volume 2018
Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Navigation and Observation
International Journal of
Hindawi
wwwhindawicom Volume 2018
Advances in
Multimedia
Submit your manuscripts atwwwhindawicom
Wireless Communications and Mobile Computing 11
DER
AUE AEC
Device Resource Usage
User Experience Energy Consumption
Figure 7 Exoplayer synthetic-MOS values per domain
Acknowledgments
The TRIANGLE project is funded by the European UnionrsquosHorizon 2020 Research and Innovation Programme (GrantAgreement no 688712)
References
[1] ETSI ldquoHuman factors quality of experience (QoE) require-ments for real-time communication servicesrdquo Tech Rep 102643 2010
[2] ITU-T ldquoP10G100 (2006) amendment 1 (0107) new appendixI - definition of quality of experience (QoE)rdquo 2007
[3] F Kozamernik V Steinmann P Sunna and E WyckensldquoSAMVIQ - A new EBUmethodology for video quality evalua-tions in multimediardquo SMPTE Motion Imaging Journal vol 114no 4 pp 152ndash160 2005
[4] ITU-T ldquoG107 the E-model a computational model for use intransmission planningrdquo 2015
[5] J De Vriendt D De Vleeschauwer and D C Robinson ldquoQoEmodel for video delivered over an LTE network using HTTPadaptive streamingrdquo Bell Labs Technical Journal vol 18 no 4pp 45ndash62 2014
[6] S Jelassi G Rubino H Melvin H Youssef and G PujolleldquoQuality of Experience of VoIP Service A Survey of AssessmentApproaches andOpen Issuesrdquo IEEECommunications Surveys ampTutorials vol 14 no 2 pp 491ndash513 2012
[7] M Li C-L Yeh and S-Y Lu ldquoReal-Time QoE MonitoringSystem forVideo Streaming ServiceswithAdaptiveMedia Play-outrdquo International Journal of Digital Multimedia Broadcastingvol 2018 Article ID 2619438 11 pages 2018
[8] S Barakovic and L Skorin-Kapov ldquoSurvey and Challengesof QoE Management Issues in Wireless Networksrdquo Journal ofComputer Networks and Communications vol 2013 Article ID165146 28 pages 2013
[9] ITU-T ldquoG1030 estimating end-to-end performance in IPnetworks for data applicationsrdquo 2014
[10] ITU-T ldquoG1031 QoE factors in web-browsingrdquo 2014[11] EU H2020 TRIANGLE Project Deliverable D22 Final report
on the formalization of the certification process requirementsanduse cases 2017 httpswwwtriangle-projecteuproject-olddeliverables
[12] Q A Chen H Luo S Rosen et al ldquoQoE doctor diagnosingmobile app QoE with automated UI control and cross-layeranalysisrdquo in Proceedings of the Conference on Internet Mea-surement Conference (IMC rsquo14) pp 151ndash164 ACM VancouverCanada November 2014
[13] M A Mehmood A Wundsam S Uhlig D Levin N Sarrarand A Feldmann ldquoQoE-Lab Towards Evaluating Quality ofExperience for Future Internet Conditionsrdquo in Testbeds andResearch Infrastructure Korakis T Li H Tran-Gia P and HS Park Eds vol 90 of TridentCom 2011 Lnicst pp 286ndash301Springer Development of Networks and Communities BerlinGermany 2012
[14] D Levin A Wundsam A Mehmood and A FeldmannldquoBerlin The Berlin Experimental Router Laboratory for Inno-vative Networkingrdquo in TridentCom 2010 Lnicst T MagedanzA Gavras N H Thanh and J S Chase Eds vol 46 of LectureNotes of the Institute for Computer Sciences Social Informaticsand Telecommunications Engineering pp 602ndash604 SpringerHeidelberg Germany 2011
12 Wireless Communications and Mobile Computing
[15] K De Moor I Ketyko W Joseph et al ldquoProposed frameworkfor evaluating quality of experience in a mobile testbed-oriented living lab settingrdquo Mobile Networks and Applicationsvol 15 no 3 pp 378ndash391 2010
[16] R Sanchez-Iborra M-D Cano J J P C Rodrigues and JGarcia-Haro ldquoAnExperimental QoE Performance Study for theEfficient Transmission of High Demanding Traffic over an AdHoc Network Using BATMANrdquo Mobile Information Systemsvol 2015 Article ID 217106 14 pages 2015
[17] P Oliver-Balsalobre M Toril S Luna-Ramırez and R GarcıaGaraluz ldquoA system testbed for modeling encrypted video-streaming service performance indicators based on TCPIPmetricsrdquo EURASIP Journal on Wireless Communications andNetworking vol 2017 no 1 2017
[18] M Solera M Toril I Palomo G Gomez and J Poncela ldquoATestbed for Evaluating Video Streaming Services in LTErdquoWireless Personal Communications vol 98 no 3 pp 2753ndash27732018
[19] A Alvarez A Dıaz P Merino and F J Rivas ldquoField mea-surements of mobile services with Android smartphonesrdquoin Proceedings of the IEEE Consumer Communications andNetworking Conference (CCNC rsquo12) pp 105ndash109 Las Vegas NevUSA January 2012
[20] NGMN Alliance ldquoNGMN 5G white paperrdquo 2015 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2015NGMN 5G White Paper V1 0pdf
[21] ldquoInfrastructure and Design for Adaptivity and Flexibilityrdquo inMobile Information Systems Springer 2006
[22] J Nielsen ldquoResponse Times The Three Important Limitsrdquo inUsability Engineering 1993
[23] NGMN Alliance ldquoDefinition of the testing framework for theNGMN 5G pre-commercial networks trialsrdquo 2018 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2018180220 NGMN PreCommTrials Framework definition v1 0pdf
[24] 3GPP TS 26246 ldquoTransparent end-to-end Packet-switchedStreaming Services (PSS) Progressive Download and DynamicAdaptive Streaming over HTTP (3GP-DASH)rdquo 2018
International Journal of
AerospaceEngineeringHindawiwwwhindawicom Volume 2018
RoboticsJournal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Active and Passive Electronic Components
VLSI Design
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Shock and Vibration
Hindawiwwwhindawicom Volume 2018
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawiwwwhindawicom
Volume 2018
Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom
The Scientific World Journal
Volume 2018
Control Scienceand Engineering
Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom
Journal ofEngineeringVolume 2018
SensorsJournal of
Hindawiwwwhindawicom Volume 2018
International Journal of
RotatingMachinery
Hindawiwwwhindawicom Volume 2018
Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Navigation and Observation
International Journal of
Hindawi
wwwhindawicom Volume 2018
Advances in
Multimedia
Submit your manuscripts atwwwhindawicom
12 Wireless Communications and Mobile Computing
[15] K De Moor I Ketyko W Joseph et al ldquoProposed frameworkfor evaluating quality of experience in a mobile testbed-oriented living lab settingrdquo Mobile Networks and Applicationsvol 15 no 3 pp 378ndash391 2010
[16] R Sanchez-Iborra M-D Cano J J P C Rodrigues and JGarcia-Haro ldquoAnExperimental QoE Performance Study for theEfficient Transmission of High Demanding Traffic over an AdHoc Network Using BATMANrdquo Mobile Information Systemsvol 2015 Article ID 217106 14 pages 2015
[17] P Oliver-Balsalobre M Toril S Luna-Ramırez and R GarcıaGaraluz ldquoA system testbed for modeling encrypted video-streaming service performance indicators based on TCPIPmetricsrdquo EURASIP Journal on Wireless Communications andNetworking vol 2017 no 1 2017
[18] M Solera M Toril I Palomo G Gomez and J Poncela ldquoATestbed for Evaluating Video Streaming Services in LTErdquoWireless Personal Communications vol 98 no 3 pp 2753ndash27732018
[19] A Alvarez A Dıaz P Merino and F J Rivas ldquoField mea-surements of mobile services with Android smartphonesrdquoin Proceedings of the IEEE Consumer Communications andNetworking Conference (CCNC rsquo12) pp 105ndash109 Las Vegas NevUSA January 2012
[20] NGMN Alliance ldquoNGMN 5G white paperrdquo 2015 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2015NGMN 5G White Paper V1 0pdf
[21] ldquoInfrastructure and Design for Adaptivity and Flexibilityrdquo inMobile Information Systems Springer 2006
[22] J Nielsen ldquoResponse Times The Three Important Limitsrdquo inUsability Engineering 1993
[23] NGMN Alliance ldquoDefinition of the testing framework for theNGMN 5G pre-commercial networks trialsrdquo 2018 httpswwwngmnorgfileadminngmncontentdownloadsTechnical2018180220 NGMN PreCommTrials Framework definition v1 0pdf
[24] 3GPP TS 26246 ldquoTransparent end-to-end Packet-switchedStreaming Services (PSS) Progressive Download and DynamicAdaptive Streaming over HTTP (3GP-DASH)rdquo 2018
International Journal of
AerospaceEngineeringHindawiwwwhindawicom Volume 2018
RoboticsJournal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Active and Passive Electronic Components
VLSI Design
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Shock and Vibration
Hindawiwwwhindawicom Volume 2018
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawiwwwhindawicom
Volume 2018
Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom
The Scientific World Journal
Volume 2018
Control Scienceand Engineering
Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom
Journal ofEngineeringVolume 2018
SensorsJournal of
Hindawiwwwhindawicom Volume 2018
International Journal of
RotatingMachinery
Hindawiwwwhindawicom Volume 2018
Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Navigation and Observation
International Journal of
Hindawi
wwwhindawicom Volume 2018
Advances in
Multimedia
Submit your manuscripts atwwwhindawicom
International Journal of
AerospaceEngineeringHindawiwwwhindawicom Volume 2018
RoboticsJournal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Active and Passive Electronic Components
VLSI Design
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Shock and Vibration
Hindawiwwwhindawicom Volume 2018
Civil EngineeringAdvances in
Acoustics and VibrationAdvances in
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Electrical and Computer Engineering
Journal of
Advances inOptoElectronics
Hindawiwwwhindawicom
Volume 2018
Hindawi Publishing Corporation httpwwwhindawicom Volume 2013Hindawiwwwhindawicom
The Scientific World Journal
Volume 2018
Control Scienceand Engineering
Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom
Journal ofEngineeringVolume 2018
SensorsJournal of
Hindawiwwwhindawicom Volume 2018
International Journal of
RotatingMachinery
Hindawiwwwhindawicom Volume 2018
Modelling ampSimulationin EngineeringHindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Chemical EngineeringInternational Journal of Antennas and
Propagation
International Journal of
Hindawiwwwhindawicom Volume 2018
Hindawiwwwhindawicom Volume 2018
Navigation and Observation
International Journal of
Hindawi
wwwhindawicom Volume 2018
Advances in
Multimedia
Submit your manuscripts atwwwhindawicom