summary of results study period 2005-2008 itu-t study group 12 performance and quality of service...

InternationalTelecommunicationUnion

2

ITU-T Study Group 12Performance and Quality of Service

Contents

Terms of referenceHighlights of achievements / Projects / FGsFuture workConclusions

Supplemental slides


3


Terms of ReferenceResponsible for Recommendations on the Responsible for Recommendations on the end-end-to-end transmission performanceto-end transmission performance of of terminals and networksterminals and networks, in relation to the , in relation to the perceived quality and acceptance by users perceived quality and acceptance by users of text, data, speech, and multi-media of text, data, speech, and multi-media applications. applications.

Although this work includes the related Although this work includes the related transmission implications of all networks and all transmission implications of all networks and all telecommunication terminals, a special focus is telecommunication terminals, a special focus is given to given to IP QoSIP QoS, , interoperabilityinteroperability and and implications for NGN, and also includes work on implications for NGN, and also includes work on performance and resource management.performance and resource management.

Lead SG for Performance and Quality of Service (QoS)


4


Highlights of achievements (I)

NewRecommendation P.505 " One-view visualisation of quality" "Quality Pie"

The test methodologies for subjective assessment of speech and audio are always evolving to take into account the new services and the new environments

SG12 is assessing the quality of codecs (26 test plans and 15 test results)- Collab. with SG16New Recommendation P.805 for conversation test.

The creation of 2 Focus Groups on car communication FITCAR (closed in Jan.2008) and CARCOM(opened in June 2008)

Round RobinTest on Human/artificial ears impedance Planned update of P.57


5


Highlights of achievements (II)

One of the major area on which SG12 has the global leadership is the development of quality models and tools for speech, but now also for data and multimedia services

Recommendation G.107 ("E-model") is updated on a regular basis. One of the major improvements has been to cover wideband speech.

Perceptual models for narrow and wideband speech– P.862.x-series Recommendations give a complete set of

models and information on how to use them. These types of models are working with reference signal.

– P.862.2 Wideband extension to Recommendation P.862 – P. 862.3 Application guide for objective Quality

Measurement based on Recommendation P.862


6


Highlights of achievements (III)

"In-service non-intrusive" quality models, without reference signal.

Recommendation P.564, which has also been implemented for wideband speech, specifies the minimum criteria for objective speech quality assessment models that predict the impact of observed IP network impairments on the one-way listening quality experienced by the end-user in IP/UDP/RTP-based telephony applications.

A call for proposal has been published for P.NAMS (Non Intrusive assessment of performance for video streaming) 2009


7


Highlights of achievements (IV)Recommendations for other media than speech, e.g. for videophone (G.1070) and for multimedia (G.1050).

– G.1070 provides algorithm estimating videophone quality to be used by QoS/QoE planners to ensure end user satisfaction. Recommendation

– G.1050 a network model for evaluating multimedia transmission performance over IP; based on statistical models on a broad range of known network configurations.

Recommendation G.1030 which provides a perceptual model for web browsing and defines a framework of tools to estimate end-to-end IP network performance for user applications.


8


Highlights of achievements (V)

One of the most popular Recommendations (Y.1541), initiated in SG13, has been updated several times by SG12 during the study period, to cover new services and media,

– e.g. Appendix VIII on effects of IP network performance on digital television transmission QoS.

This fundamental Recommendation should be the core for a global end-to-end QoS for interoperability.

The set of existing Recommendations has been completed by Y.1542, "a Framework for achieving End-to-End Performance objectives", linked with Y.1541.


9


Highlights of achievements (VI)SG12 experts and Officials has been active in Focus Groups and several Recommendations have been developed :FG-IPTV-GSI (G.1080 and 1081 have been approved and two other Rec. for monitoring are under progress), FG-NGN-GSI (Y.1542 has been approved, TR-NGN.QoS and TR-NGN.NHNperf are under progress).A new recommendation addressing Ethernet performance, Recommendation Y.1563 "Ethernet frame transfer and availability performance" has been approved in May 2008.For SLA (service level agreements) matters, G.1040 provides performance metric to be used by network service providers.


10


Future Work (I)For speech and audio quality : The hot topics are "bandwidth extension" (superwideband and fullband) and "binaural perception" consequences on loudness, binaural modelling,

acoustical interfaces (for terminals and test equipments such as artificial ear an mouth) and test methodologies (subjective and objective test methods).

Result from FG CARCOM : a specification for wideband car communication

To identify end-user performance expectations and associated metrics for audio, video, text, graphics quality, and control functionality.

to improve the accessibility of services.


11


Future Work (II)Industry needs to assess Quality of all types of media.

SG 12to work on audiovisual quality, and visual and multimedia quality measurements in telecommunication services.

This work to be done in close partnership with VQEG.SG12, the major actor for the development of quality modelling for speech. This leadership should be increased.

development of perceptual and parametric models for voice, audio, audiovisual and multimedia,

application of measurement tools and development and application of all planning and measurement tools. framework for diagnostic functions and their interaction with external objective models predicting media quality.

operational aspects of QoS, for the benefits of in particular developing countries.


12


Future Work (III)Quality models – Next future- New objective models for quality predictions are expected for speech quality evaluation (P.OLQA) at the end of 2009 (call already sent) and for objective prediction of P.835 scores (P.ONRA). - The plans to define a model for spoken conversations have been agreed.- Regarding multimedia, the call for proponents for P.NAMS has been sent out a new Recommendation planned for 2009, Next step following P.NAMS : P.NBAMS a non-intrusive bit stream model for the assessment of performance of multimedia streaming.QoE, QoS and Perf for multimedia (IPTV, HN)

G.OMVAS “Opinion Model for Video and Audio Streaming applications” and

Recs. on IPTV Quality monitoring (in addition to G. 1081)


13


Future Work (IV)NGN performance interworking issues requiring consideration include end-to-end multimedia performance interworking and interoperability issues as well as impacts of interworking functions. To define static and dynamic apportionment of performance objectives across networks and technologies.To examine the suitability of numerical objectives of IP QoS provisional Classes 6 and 7 given in Recommendation Y.1541.

– e.g. for MPEG-4 video and various Audio schemes. Urgent topics to be solved : IP Service Availability applicable in the network operator’s environmentFor QoS, QoE and Performance, SG12 plans to contribute significantly to improve the interoperability and convergence of services and networks.


14


Future Work (V)• A special focus is given to interoperability

to ensure end-to-end users' satisfaction. • SG12 also provides cross-ITU coordination

for the many aspects of Quality-of-Service (QoS) and Quality of Experience (QoE) and to foster consistency among ITU-T Study Groups, as well as with other related bodies such as the IETF, 3GPP so as to advance the interoperability of any QoS solutions.


15


Conclusion

In order to cover the future plans, it is proposed to rename Study Group 12 to “Performance, QoS and QoE” and define its mandate as follows:Study Group 12 is responsible for Recommendations on

performance, QoS and QoE for the full spectrum of terminals, networks and services ranging from speech over fixed circuit based networks to multimedia applications over networks that are mobile and packet-based.

A special focus is given to interoperability to ensure end-to-end users' satisfaction.

SG12 is Lead Study Group on QoS and QoE.


16


Supplemental Slides

Management teamStructure and ImplementationsWorking PartiesStatisticsWorkshopsQoS Flyer


17


Management Team

Judit Katona Kiss, TSB

Klemens Adler (G)

VP & WP2/12 Chair

Chuck Dvorak (US) VP & Lead SG Resp.

Gérald Lecucq (F)WP1/12 co-Chair

Paul Coverdale (Ca)VP & WP3/12 ChairCatherine

Quinquis (F)WP1/12 Co-Chair

Missing:Jean-Jacques Massima-Landji (Gabon) VP

JY Monfort ,SG Chair, is behind the camera


18


Study Group Structure

WP1 Telephonometry, Terminals and Subjective assessmentDevelops Recommendations on artificial heads/ears/Mouths/signals and associated test methodsDevelops subjective methods and test plans

WP2 Performance, Planning and Tools Develops quality models for speech and multimedia (E-Model, Perceptual models, INMD,…)

WP3 IP-Related QoS and resource management Defines parameters and models for QoS classes and IP performance.

Develop frameworks for IP-related QoS


19


Study Group ImplementationsFocus groups FITCAR and CARCOMAfter 5 meetings, the specification on characteristics and test methods for narrowband speech terminals in cars has been finalised by FG FITCAR. The specification has been submitted to SG12. The draft new Recommendation P.1100, based on the specification produced by the Focus Group FITCAR had consent at the last meeting of SG12.The new Focus Group CARCOM is focused on Wideband Speech

APSC-TelemovThe Advisory Panel for Standards Cooperation on Telecommunications related to Motor Vehicles - APSC TELEMOV was created at the end of the previous study period as a result of the Workshop on Standardization in Telecommunication for Motor Vehicles and intended as a cooperation group on all aspects of standardization related to telecommunications within and for motor vehicles.


20


Study Group Implementations

SG12 Regional Group for AfricaTo encourage to take an active participation in work of ITU-T SG12 by African Regulators and Operators, and in the implementation of the ITU-T Recommendations on Performance/QoS and QoE…

JRG on Multimedia Quality assessment has been formed from SG12 and SG9 in early 2004 with the motivation to represent the interest of ITU-T within VQEG. The work of JRG is successfully focussing on Multimedia quality aspects within VQEG Multimedia (MMQA) working group


21


WP 1 /12 – Telephonometry, terminalsand Subjective assessment

Q.4/12

Q.5/12

Hands-Free communications in vehiclesMr H.W. Gierlich

Telephonometrics methodologies for handsets and headset terminals Mr L. Madec

Q.7/12 Methods, tools and test plans for the subjective assessment… Mr P. Usai and Mrs C. Quinquis

Q.3/12

Analysis methods using complex measurement signals…. Mr H.W. Gierlich

Q.6/12

Speech transmission characteristics of speech terminals … Mr Shi

Denian


22


WP 2/12 – Performance, Planning and Tools

Q.9/12

Q.10/12

Perceptual-based objective methods for voice, audio and visual quality… Mr J. Berger

Transmission planning and performance considerations… Mr V. Siply

Q.20/12 Objective assessment of conversational speech Quality in networks Mr J. Pomy

Objective assessment of conversational speech Quality in networks Mr J. Pomy

Q.8/12

In-Service Non Intrusive Assessment of voice transmission performance Mr V. Barriac

Q.14/12

Extension of the E-Model Mrs U. Jekosch and Mr S. Möller

Q.12/12Performance evaluation of services based on speech technology Mr S.Möller&MrA. Raake


23


WP 3/12 – IP-Related QoS andRessource management

Q.13/12

Q.16/12

IP based multimedia services end to end transm. planning …. M. J. Pomy

IPTV and multimedia QoE/QoS performance requirements and assessment methods A. Takahashi

IPTV and multimedia QoE/QoS performance requirements and assessment methods A. Takahashi

Q.19/12

Performance of IP-based networks Mr A. Morton -Mr H.

Kim(associate)

Speech transmission Charact. And Meas.methods for IP terminals and gateways Mr U. Brunner

Q.11/12

Broadband and IP-related resource management Mr D. Mustill

Q.17/12

Call processing performance Call processing performance

Q.2/12


24


Statistics

During the study period, Study Group 12 examined a total of 364 contributionsOn the basis of these documents and of a thousand of temporary documents, Study Group 12:– drew up 21 new Recommendations;– amended/revised 23 Recommendations;– developed 1 Supplement,– developed 27 Test plans and 15 Test reports in the scope of collaboration with ITU-T SG16 for speech and audio coders.


25


Workshops organized by SG12 (I)

End-to-End QoS-QoE / 14-16 June 2006 / Geneva

http://www.itu.int/ITU-T/worksem/qos/200606/index.htmlOrganised by SG12 with the collaboration with SG2 and SG16

The sessions were dedicated to Inter-domain QoS, Interoperability testing and terminal issues, Home Networking QoS/QoE, Quality of Experience (QoE) of new services, NGN-related QoS and the accumulation of network impairements, and Quality prediction

This workshop brought together more than 100 experts from all over the world who are involved in the definition and the delivery of end-to-end quality of experience (QoE) and quality of service (QoS).


26


Workshops organized by SG12 (II)From Speech to Audio/10-12 Sept 2008/Lannion, France

http://www.itu.int/ITU-T/worksem/speechaudio/index.html

The workshop was organized by WP1/12 and involved participants from ITU-T SG12, SG16, ITU-R 6C. Representatives of ETSI STQ, 3GPP SA4 also joint the workshop that was supported by France Telecom, Alcatel Lucent and DEGA (the German acoustical society)

The workshop was very productive and agreed on some actions

- to enhance the constructive dialogue between universities, standardization bodies and companies.

Ideas to improve the future collab have been raised need to be implemented

- to provide inputs to develop new methodologies covering the complete audio bandwidth from Narrow Band to Fullband New Recs. There is a need for improved collab between ITU-T and ITU-R for audio quality.

Keynote Speakers

Pr. J. BlauertBochum Univ.

Pr. S. MeunierUniv. Aix-Marseille


27


Workshops supported by SG12The Fully Networked Car, Geneva (Palexpo)

– http://www.itu.int/ITU-T/worksem/ict-auto/index.html

2-4 March 2005 7 - 9 March 2007 5 - 7 March 2008

– The next one : 4 - 5 March 2009 Opportunities and Challenges in Home Networking Geneva, 13 - 14 October 2005

2nd ETSI STQ Workshop on Wideband Speech Quality in Terminals and Networks: Assessment and Prediction, 22-23 June 2005, Mainz, Germany

ETSI Workshop on 'Effects of transmission performance on Multimedia QoS'Prague, Czech Republic, 17-19 June 2008


28


A QoS Flyer (page 2)

True Quality of Service captures the User Experience (QoE)The G.1000 Framework helps capture the many QoS dimensions, andacknowledges that QoS can be viewed from four different perspectives


29


Acknowledgements

Thanks to the Management Team (including Rodolfo Ceruti) for the strong support, to Hans Gierlich (as Chair of the Focus Group and a Fully networked car workshop), to the efficient Rapporteurs and contributors and to Daniel Waturu for his involvement in SG12 Regional Group for Africa. Many thanks to TSB support and specially Judit and Sarah for SG12, Sami and his team for the specific websites (E-model,..), Paolo and his team for the organization of workshops, Sylvie, Kristin and Richard for the meetings facilities.


30


Thank you!Mr Jean-Yves MonfortFrance Telecom/Orange-Labs

Standardization Senior Manager & Senior expert in QoS

During three years, I was deputy Director for standards in France Telecom R&D.

I was previously involved as deputy Head of a Lab. Dealing with speech, Audio, video and usability Quality of services, terminals and equipments

One of my hobbies

summary of results study period 2005-2008 itu-t study group 12 performance and quality of service...

Documents

quality of service qos

itut study group

perceived quality

telecommunication terminals

ip qos

users of text

special focus

multimedia applications