collaborative measurement of internet quality in lebanon...

52
Collaborative Measurement of Internet Quality in Lebanon http://comiqual.usj.edu.lb Marc Ibrahim Saint-Joseph University - Lebanon [email protected]

Upload: others

Post on 05-Jul-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

Collaborative Measurement of Internet Quality in Lebanon

http://comiqual.usj.edu.lb

Marc Ibrahim

Saint-Joseph University - Lebanon

[email protected]

Page 2: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

Outline

2 [email protected]

Large scale measurement platforms

Comiqual

Toward LMAP standard

Page 3: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

Item U.S. Adults (%)

Food 11

Spouse/significant other 29

Car 30

Internet access 44

Computer/Laptop 49

Mobile phone 51

Television 55

Sex 58

Tablet computer 59

eReader 63

Navigation system 69

Social networking sites 78

Could live a week or more without ...

3 [email protected]

Source : Harris Interactive. The Harris Poll® #13, January 30, 2014

Page 4: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

- Assess QoS offer - Compare to others - Understand the user

- Foster competition - Quality requirements and

enforcement - Compare to other countries

- Quantify QoE - Check SLA (corporate) - Diagnose faults - Compare offers

Internet stakeholders: iEyes

4 [email protected]

QoE

Co

ntro

l

Se

rvic

e

Regulator

Need for capturing the «state» of the Internet

Consumer

Provider

Page 5: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Internet is : + Well engineering and designed system undergoing continous optimization

– Best effort, heterogeneous, with information hiding practices

• Large-scale measurements : – Collecting performance metrics from a large

number of measurement points

– A tool for all stakeholders to get a deep insight about Internet performance

Large-scale measurements Measuring the hidden part of the i-ceberg

5 [email protected]

Page 6: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Software-based or hardware-based probes. Hardware probes : + Insensitivity to end-user device specs and state

+ Ability to avoid interfering with user traffic

- Less mobility

• Active or/and passive measurements

• Commercial or/and academic

• Fixed or/and mobile Internet access

• End-user or/and ISP perspective

Characteristics of Large-scale measurement platforms

6 [email protected]

Page 7: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Launched by RIPE NCC (http://atlas.ripe.net )

• Hardware-based probes

– TP-link + OpenWRT

– End-user home network

– Anywhere in ISP network

– Anchors: more powerful probes

RIPE Atlas

7 [email protected]

Page 8: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Simple active measurements : – Ping, traceroute

– DNS, HTTP, SSL certificate retrieval queries

• User defined measurements : – Credits system

– Customized measurement tasks

– Can use any subset of Atlas probes

– Useful for testing own network accessibility and efficiency

• Data available for research studies

RIPE Atlas

8 [email protected]

Page 9: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

Architecture

9 [email protected]

Page 10: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

SamKnows

10 [email protected]

Page 11: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Speed test

• OpenSignal

• Portolan

• Sensorly

• Mobiperf

• Netradar

Others platforms

11 [email protected]

Page 12: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

Outline

12 [email protected]

Large scale measurement platforms

Comiqual

Toward LMAP standard

Page 13: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Small country: 10k sqkm, 4M inhabitants

• NRI 2013 rank 95

• 1 incumbent public operator (monopoly MoT)

• 2 state-owned mobile operators (duopoly) – Alfa: managed by Orascom / Touch: managed by Zain

– More than 1.8M smartphones registered

– 3G less than 3 years old, now start of data 4G-LTE commercial phase

– Stress on the network !!

• TRA with no Authority

• Politics.. No transparency..

• #Ontornet #FlipTheSwitch #OpenTheTap #UnleashTheBandwidth campaign

Lebanese context

13 [email protected]

Page 14: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

Comiqual in one slide http://comiqual. usj.edu.lb

14 [email protected]

• Transparent, neutral

• User experience: measurement software on user devices

• Collaborative (crowd-sourcing): every one can participate

Platform for measuring

Internet quality in Lebanon

• A tool for users to assess, compare, and choose

• A user feedback useful for mobile operators and ISPs

• Contribute to the effort toward broadband Objectives:

• Measurement platform. Measurements agents and servers

• Open data Main output

Page 15: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

From idea to platform

15 [email protected]

ISOC QoS Workgroup

USJ project team

Funding: USJ + ISOC

Start: June 2013

Beta version online:

Sept 2014

Page 16: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

16 [email protected]

Controller

Collector

MA MP

Control channel

Report channel

Meas. traffic

Meas. control

Database

Web UI admin

results

Raw data

Meas.

Center

user

Config/Control

Meas.

Agent

Meas.

Peer

Page 17: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• The central entity controlling all MAs and collecting measurement results.

• Control : – Authentication – Send measurement tasks – Determines MA provider name

• Collect : receive and store measurement results in the system database.

Measurement center

17 [email protected]

Page 18: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Software-based: mobile application running on end-user smartphone. – MA = device + user

• Hardware-based (Standalone) • Raspberry Pi and TP-Link access points (3020 and 3040) • Typically connected to an ADSL modem

• MAs are jointly controlled by MC and users

Measurement agent

18 [email protected]

Page 19: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• A prober to execute measurement

• A manager to control the MA and communication with the MC

• A “contexter” to find the measurement context

Measurement agent

19 [email protected]

PROBER CONTEXTER

MANAGER

Page 20: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Any server on the Internet measurements are done with. • A dedicated measurement server is installed at Beirut IXP

– MAs will be performing measurements with this server to assess the national IXP performance

• MA and MP exchange two type of data – The measurement traffic that is injected to perform the test – The measurement control data that controls the measurement

execution

Measurement peer

20 [email protected]

Page 21: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• The Comiqual web site gives access for – Any user to visualize aggregated results on google map – Registered users to view their measurements history – Administrator to manage the platform

• Create, activate, disactivate measurement tasks • Create new measurements • Control all MAs activity

Web user interface

21 [email protected]

Page 22: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

Global overview

22 [email protected] 22

ISP2 ISP1

MOT

Beirut IX Meas. Server

Meas. agent

Link

Meas. traffic

Control and collection traffic

Standalone device

mobile device

Measurement Center = Controller + Collector

Page 23: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Comiqual uses active measurements – Network measurement that generates a measurement

traffic in order to compute performance metrics.

• Two main categories of measurements will be performed: – Access measurements : Received signal power, access

delay. – Active IP measurements : end-to-end quality

• The user can choose to make his measurement anonymous or not – A user can view his non anonymous measurement on the

project web site

Measurements

23 [email protected]

Page 24: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

Measurements

24 [email protected]

Variety of tools. Duplication for backup. • RTT: Ping, HTTP

ping • throughouput:

Mlab, Iperf, speed test, our tool on Beirut IX

• UDP measurement tool coded for BeirutIX

Page 25: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Measurement metric

– Quality output of a measurement.

– e.g.: average RTT, max RTT, TCP download throughput, etc.

• Measurement argument

– A measurement tool has in general some options to be specified like packet size, timeout, target URL, etc.

– e.g. ping –n 10 –l 100 www.test.net

Measurement parameters

25 [email protected]

Page 26: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Measurement context

– The context is a set of variables describing the state of MA, its connectivity, its environment, etc., while performing a given measurement.

– ISP, mobile operator, signal strength, time, cell-id, IPvx, battery level, GPS coordinates (start and end for mobility), MA application version, ...

Measurement parameters

26 [email protected]

Page 27: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• A measurement that MA is asked to do.

• Two type of tasks:

– Instructed measurement task : measurement task sent by MC to MA.

– User defined measurement task : Measurement task initiated manually by the user.

Measurement tasks

27 [email protected]

Page 28: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• A task consists mainly of: – Creation and end dates of the task – A measurement to do – Arguments of the measurement – Metrics to be obtained by the measurement – Constraints

• Constraints : – A measurement task can have a set of constraints on

context parameters. The MA will not perform the task until the context constraints are met.

– e.g.: perform a ping in a given region, at a given time, for a a given type of mobiles, ...

Measurement tasks

28 [email protected]

Page 29: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

Measurement data model

29 [email protected]

constraint object

{

"context_name":

{

"lower_bound":"value",

"upper_bound":"value",

"values":

[

value1,

...

valueN

]

}

}

Page 30: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

task object

{

"task_id": "unique id of the task",

"task_name": "unique name of the task",

"version": "task version",

"description": "description text",

"meas_name": "name of the measurement",

"end_date": "date of task end",

"repeat_interval": "daily or weekly",

"arguments": [

{"arg1_name":"value"},

...

],

"metrics": [

"metric1_name",

...

],

"constraints": [

{constraint_object1},

...

]

}

Measurement data model

30 [email protected]

Page 31: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

meas_result

{

"task_id":"id of meas task"

"success":"OK or reason of

failure":

"arguments":

[

{"arg1_name":"value"},

...

{"argN_name":"value"}

],

"metrics":

[

{"metric1_name":"value"},

...

{"metricN_name":"value"},

],

}

Measurement data model

31 [email protected]

task_result

{

"meas_result":

{//meas_result object}

"context":

{

//context object.

}

}

Page 32: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Simple control and reporting protocol

• REST architechture – JSON messages encapsulated in HTTP standard methods

(GET, PUT, POST)

• TLS security using HTTPS

• MA initiates control and reporting communications

• If not anonymous mode, user should authenticate by either using a comiqual account or other third-party identity services (google, facebook, ...) – authentication cookies are used in every message to

ensure the user is authenticated

MA-MC communication

32 [email protected]

Page 33: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

33 [email protected]

instruct message

{

"ma_id": ma id,

"controller_url" :http://...

"collector_url" :http://...

"keep_tasks_id":[

id1,

...

],

"new_tasks": [

{//task1 object},

...

]

}

Page 34: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

PROBER CONTEXTER

MANAGER Controller

Collector

MA MC

CHECKIN message

checkin

{

"ma_id": ma_id,//-1 if first time

"device":{//device object.

//Mandatory when ma_id==-1},

“probeness”: two possible values for now

"context": {//context object. optional},

"current_tasks":

[

{"task_id":value,"task_version":value},

...

{"task_id":value,"task_version":value}

]

}

//+ cookie in the HTTP header

Page 35: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

PROBER CONTEXTER

MANAGER Controller

Collector

MA MC

INSTRUCT message

instruct message

{

"ma_id": ma id,

"controller_url" :http://...

"collector_url" :http://...

"keep_tasks_id":[

id1,

...

],

"new_tasks":[

{//task1 object},

...

]

}

//+ cookie in the HTTP header

Page 36: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

PROBER CONTEXTER

MANAGER Controller

Collector

MA MC

Page 37: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

PROBER CONTEXTER

MANAGER Controller

Collector

MA MC

REPORT message

ma_report

{

"ma_id": "value"

"report":

[

{//task_result object1},

...

{// task_result objectN }

]

}

//+cookie in the HTTP header

Page 38: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Storage of system and measurement data – Users, measurement parameters, measurement tasks – Raw measurement data

• Flexible database design – Simple creation of new measurement and parameters

(no predefined fields)

Database

38 [email protected]

meas_key

meas_param_id

meas_param_value

Page 39: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Measurement results will be publicly available

• JSON will be used as an open data format

• Users will be able to:

– Get the raw open data in JSON format

– Visualize the most important indicators on a map

– Define their own statistics using an online statistics tool we are developing.

Open data

39

Page 40: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

Issues and perspectives

40 [email protected]

• Perform throughput measurement with minimum TCP traffic

• Find incentives to make people contribute

• Converge toward LMAP standard

Page 41: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

Outline

41 [email protected]

Large scale measurement platforms

Comiqual

Toward LMAP standard

Page 42: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• No interaction between current platforms – Proprietary and closed systems

– Redundancy of measurements

• Deployment cost : – No modular components ready to use

• Tests and metric are not well defined – Problem comparing results from different

platforms

– Hard to aggregate measurement results

Need for standards

42 [email protected]

Page 43: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Architecture : Despite their differences, all platforms have common architectural components

• Measurements: – How to measure a specific metric (related to IPPM) – Scheduling measurement execution – Location of the probe (clarify the exact meaning of

Internet access) – Measurement environment : define and avoid cross

traffic while measuring the network

• Control and report protocols

Possible tracks for standardization

43 [email protected]

Page 44: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Large-Scale Measurement of Broadband Performance

• Recent IETF workgroup

• Working on standardizing large-scale measurement systems

• focusing on control and report protocols

• Considers only measurement systems under the control of a single organization (LMAP domain

LMAP

44 [email protected]

Page 45: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• ^

• |

• +-------------+ IPPM

• +---------------+ Measurement | Measurement | Scope

• | Measurement |<------------>| Peer | |

• | Agent | Traffic +-------------+ v

• +------->| | ^

• | +---------------+ |

• | ^ | |

• | Instruction | | Report |

• | | +-----------------+ |

• | | | |

• | | v LMAP

• | +------------+ +------------+ Scope

• | | Controller | | Collector | |

• | +------------+ +------------+ v

• | ^ ^ | ^

• | | | | |

• | | +----------+ | |

• | | | v |

• +------------+ +----------+ +--------+ +----------+ |

• |Bootstrapper| |Subscriber|--->| data |<---|repository| Out

• +------------+ |parameter | |analysis| +----------+ of

• |database | | tools | Scope

• +----------+ +--------+ |

• |

• v

LMAP components

45 [email protected]

Page 46: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

• Information model as an abstract definition of information exchanged between MA and Controller/Collector

• Guide standardization of one or more control and reporting protocols

• Information model sections : – Configuration

– Instruction

– Reporting results

– Capability of MA

LMAP information model

46 [email protected]

Page 47: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

object {

ma-task-obj ma-instruction-tasks<0..*>;

ma-channel-obj ma-report-channels<0..*>;

ma-schedule-obj ma-instruction-schedules<0..*>;

ma-suppression-obj ma-suppression;

} ma-instruction-obj;

LMAP information model example

47 [email protected]

Page 48: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

object {

string ma-schedule-name;

ma-sched-task-obj ma-schedule-tasks<0..*>;

ma-timing-obj ma-schedule-timing;

} ma-schedule-obj;

LMAP information model example

48 [email protected]

Page 49: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

object { union { ma-periodic-obj ma-timing-periodic; ma-calendar-obj ma-timing-calendar; ma-one-off-obj ma-timing-one-off; ma-immediate-obj ma-timing-immediate; ma-startup-obj ma-timing-startup; } [int ma-timing-random-spread;] // milliseconds } ma-timing-obj;

LMAP information model example

49 [email protected]

Page 50: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

object {

string ma-task-name;

uri ma-task-registry-entry;

string ma-role;

name-value-pair ma-task-options<0..*>;

} ma-task-obj;

LMAP information model example

50 [email protected]

Page 51: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

1. Marcelo Bagnulo, Trevor Burbridge, Sam Crawford, Philip Eardley, Juergen Schoenwaelder, and Brian Trammell. Building a Standard Measurement Platform, IEEE Communications Magazine , May 2014

2. Marcelo Bagnulo, Philip Eardley, Brian Trammell, and Rolf Winter. Standardizing Large-Scale Measurement Platforms, ACM SIGCOMM Computer Communication Review, Volume 43, Number 2, April 2013

3. RIPE Atlas. https://atlas.ripe.net 4. SamKnows. https://www.samknows.com 5. P. Eardley, A. Morton, M. Bagnulo, T. Burbridge, P. Aitken, and A.

Akhter, A framework for large-scale measurement platforms (LMAP), Internet Engineering Task Force, Internet-Draft draft-ietf-lmap-framework-08, Aug. 2014, work in Progress. [Online]. Available: http://tools.ietf.org/html/draft-ietf-lmap-framework-08

Références

51 [email protected]

Page 52: Collaborative Measurement of Internet Quality in Lebanon …comiqual.usj.edu.lb/documentations/Comiqual presentation... · 2015-03-27 · •User experience: measurement software

52 [email protected]

THANK YOU