collaborative measurement of internet quality in lebanon...
TRANSCRIPT
Collaborative Measurement of Internet Quality in Lebanon
http://comiqual.usj.edu.lb
Marc Ibrahim
Saint-Joseph University - Lebanon
Item U.S. Adults (%)
Food 11
Spouse/significant other 29
Car 30
Internet access 44
Computer/Laptop 49
Mobile phone 51
Television 55
Sex 58
Tablet computer 59
eReader 63
Navigation system 69
Social networking sites 78
Could live a week or more without ...
Source : Harris Interactive. The Harris Poll® #13, January 30, 2014
- Assess QoS offer - Compare to others - Understand the user
- Foster competition - Quality requirements and
enforcement - Compare to other countries
- Quantify QoE - Check SLA (corporate) - Diagnose faults - Compare offers
Internet stakeholders: iEyes
QoE
Co
ntro
l
Se
rvic
e
Regulator
Need for capturing the «state» of the Internet
Consumer
Provider
• Internet is : + Well engineering and designed system undergoing continous optimization
– Best effort, heterogeneous, with information hiding practices
• Large-scale measurements : – Collecting performance metrics from a large
number of measurement points
– A tool for all stakeholders to get a deep insight about Internet performance
Large-scale measurements Measuring the hidden part of the i-ceberg
• Software-based or hardware-based probes. Hardware probes : + Insensitivity to end-user device specs and state
+ Ability to avoid interfering with user traffic
- Less mobility
• Active or/and passive measurements
• Commercial or/and academic
• Fixed or/and mobile Internet access
• End-user or/and ISP perspective
Characteristics of Large-scale measurement platforms
• Launched by RIPE NCC (http://atlas.ripe.net )
• Hardware-based probes
– TP-link + OpenWRT
– End-user home network
– Anywhere in ISP network
– Anchors: more powerful probes
RIPE Atlas
• Simple active measurements : – Ping, traceroute
– DNS, HTTP, SSL certificate retrieval queries
• User defined measurements : – Credits system
– Customized measurement tasks
– Can use any subset of Atlas probes
– Useful for testing own network accessibility and efficiency
• Data available for research studies
RIPE Atlas
Architecture
SamKnows
• Speed test
• OpenSignal
• Portolan
• Sensorly
• Mobiperf
• Netradar
Others platforms
• Small country: 10k sqkm, 4M inhabitants
• NRI 2013 rank 95
• 1 incumbent public operator (monopoly MoT)
• 2 state-owned mobile operators (duopoly) – Alfa: managed by Orascom / Touch: managed by Zain
– More than 1.8M smartphones registered
– 3G less than 3 years old, now start of data 4G-LTE commercial phase
– Stress on the network !!
• TRA with no Authority
• Politics.. No transparency..
• #Ontornet #FlipTheSwitch #OpenTheTap #UnleashTheBandwidth campaign
Lebanese context
Comiqual in one slide http://comiqual. usj.edu.lb
• Transparent, neutral
• User experience: measurement software on user devices
• Collaborative (crowd-sourcing): every one can participate
Platform for measuring
Internet quality in Lebanon
• A tool for users to assess, compare, and choose
• A user feedback useful for mobile operators and ISPs
• Contribute to the effort toward broadband Objectives:
• Measurement platform. Measurements agents and servers
• Open data Main output
From idea to platform
ISOC QoS Workgroup
USJ project team
Funding: USJ + ISOC
Start: June 2013
Beta version online:
Sept 2014
Controller
Collector
MA MP
Control channel
Report channel
Meas. traffic
Meas. control
Database
Web UI admin
results
Raw data
Meas.
Center
user
Config/Control
Meas.
Agent
Meas.
Peer
• The central entity controlling all MAs and collecting measurement results.
• Control : – Authentication – Send measurement tasks – Determines MA provider name
• Collect : receive and store measurement results in the system database.
Measurement center
• Software-based: mobile application running on end-user smartphone. – MA = device + user
• Hardware-based (Standalone) • Raspberry Pi and TP-Link access points (3020 and 3040) • Typically connected to an ADSL modem
• MAs are jointly controlled by MC and users
Measurement agent
• A prober to execute measurement
• A manager to control the MA and communication with the MC
• A “contexter” to find the measurement context
Measurement agent
PROBER CONTEXTER
MANAGER
• Any server on the Internet measurements are done with. • A dedicated measurement server is installed at Beirut IXP
– MAs will be performing measurements with this server to assess the national IXP performance
• MA and MP exchange two type of data – The measurement traffic that is injected to perform the test – The measurement control data that controls the measurement
execution
Measurement peer
• The Comiqual web site gives access for – Any user to visualize aggregated results on google map – Registered users to view their measurements history – Administrator to manage the platform
• Create, activate, disactivate measurement tasks • Create new measurements • Control all MAs activity
Web user interface
Global overview
22 [email protected] 22
ISP2 ISP1
MOT
Beirut IX Meas. Server
Meas. agent
Link
Meas. traffic
Control and collection traffic
Standalone device
mobile device
Measurement Center = Controller + Collector
• Comiqual uses active measurements – Network measurement that generates a measurement
traffic in order to compute performance metrics.
• Two main categories of measurements will be performed: – Access measurements : Received signal power, access
delay. – Active IP measurements : end-to-end quality
• The user can choose to make his measurement anonymous or not – A user can view his non anonymous measurement on the
project web site
Measurements
Measurements
Variety of tools. Duplication for backup. • RTT: Ping, HTTP
ping • throughouput:
Mlab, Iperf, speed test, our tool on Beirut IX
• UDP measurement tool coded for BeirutIX
• Measurement metric
– Quality output of a measurement.
– e.g.: average RTT, max RTT, TCP download throughput, etc.
• Measurement argument
– A measurement tool has in general some options to be specified like packet size, timeout, target URL, etc.
– e.g. ping –n 10 –l 100 www.test.net
Measurement parameters
• Measurement context
– The context is a set of variables describing the state of MA, its connectivity, its environment, etc., while performing a given measurement.
– ISP, mobile operator, signal strength, time, cell-id, IPvx, battery level, GPS coordinates (start and end for mobility), MA application version, ...
Measurement parameters
• A measurement that MA is asked to do.
• Two type of tasks:
– Instructed measurement task : measurement task sent by MC to MA.
– User defined measurement task : Measurement task initiated manually by the user.
Measurement tasks
• A task consists mainly of: – Creation and end dates of the task – A measurement to do – Arguments of the measurement – Metrics to be obtained by the measurement – Constraints
• Constraints : – A measurement task can have a set of constraints on
context parameters. The MA will not perform the task until the context constraints are met.
– e.g.: perform a ping in a given region, at a given time, for a a given type of mobiles, ...
Measurement tasks
Measurement data model
constraint object
{
"context_name":
{
"lower_bound":"value",
"upper_bound":"value",
"values":
[
value1,
...
valueN
]
}
}
task object
{
"task_id": "unique id of the task",
"task_name": "unique name of the task",
"version": "task version",
"description": "description text",
"meas_name": "name of the measurement",
"end_date": "date of task end",
"repeat_interval": "daily or weekly",
"arguments": [
{"arg1_name":"value"},
...
],
"metrics": [
"metric1_name",
...
],
"constraints": [
{constraint_object1},
...
]
}
Measurement data model
meas_result
{
"task_id":"id of meas task"
"success":"OK or reason of
failure":
"arguments":
[
{"arg1_name":"value"},
...
{"argN_name":"value"}
],
"metrics":
[
{"metric1_name":"value"},
...
{"metricN_name":"value"},
],
}
Measurement data model
task_result
{
"meas_result":
{//meas_result object}
"context":
{
//context object.
}
}
• Simple control and reporting protocol
• REST architechture – JSON messages encapsulated in HTTP standard methods
(GET, PUT, POST)
• TLS security using HTTPS
• MA initiates control and reporting communications
• If not anonymous mode, user should authenticate by either using a comiqual account or other third-party identity services (google, facebook, ...) – authentication cookies are used in every message to
ensure the user is authenticated
MA-MC communication
instruct message
{
"ma_id": ma id,
"controller_url" :http://...
"collector_url" :http://...
"keep_tasks_id":[
id1,
...
],
"new_tasks": [
{//task1 object},
...
]
}
PROBER CONTEXTER
MANAGER Controller
Collector
MA MC
CHECKIN message
checkin
{
"ma_id": ma_id,//-1 if first time
"device":{//device object.
//Mandatory when ma_id==-1},
“probeness”: two possible values for now
"context": {//context object. optional},
"current_tasks":
[
{"task_id":value,"task_version":value},
...
{"task_id":value,"task_version":value}
]
}
//+ cookie in the HTTP header
PROBER CONTEXTER
MANAGER Controller
Collector
MA MC
INSTRUCT message
instruct message
{
"ma_id": ma id,
"controller_url" :http://...
"collector_url" :http://...
"keep_tasks_id":[
id1,
...
],
"new_tasks":[
{//task1 object},
...
]
}
//+ cookie in the HTTP header
PROBER CONTEXTER
MANAGER Controller
Collector
MA MC
PROBER CONTEXTER
MANAGER Controller
Collector
MA MC
REPORT message
ma_report
{
"ma_id": "value"
"report":
[
{//task_result object1},
...
{// task_result objectN }
]
}
//+cookie in the HTTP header
• Storage of system and measurement data – Users, measurement parameters, measurement tasks – Raw measurement data
• Flexible database design – Simple creation of new measurement and parameters
(no predefined fields)
Database
meas_key
meas_param_id
meas_param_value
• Measurement results will be publicly available
• JSON will be used as an open data format
• Users will be able to:
– Get the raw open data in JSON format
– Visualize the most important indicators on a map
– Define their own statistics using an online statistics tool we are developing.
Open data
39
Issues and perspectives
• Perform throughput measurement with minimum TCP traffic
• Find incentives to make people contribute
• Converge toward LMAP standard
• No interaction between current platforms – Proprietary and closed systems
– Redundancy of measurements
• Deployment cost : – No modular components ready to use
• Tests and metric are not well defined – Problem comparing results from different
platforms
– Hard to aggregate measurement results
Need for standards
• Architecture : Despite their differences, all platforms have common architectural components
• Measurements: – How to measure a specific metric (related to IPPM) – Scheduling measurement execution – Location of the probe (clarify the exact meaning of
Internet access) – Measurement environment : define and avoid cross
traffic while measuring the network
• Control and report protocols
Possible tracks for standardization
• Large-Scale Measurement of Broadband Performance
• Recent IETF workgroup
• Working on standardizing large-scale measurement systems
• focusing on control and report protocols
• Considers only measurement systems under the control of a single organization (LMAP domain
LMAP
• ^
• |
• +-------------+ IPPM
• +---------------+ Measurement | Measurement | Scope
• | Measurement |<------------>| Peer | |
• | Agent | Traffic +-------------+ v
• +------->| | ^
• | +---------------+ |
• | ^ | |
• | Instruction | | Report |
• | | +-----------------+ |
• | | | |
• | | v LMAP
• | +------------+ +------------+ Scope
• | | Controller | | Collector | |
• | +------------+ +------------+ v
• | ^ ^ | ^
• | | | | |
• | | +----------+ | |
• | | | v |
• +------------+ +----------+ +--------+ +----------+ |
• |Bootstrapper| |Subscriber|--->| data |<---|repository| Out
• +------------+ |parameter | |analysis| +----------+ of
• |database | | tools | Scope
• +----------+ +--------+ |
• |
• v
LMAP components
• Information model as an abstract definition of information exchanged between MA and Controller/Collector
• Guide standardization of one or more control and reporting protocols
• Information model sections : – Configuration
– Instruction
– Reporting results
– Capability of MA
LMAP information model
object {
ma-task-obj ma-instruction-tasks<0..*>;
ma-channel-obj ma-report-channels<0..*>;
ma-schedule-obj ma-instruction-schedules<0..*>;
ma-suppression-obj ma-suppression;
} ma-instruction-obj;
LMAP information model example
object {
string ma-schedule-name;
ma-sched-task-obj ma-schedule-tasks<0..*>;
ma-timing-obj ma-schedule-timing;
} ma-schedule-obj;
LMAP information model example
object { union { ma-periodic-obj ma-timing-periodic; ma-calendar-obj ma-timing-calendar; ma-one-off-obj ma-timing-one-off; ma-immediate-obj ma-timing-immediate; ma-startup-obj ma-timing-startup; } [int ma-timing-random-spread;] // milliseconds } ma-timing-obj;
LMAP information model example
object {
string ma-task-name;
uri ma-task-registry-entry;
string ma-role;
name-value-pair ma-task-options<0..*>;
} ma-task-obj;
LMAP information model example
1. Marcelo Bagnulo, Trevor Burbridge, Sam Crawford, Philip Eardley, Juergen Schoenwaelder, and Brian Trammell. Building a Standard Measurement Platform, IEEE Communications Magazine , May 2014
2. Marcelo Bagnulo, Philip Eardley, Brian Trammell, and Rolf Winter. Standardizing Large-Scale Measurement Platforms, ACM SIGCOMM Computer Communication Review, Volume 43, Number 2, April 2013
3. RIPE Atlas. https://atlas.ripe.net 4. SamKnows. https://www.samknows.com 5. P. Eardley, A. Morton, M. Bagnulo, T. Burbridge, P. Aitken, and A.
Akhter, A framework for large-scale measurement platforms (LMAP), Internet Engineering Task Force, Internet-Draft draft-ietf-lmap-framework-08, Aug. 2014, work in Progress. [Online]. Available: http://tools.ietf.org/html/draft-ietf-lmap-framework-08
Références
THANK YOU