PERN Network Analysis, 2010-2011

Prepared by NUST SEECS in collaboration with SLAC, USA

For full report please see: https://confluence.slac.stanford.edu/display/IEPM/Pakistani+Case+Study+2010-2011

Focus of this presentation

• Internet performance monitoring – motivation.

• PERN network performance monitoring - progress.

• PERN network analysis – (2010).• Outstanding issues that require attention.• Conclusion of analysis.

Internet Performance Monitoring - Motivation

World Internet Penetration in 2010

Correlation of Internet Performance and the UNDP Human Development Index

Role of network measurements in minimizing the Digital Divide

• If digital divide is not measured, there is no way we can eliminate it.

• SEECS-NUST in collaboration with SLAC provide:– A driving force to help minimize the digital divide.– Monitoring and tracking of bandwidth (BW) progress.– Raise awareness: locally, regionally and globally.– Technical help with modernizing the infrastructure:

• Provide tools for effective use.• Designing, commissioning and development

• Encourage and work on inter-regional projects:– Asia-Pacific: TEIN2, TEIN3– US-Brazil: RedCLARA

Overview and Introduction to PingER• Funded by HEC in Pakistan since 2008.• Measurements since 1995

– Reports link reliability and quality.– Complete overhaul by NUST researchers in 2003.– Complete update in progress.

• Countries monitored:– Contain 98% of the world population.– 99% of the world’s internet users.

• 930 remote nodes at 786 sites in:– 164 nations; 55 monitoring nodes– 169 nodes in 50 African countries

• Strong collaboration with ICTP/Trieste, Italy and NUST SEECS, Pakistan.– 35 monitoring nodes in Pakistan.

• Excellent, vital work.

Countries: N. America (3), Latin America (21), Europe (30), Balkans (10),

Africa (50), Middle East (13), Central Asia (9), South Asia (8), East Asia (4),

SE Asia (10), Russia (1), China (1) and Oceania (4)

How PingER works?

• Monitoring hosts ping remote hosts with 10 pings every 30 minutes.

• From this data we measure:– minimum and average round trip times (RTT),– Jitter (IPDV),– loss,– Un-reachability (all 10 pings fail)– and derive throughput and mean opinion score (MOS).

• Data gathered from monitoring sites on a daily basis by the archiving sites at NUST, SLAC and FNAL.

PingER-Pakistan deployment

Status of PingER-Pakistan deployment

• 8 nodes till January 2009:– red = monitoring nodes– green = monitored nodes

• RTT as seen from SLAC:

Development and deployment in 2010• Put together PERN network monitoring infrastructure.• Possible because of:

– PingER network administrators training workshops.– Site visits by NUST SEECS team.– Strong collaboration between NUST and SLAC.

• Installed PingER monitoring tools and started gathering data at 35 sites.– Working on an additional 25 monitoring sites.– Monitoring host – remote host pairs increased from 30 to over 500.

• Deployment of 3rd PingER archive site at NUST SEECS.– The other two being at SLAC, USA and FNAL, USA.– Pakistani data archived at NUST only.– World wide data also archived at NUST.– NUST manages the archive repository both at NUST and SLAC.

• Deployment of visualization tools and aids.– Smokeping graphing utility.– Enhancement of PingER coverage maps.

• Archival of traceroutes among all Pakistani universities.

Issues during deployment• Difficulty varies from site to site.• Installation of PingER software has not caused any delays.

– 14+ years of development effort has gone into PingER.– NUST SEECS has been associated with it for 7+ years.

• The delays (from installation to data gathering) have mostly been due to:– getting administrative approval within university– getting access to the concerned local people– delays in making the DNS record entry

• No DNS entry for Lahore School of Economics. Required to enhance PingER tools.

• Problems once it starts taking data are:– poor power availability– lack of backup power

Analysis from PingER-Pakistan data

PERN2 Topology differences

• Regions:• Peshawar• Islamabad• Lahore• Karachi• Quetta

Unreachability

• An unreachable host doesn’t reply to any pings.

• We chose a reliable host at SLAC (pinger.slac.stanford.edu) and analyzed the unreachability of Pakistani hosts.

RTT and packet loss (inter-city)• Pak to Pak RTT analysis.• The minimum RTT to Peshawar

and Quetta (graph at left) appears to have reduced dramatically after April 2010.

• Partially due to bringing on new hosts having lower RTT.

• Also most nodes shifted to the PERN network in April and May 2010.

• Blue dots = median losses between pairs.• Red line = number of pairs.• Packet loss has increased over the last year.• This is due to the shift to PERN network,

which means that it is nearly at maximum utilization.

Intra-city, e.g. Islamabad• Large differences in

minimum RTT.• PERN and NCP

(N.E. Islamabad)– Less than 10msec

(blue line, exactly 1.3msec)

• PERN and NUST (S.W. Islamabad)– 40-80msec (red line,

exactly 44msec)• Presumably due to

the public routing in Islamabad region.

Throughput• We derive the throughput from the loss and RTT measurements as:

– throughput = 1460*8[bits]/(RTT[msec]*sqrt(loss)) kbits/s– Note: this is not actual throughput.

• Throughput has generally increased as number of nodes have increased.

Mean Opinion Score (MOS)• Telecom industry uses MOS as a voice quality

metric.– 1= bad; 2=poor; 3=fair; 4=good; 5=perfect.– Typical range: 3.5 to 4.2– Excellent connection : >4.2

• Trend shows overall improvement (bottom graph) thanks to PERN network.– If new hosts included then performance drops a bit

(left graph).

Measuring Alpha

• The speed of light in fibre is roughly 0.66*c– ‘c’ = speed of light in vacuum i.e. 299,792,458 m/s

• Using 300,000 km/s as ‘c’ this yields:– Alpha = round trip distance[km] / 100[km/ms] * min RTT [ms]

• Alpha is a way to derive distance between two hosts (using minimum RTT).– Large values of alpha close to one indicate a direct path.– Small values usually indicate a very indirect path.

• This assumes no queuing and minimal network device delays.

Alpha• Direct links between (alpha close to

1):– Karachi and Lahore– Karachi and Islamabad– Karachi and Peshawar

• Very indirect link between Islamabad and Quetta (low alpha).– Route goes via Karachi in the south

and then back northwards to Quetta.

• More indirect links (lower alpha):– Islamabad and Lahore– Islamabad and Peshawar– Lahore and Peshawar– Islamabad is a common element

between these links

• Islamabad's intra-city traffic experiences multiple hops (within a few square kms).

• Outbound Islamabad traffic also experiences a slightly indirect route (multiple hops).

• Traffic passing between Peshawar and Lahore shows a much direct route.

UET Taxila Case Study

The peculiar case of UET Taxila• UET Taxila shows unusual behaviour.

– Started monitoring UET Taxila from SEECS since September 2010.• Conclusions from smokeping graph below:

– The monthly average RTTs are typically 100ms.– The min_RTTs are under 10ms.– Jitter/IPDV are typically quite large (> 20ms)– Unreachability is high to UETTAXILA from all over Pakistan.– Losses from SEECS are between 2.5% and 7%, which is high.

UET Taxila - Congestion• Network congestion:

– Smokeping plot shows min RTT < 50ms and very large differences in min and max.

– Region between Nov 18 and Dec 1 shows much variability.• At nights the RTTs are low (since people are asleep).• RTTs increase as load goes up and links getting congested.• Heavy queuing ensues with losses and extended RTTs.

• Could be a last mile problem: Could be a last mile problem: traceroutes reveal larger variation (congestion queuing) delays at last few hops (Taxila routers).

SEECS (Islamabad)

Rawalpindi Exchange

Router at TaxilaUET Taxila

UET Taxila – Indepth traffic analysis• Minimum RTT drops to 35 ms from 60 ms on or about November

10th, 2010.– More confusion, we decided to archive pings from SEECS to UET

Taxila.• Initial ping data analysis from Jan 14th to 27th:

– Heavy network utilization from midnight to 4am in the morning.• Possible data transfer between NCP and UET Taxila (research collaboration).

• Some activity details are below.

Concluding words ..

Achievements• Extensive end-to-end (E2E) PERN network monitoring

infrastructure.– In 2010, grew from 30 monitoring-remote node pairs to over 500

covering most of the major universities in main regions of Pakistan.• Strong collaboration between NUST SEECS and SLAC.

– Exchange visitors (NUST students/RAs) visit SLAC.– NUST manages the PingER project both at SLAC and NUST

SEECS.– Students working on Pinger project invariably get fully funded PhD

opportunities.• Development of new PingER tools.

– Smokeping graphing utility.– MOS and Alpha incorporated.– All the tools have been replaced by NUST SEECS with newer

versions.• Enhancement of existing PingER tools.

Issues needing attention• Very good MOS, VoIP tools such as Skype and PERN conferences should

work well between PERN connected hosts.

• High variability in the reliability (unreachability) of hosts.– Loss of power and power shortage.– An effort needs to be made to understand and improve power reliability and the

provision of backup for several sites.• Delays in installation and start up of monitoring hosts.

– Due to weak local support at some sites.• More work needs to be done to understand why Karachi looks bad.• Low values of alpha suggest that there may be a lot of indirect routing in the

Islamabad region.– Further work with PERN is required to see if this can be remedied.

• PERN network configuration changes with time.– Archiving traceroutes between all Pakistani universities to record topology history.– PERN are encouraged to provide the addresses and locations of the routers and if

possible the rough fibre routes or lengths between sites

Acknowledgements• Dr. Arshad Ali - arshad.ali@seecs.edu.pk, NUST• Dr. Les Cottrell - cottrell@slac.stanford.edu, Stanford University• Dr. Anjum Naveed - anjum.naveed@seecs.edu.pk, NUST• Dr. Adnan Khalid - adnan.khalid@seecs.edu.pk, NUST• Zafar Gilani – zafar@slac.stanford.edu, NUST/Stanford University• Fahad Satti – fahad@slac.stanford.edu, NUST/Stanford University• Muhammad Zeeshan - muhammad.zeeshan@seecs.edu.pk, NUST• Kashif Sattar - 08msitkashifsattar@seecs.edu.pk, NUST• Amber Zeb - 08mscseazeb@seecs.edu.pk, NUST• Sadia Rehman - 08mscsesrehman@seecs.edu.pk, NUST• Ajmal Farooq - ajmal.farooq@seecs.edu.pk, NUST • Imran Ashraf - imran.ashraf@seecs.edu.pk, NUST• SEECS Systems' Administration Team• Pakistani Universities collaborating with us.

– We appreciate this but further assistance/input from HEC is requested.• Umar Kalim - umar.kalim@seecs.edu.pk, NUST/Virginia Tech

Thank you!

For full report please visit: https://confluence.slac.stanford.edu/display/IEPM/Pakistani+Case+Study+2010-2011