visualizing 4g experience by country, across networks, opensignal

Visualizing 4G Experienceby country, across networks

James Robinson, CTO OpenSignal

#bigdatashow @jamesCRROpenSignal

We collect data.• We’ve built the world’s largest global

database of cellular data• Our data comes via Android and iOS apps

(a sensor network)• SmartUK: UK’s Most Innovative Mobile Co.

And visualize it.

3.5M devices

9.0Mmap tiles

• Market reports for mobile network operators

• Independent coverage maps for consumers

• One-off reports for everybody4G is in focus now.

#bigdatashow @jamesCRROpenSignal

How and why we made these

coverage rollout

speeds

#bigdatashow @jamesCRROpenSignal

opensignal.com/jr/big-data-show

#bigdatashow @jamesCRROpenSignal

The Lifecycle of a Data point

1. Acquisition

2. Storage

3. Visualization

#bigdatashow @jamesCRROpenSignal

AcquisitionAutomated crowd-sourcing: a sensor network

Smartphones are ideal probes of network

performance.

We’re re-purposing sensors.

Consumers come first, data comes second.

#bigdatashow @jamesCRROpenSignal

AcquisitionOther sensor networks

• Google maps • Waze (traffic)

Similar to OpenSignal:• Rootmetrics• Sensorly

Sensor creep Galaxy S (S4 has 3 more!)

#bigdatashow @jamesCRROpenSignal

StorageAutomated crowd-sourcing: a sensor network

Going via MySQL:• Analysis on the

datastream• Quick temperature check

on the app• S3 not ideal for appending

data to files

Relatively easy to bring files from S3 into Hadoop running on EC2

Compressed in app MySQL buffer

Amazon S3

#bigdatashow @jamesCRROpenSignal

Visualization: themes

A varied tool box.Use of open-source &

web technologies.

#bigdatashow @jamesCRROpenSignal

Visualization 1: 4G Coverage

We wanted to create a resource where people could compare

networks for areas that matter to them.

Google Maps was the natural choice.

• Familiar - to designers and consumers • Scalable - and battle tested• Flexible – Bayeux tapestry

#bigdatashow @jamesCRROpenSignal

Visualization 1: 4G CoverageWhat we didn’t want.

• We’re mapping user experience, not modelling cellular propagation

• Coverage feels organic, maps should reflect that

• Everything should be in one place and easily filterable

#bigdatashow @jamesCRROpenSignal

Visualization 1: 4G CoverageHow we do it.

Pull data into HadoopPre-aggregate for different zoom levelsOutput MySQL tables

Generate tiles when needed- When users scroll to an area on the map,

query the server- Check if a tile already exists- Tiles generated in PHP (i.e. on server)- Could move to HTML5 or a javascript

language (D3!) – client based- Store new tiles on server

MTS (network)3G (10 poss)

Zoom 10Novosibirsk (x,y)

#bigdatashow @jamesCRROpenSignal

Visualization 2: 4G RolloutWe wanted to show:• Countries with LTE• When it was

deployed• Planned

deployments• Individual networks

D3 DATA DRIVEN DOCUMENTS

• Created by Mike Bostock of NYT.

• Opensource.• JS & SVG

based.• Engineers

should love it.

& it had to look good

#bigdatashow @jamesCRROpenSignal

Visualization 2: 4G RolloutWhat we could have done.

• Don’t use pins for country level data!• Better & simpler: Google Fusion Tables, or

Google Viz (but no time dimension)• Custom tiles (time dimension but hard to

make interactive)

Pins are OK for cities.

Fusion tables: shallow learning curve, more flexible than you initially think, but less flexible than you’d like:

#bigdatashow @jamesCRROpenSignal

Visualization 2: 4G RolloutHow we do it.

• Countries defined by geojson (various sources available)

• Data on rollout also in json

• The result of a graphic designer/front-end code, working with a data analyst and a copy writer

• One data analyst with knowledge of javascript could get similar results

#bigdatashow @jamesCRROpenSignal

Visualization 3: 4G Speed

• We had 11 countries and 22 networks with good data on 3G speed.

• We could have just put everything in one chart (33 bars) or two charts (11 and 22)

• But it wouldn’t be extensible or so easily navigable.

#bigdatashow @jamesCRROpenSignal

Visualization 3: 4G SpeedHow we do it.

• Use interactivity as a way of hiding data• Give hints that the data can be explored• Re-scaling axes can be confusing

#bigdatashow @jamesCRROpenSignal

Final Thoughts 1D3 powerful for• Transitioning between data sets / visualization

types• Your company already has people who’d love

to use it (they just don’t know it yet)But … it takes more time to set up each visualization than Excel/Tableau/R

When starting to analyse, don’t have one tool or visualization in mind

But know what’s out there

#bigdatashow @jamesCRROpenSignal

Final thoughts 2

You use open-source tools for analysis – why not visualization?Excel could make a comeback – but unlikely to be cutting edgeA visualization is great when everyone can understand it4G rollouts are a very mixed bag

Thank youOpenSignal.com

@jamesCRR