offline downloading in china: a comparative study [email protected] oct. 30, 2015 zhenhua...
TRANSCRIPT
![Page 1: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/1.jpg)
1
Offline Downloading in China:
A Comparative Study
[email protected]://www.greenorbs.org/people/lzh/
Oct. 30, 2015
Zhenhua Li Christo Wilson Tianyin Xu
Yao Liu Zhen Lu Yinlong Wang
ACM SIGCOMMACM SIGMETRICS
![Page 2: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/2.jpg)
2
Outline
1 Background2 Problem
3 System4 Workload
5 Performance6 Optimization
![Page 3: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/3.jpg)
3
Internet Access across the World
Internet Penetration
Broadband: > 25 Mbps of Download Bandwidth
Broadband: ≈ 4–10 Mbps, Unstable and Limited
Not only penetration,
but also quality of
connections !
![Page 4: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/4.jpg)
4
Pains of the Developing World
Downloading large files requires high-quality network connections!
DTN - Delay Tolerant Networking ?
![Page 5: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/5.jpg)
5
The Case of Modern China
Promises 46% of China’s population has come online
World-class companies like Tencent, Baidu, Alibaba, and Sina Weibo (Microblog)
![Page 6: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/6.jpg)
6
The Case of Modern China
Challenges
Over 72% of China’s Internet users have low-quality network connections
Low access bandwidth
Unstable/ unreliable connection
ISP barrier (Poor inter-connectivity
between ISPs)
Other reasons
![Page 7: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/7.jpg)
7
“Offline Downloading” in China
InternetUser
2. pre-download
Proxy
3. fetch
1. request
Time file requeststart pre-downloading
finish pre-downloading
Data flow
start fetching
finish fetching
freeto beoffline
An HTTP/FTP/P2P link
![Page 8: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/8.jpg)
8
Typical Implementation (1): Cloud-based
InternetUser
2. pre-download
Proxy
3. fetch
1. request
Caches PBs of files in a datacenter that is within or directly peered with the requesting user’s ISP
TencentXuanfeng
BaiduCloudDisk
Xunlei
![Page 9: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/9.jpg)
9
Typical Implementation (2): Smart AP-based
InternetUser
2. pre-download
Proxy
3. fetch
1. request
Caches data in an embedded or connected storage device, e.g., an SD card, a flash drive, or a disk drive
HiWiFi
MiWiFi
Newifi
![Page 10: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/10.jpg)
10
Great Success in Industry
TencentXuanfeng
BaiduCloudDisk
Xunlei
Over 30M users
Over 150M users
Over 80M users
HiWiFi > 1.5M shipments
MiWiFi> 2M shipments
Newifi> 0.6M shipments
![Page 11: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/11.jpg)
11
2 Problem
![Page 12: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/12.jpg)
12
The 1st Problem
Is offline downloading really effective in most cases?Internet
User
2. pre-download
Proxy
3. fetch
1. request
High success rate?
High speed?
![Page 13: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/13.jpg)
13
The 2nd Problem
Which offline downloading approach should be selected?
OR OR Both?
![Page 14: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/14.jpg)
14
The 3rd Problem
When is offline downloading useless or even worse?Internet
User
2. pre-download
Proxy
3. fetch
1. request
User-side access
bandwidth
Cloud-side service
capabilityISP barrier Transfer
protocolFile
popularityHardware & filesystem
![Page 15: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/15.jpg)
15
General Problem: Selection Dilemma
Common Downloading or Offline
Downloading?
Cloud-based or Smart AP? And which smart
AP?
Our work is the first quantitative and comparative study on these
problems based on a large-scale dataset from Xuanfeng cloud and benchmark experiments of popular
smart APs.
![Page 16: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/16.jpg)
16
3 System
![Page 17: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/17.jpg)
17
Xuanfeng Cloud
ISPs
DB
Pre-downloading servers
files
Storage servers
Uploading servers
requests
Internet
http://xf.qq.com
http://lixian.qq.com/main.html
2-PB Collaborative Cache
Privileged Network Path
![Page 18: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/18.jpg)
18
Smart APs
OpenWrt operating system
CPU RAMNIC(xDSL)
NIC(WiFi)
SATAInterface
USBInterface
SDInterface
Opkg applications
UserDevice
Internet
Shell≈ $20
≈ $100
≈ $20
![Page 19: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/19.jpg)
19
4 Workload
![Page 20: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/20.jpg)
20
Xuanfeng Dataset
User Requests
• User ID• IP address• Access bandwidth• Request time• File type• File size• Original data
source• Transfer protocol
Pre-downloading Trace
• Start time• Finish time• Acquired file size• Traffic usage• Cloud cache hit• Avg. speed• Peak speed• Success or failure
Fetching Trace
• User ID• IP address• Access bandwidth• Start time• Finish/pause time• Acquired file size• Traffic usage• Avg. speed• Peak speed
Complete running logs during a whole week in 2015, involving 4M tasks, 0.78M users & 0.56M unique files
![Page 21: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/21.jpg)
21
File Type, Size & Transfer Protocol
75%
15%
10%
File Type
Video SoftwareOther BitTorrent eMule HTTP/FTP
0
10
20
30
40
50
60
70
80
68
1913
Transfer Protocol
Median: 115 MBAverage: 350 MBMaximum: 4 GB
25% files < 8 MB
![Page 22: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/22.jpg)
22
File Popularity
Zipf ≈ Power law
SE ≈ Stretched Exponential
Matthew effect (for non-videos) + Fetch-at-most-once effect (for videos)
![Page 23: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/23.jpg)
23
Smart APs: Benchmark
ADSLLink
HiWiFi
MiWiFi
Newifi
Storage server
Performancedata
Internet
*Note: We assume that the smart AP based offline downloading systems have similar workload characteristics to Xuanfeng, since most end users are not familiar with the technical details and cannot differentiate these services.
Sampled workload from the Tencent Xuanfeng dataset
![Page 24: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/24.jpg)
24
5 Performance
![Page 25: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/25.jpg)
25
Xuanfeng: Pre-downloading Speed and Fetching Speed
Median: 25 KBpsAverage: 69 KBpsMedian: 287 KBpsAverage: 504 KBps
Owing to the privileged network path, Xuanfeng significantly improves users’ perceived downloading speeds by 7 – 11 times (fetching speed / pre-downloading speed)
![Page 26: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/26.jpg)
26
Xuanfeng: Unsatisfactory Fetching Speed
128% of fetching speeds are below 125 KBps (= 1 Mbps, typical playback bitrate of HD videos)
9.6%ISP barrier
10.8%Low user-side access bandwidth
1.5%Lack of cloud-side upload bandwidth
6.1%Unknown...
The cloud-based approach performs poorly once there is a bandwidth bottleneck in the privileged network path between the cloud and the user
☜
![Page 27: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/27.jpg)
27
Xuanfeng: Shortage of Cloud Bandwidth
☜1.5%Lack of cloud-side upload bandwidth
0.84% of highly popular files account for 39% of all downloads
87% of requested files are hosted in peer-to-peer (P2P) data swarms
2The cloud is threatened by running out of upload bandwidth due to unnecessarily sending highly popular P2P files. As the user base continues to grow, the cloud will have to reject more (>1.5%) fetching requests.
![Page 28: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/28.jpg)
28
DB
filesrequests
Xuanfeng: Pre-downloading Failure2-PB Collaborative Caching 8.7% Failure
16.4% FailureX
The cloud cache effectively avoids nearly half of pre-downloading failures
High popularity ≈ Low failure ratio
![Page 29: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/29.jpg)
29
Smart APs: Pre-downloading Failure
Failure Ratio
Xuanfeng Cloud
Smart APs
Overall 8.7% 16.8%Unpopular
files 13% 42%
36% of offline downloading requests are issued for unpopular files
86% Insufficient seeds in a peer swarm
10% Poor HTTP/FTP connections
4% Unknown...
3Smart APs frequently fail during pre-downloading unpopular files
![Page 30: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/30.jpg)
30
Smart APs: Pre-downloading Speed
Speed Xuanfeng Cloud Smart APs
Median 25 KBps 27 KBpsAverage 69 KBps 64 KBps>
<
?
4A smart AP’s pre-downloading speed can be restricted by its hardware and/or filesystem, since some types of storage devices and filesystems do not fit the pattern of frequent, small data writes during pre-downloading
![Page 31: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/31.jpg)
31
Smart APs: Pre-downloading Speed
NTFS is incompatible with the OpenWrt OS
USB flash drive is unsuitable for frequent, small data writes
![Page 32: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/32.jpg)
32
Performance Summary
The two approaches are subject to distinct performance bottlenecks while also being complementary to each other
Xuanfeng Cloud Smart APs
Bottleneck 1: Unsatisfactory fetching speed
Merit 3: Stably high fetching speed
Bottleneck 2: Shortage of cloud bandwidth Merit 4: No cloud infrastructure
Merit 1: Effective avoidance of pre-downloading failures
Bottleneck 3: Frequent failures during pre-downloading
Merit 2: No hardware cost at the user side
Bottleneck 4: Hardware/filesystem
restrictions on pre-downloading
![Page 33: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/33.jpg)
33
6 Optimization
![Page 34: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/34.jpg)
34
ODR Middleware Help users automatically select a proper
(offline) downloading way
User device
1. Request
ODR(Offline Downloading Redirector)
DB
Smart AP
Xuanfeng
Cloud3. Redirect
2. Query
Storage device
Primary goal: minimizing the downloading time and failure ratio
Secondary goal: minimizing the upload bandwidth burden on the cloud
![Page 36: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/36.jpg)
36
ODR Design & Performance
Highly popular file ?
Y
N
Cloud cache hit ?
Y
NCloud pre-download succeed ?
Y
N
Bottleneck 4 ?Y
N
Bottleneck 1 ?Y
NCloud
User device
Smart AP
failData source: Cloud
Data source: Original
Bottleneck 2
Bottleneck 3
Cloud pre-download
P2P file ?
Y
N
HTTP/FTP fileCloud +
Smart AP
Even using this coarse-grained design, we manage to remarkably reduce the current performance bottlenecks of offline downloading !
Heuristic rules extracted from measurement findings
![Page 37: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/37.jpg)
SummaryWhat Offline Downloading is and Why it is popular
in China
Selection Dilemma of Offline Downloading
Measurement findings of Xuanfeng and popular smart APs
◆The two approaches are subject to distinct performance bottlenecks, while also being complementary to each other.
We feel that offline downloading has broad applicability to other areas of the world that lack broadband penetration. By deploying offline downloading technologies, coupled with our proposed ODR middleware, the Internet experiences for users in many parts of the world can be improved.
![Page 38: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/38.jpg)
Thank you! Any questions?
Xuanfeng is pronounced as[ʃuæn fəŋ] or simply XF
BaiduCloudDisk
Xunlei is pronounced as[ʃunlei] or simply XL
HiWiFi
MiWiFi
Newifi
![Page 39: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/39.jpg)
Backup slides
![Page 40: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/40.jpg)
Xunlei, Xuanfeng, and Baidu CloudDisk
TencentXuanfeng
BaiduCloudDisk
Xunlei
Over 30M users
Over 150M users
Over 80M users
Launched in 2009
Launched in 2010
Launched in 2012
![Page 41: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/41.jpg)
Xunlei, Xuanfeng, and Baidu CloudDisk
TencentXuanfeng
BaiduCloudDisk
Xunlei
Over 30M users
Over 150M users
Over 80M users
Charges each user nearly $2 per month
Conditionally free --- should be the VIP user of Tencent
Totally free --- the user even does not need to register
![Page 42: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/42.jpg)
Offline Downloading outside China Besides those developing countries,
developed countries can also benefit from offline downloading (based) services.
SMART WI-FI ROUTERS
USB storage interface
![Page 43: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/43.jpg)
State-of-Art Downloading Techniques CDN typically only help to deliver files
for content providers who pay for the service.
The business model of offline downloading is the opposite of CDN, because it charges (or sometimes frees) its users, i.e., content receivers, for better downloading experiences.
Offline Downloading ≈
DTN + reverse-CDN
ICNCCNNDN
1) in-network storage for caching2) decoupling content senders and receivers3) disruption tolerance4) multi-party communication through replication5) mobility and multi-homing
![Page 44: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/44.jpg)
Performance: Delay
Median: 82 minutesAverage: 370 minutes
Median: 7 minutesAverage: 27 minutes
Xuanfeng reduces users’ perceived downloading delay by 12 – 14 times
![Page 45: Offline Downloading in China: A Comparative Study lizhenhua1983@gmail.com Oct. 30, 2015 Zhenhua LiChristo Wilson Tianyin](https://reader035.vdocument.in/reader035/viewer/2022062409/5697bfc61a28abf838ca7613/html5/thumbnails/45.jpg)
Performance: End-to-End
Median: 10 minutesAverage: 68 minutes
Median: 233 KBpsAverage: 380 KBps
The collaborative cache of Xuanfeng remarkably avoids 89% pre-downloads