running wikipedia.org - varnishcon 2016 amsterdam...wikimedia foundation i non-pro˝t organization...
TRANSCRIPT
![Page 1: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/1.jpg)
Running Wikipedia.orgVarnishcon 2016 Amsterdam
Emanuele RoccaWikimedia Foundation
June 17th 2016
1
![Page 2: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/2.jpg)
1,000,000 HTTP Requests
1
![Page 3: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/3.jpg)
Outline
I Wikimedia FoundationI Tra�c EngineeringI Upgrading to Varnish 4I Future directions
2
![Page 4: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/4.jpg)
Wikimedia Foundation
I Non-pro�t organization focusing on free,open-content, wiki-based Internet projects
I No ads, no VC moneyI Entirely funded by small donorsI 280 employees (67 SWE, 17 Ops)
3
![Page 5: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/5.jpg)
Alexa Top Websites
Company Revenue Employees Server countGoogle $75 billion 57,100 2,000,000+Facebook $18 billion 12,691 180,000+Baidu $66 billion 46,391 100,000+Yahoo $5 billion 12,500 100,000+Wikimedia $75 million 280 1,000+
4
![Page 6: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/6.jpg)
Tra�c Volume
I Average: ~100k/s, peaks: ~140k/sI Can handle more for huge-scale DDoS attacks
5
![Page 7: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/7.jpg)
DDoS Example
Source: jimieye from �ickr.com (CC BY 2.0)
6
![Page 8: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/8.jpg)
The Wikimedia Family
7
![Page 9: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/9.jpg)
Values
I Deeply rooted in the free culture and freesoftware movements
I Infrastructure built exclusively with free andopen-source components
I Design and build in the open, together withvolunteers
8
![Page 10: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/10.jpg)
Build In The Open
I github.com/wikimediaI gerrit.wikimedia.orgI phabricator.wikimedia.orgI grafana.wikimedia.org
9
![Page 11: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/11.jpg)
Tra�c Engineering
10
![Page 12: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/12.jpg)
Tra�c Engineering
I Geographic DNS routingI Remote PoPsI TLS terminationI Content cachingI Request routing
11
![Page 13: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/13.jpg)
Component-level Overview
I DNS resolution (gdnsd)I Load balancing (LVS)I TLS termination (Nginx)I In-memory cache (Varnish)I On-disk cache (Varnish)
12
![Page 14: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/14.jpg)
Cluster Map
eqiad: Ashburn, Virginia - cp10xxcodfw: Dallas, Texas - cp20xxesams: Amsterdam, Netherlands - cp30xxulsfo: San Francisco, California - cp40xx
13
![Page 15: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/15.jpg)
CDN
I No third-party CDN / cloud providerI Own IP network: AS14907 (US), AS43821 (NL)I Two "primary" data centers
I Ashburn (VA)I Dallas (TX)
I Two caching-only PoPsI AmsterdamI San Francisco
14
![Page 16: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/16.jpg)
CDN
I AutonomyI PrivacyI Risk of censorship
15
![Page 17: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/17.jpg)
CDN
I Full control over caching/purging policyI Lots of functional and performanceoptimizations
I Custom analyticsI Quick VCL hacks in DoS scenarios
16
![Page 18: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/18.jpg)
17
![Page 19: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/19.jpg)
GeoDNS
I 3 authoritative DNS servers running gdnsd +geoip plugin
I GeoIP resolution, users get routed to the"best" DC
I edns-client-subnetI DCs can be disabled through DNScon�guration updates
18
![Page 20: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/20.jpg)
con�g-geo
FR => [ esams , eqiad , codfw , u ls fo ] , # FranceJP => [ ulsfo , codfw , eqiad , esams ] , # Japan
https://github.com/wikimedia/operations-dns/
19
![Page 21: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/21.jpg)
![Page 22: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/22.jpg)
21
![Page 23: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/23.jpg)
LVS
I Nginx servers behind LVSI LVS servers active-passiveI Load-balancing hashing on client IP (TLSsession persistence)
I Direct Routing
22
![Page 24: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/24.jpg)
Pybal
I Real servers are monitored by a softwarecalled Pybal
I Health checks to determine which servers canbe used
I Pool/depool decisionsI Speaks BGP with the routers
I Announces service IPsI Fast failover to backup LVS machine
23
![Page 25: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/25.jpg)
Pybal + etcd
I Nodes pool/weight status de�ned in etcdI confctl: CLI tool to update the state of nodesI Pybal consuming from etcd with HTTP LongPolling
24
![Page 26: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/26.jpg)
25
![Page 27: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/27.jpg)
Nginx + Varnish
I 2x varnishd running on all cache nodesI :80 -smallocI :3128 -spersistent
I Nginx running on all cache nodes for TLStermination
I Requests sent to in-memory varnishd on thesame node
26
![Page 28: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/28.jpg)
27
![Page 29: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/29.jpg)
Persistent Varnish
I Much larger than in-memory cacheI Survives restartsI E�ective in-memory cache size: ~avg(mem size)I E�ective disk cache size: ~sum(disk size)
28
![Page 30: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/30.jpg)
29
![Page 31: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/31.jpg)
Inter-DC tra�c routing
cache : : route_table :eqiad : ’ d i rect ’codfw : ’ eqiad ’u ls fo : ’ codfw ’esams : ’ eqiad ’
30
![Page 32: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/32.jpg)
Inter-DC tra�c routing
I Varnish backends from etcd:directors.vcl.tpl.erb
I puppet template -> golang template -> VCL �le
I IPSec between DCs
31
![Page 33: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/33.jpg)
32
![Page 34: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/34.jpg)
X-Cache
Cache miss:$ cu r l −v https : / / en . wikipedia . org ? test=$RANDOM 2>&1 | grep X−CacheX−Cache : cp1068 miss , cp3040 miss , cp3042 miss
Cache hit:$ cu r l −v https : / / en . wikipedia . org | grep X−CacheX−Cache : cp1066 h i t /3 , cp3043 h i t / 5 , cp3042 h i t /2 138 1
Forcing a speci�c DC:$ cu r l −v https : / / en . wikipedia . org ? test=$RANDOM \
−−resolve en . wik ipedia . org : 4 43 : 208 . 80 . 1 5 3 . 2 24 2>&1 | grep X−CacheX−Cache : cp1066 miss , cp2016 miss , cp2019 miss
33
![Page 35: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/35.jpg)
X-Cache
Cache miss:$ cu r l −v https : / / en . wikipedia . org ? test=$RANDOM 2>&1 | grep X−CacheX−Cache : cp1068 miss , cp3040 miss , cp3042 miss
Cache hit:$ cu r l −v https : / / en . wikipedia . org | grep X−CacheX−Cache : cp1066 h i t /3 , cp3043 h i t / 5 , cp3042 h i t /2 138 1
Forcing a speci�c DC:$ cu r l −v https : / / en . wikipedia . org ? test=$RANDOM \
−−resolve en . wik ipedia . org : 4 43 : 208 . 80 . 1 5 3 . 2 24 2>&1 | grep X−CacheX−Cache : cp1066 miss , cp2016 miss , cp2019 miss
33
![Page 36: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/36.jpg)
X-Cache
Cache miss:$ cu r l −v https : / / en . wikipedia . org ? test=$RANDOM 2>&1 | grep X−CacheX−Cache : cp1068 miss , cp3040 miss , cp3042 miss
Cache hit:$ cu r l −v https : / / en . wikipedia . org | grep X−CacheX−Cache : cp1066 h i t /3 , cp3043 h i t / 5 , cp3042 h i t /2 138 1
Forcing a speci�c DC:$ cu r l −v https : / / en . wikipedia . org ? test=$RANDOM \
−−resolve en . wik ipedia . org : 4 43 : 208 . 80 . 1 5 3 . 2 24 2>&1 | grep X−CacheX−Cache : cp1066 miss , cp2016 miss , cp2019 miss
33
![Page 37: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/37.jpg)
Cache clusters
I Text: primary wiki tra�cI Upload: multimedia tra�c (OpenStack Swift)I Misc: other services (phabricator, gerrit, ...)I Maps: maps.wikimedia.org
34
![Page 38: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/38.jpg)
Terminating layer - text cluster
I Memory cache: 69%I Local disk cache: 13%I Remote disk cache: 4%I Applayer: 14%
35
![Page 39: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/39.jpg)
Terminating layer - upload cluster
I Memory cache: 68%I Local disk cache: 29%I Remote disk cache: 1%I Applayer: 2%
36
![Page 40: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/40.jpg)
Upgrading toVarnish 4
37
![Page 41: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/41.jpg)
Varnish VCL
I Puppet ERB templating on top of VCLI 22 �les, 2605 linesI Shared across:
I clusters (text, upload, ...)I layers (in-mem, on-disk)I tiers (primary, secondary)
I 21 VTC test cases, 715 lines
38
![Page 42: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/42.jpg)
Varnish 3
I 3.0.6-plus with WMF patchesI consistent hashingI VMODs (in-tree!)I bug�xes
I V3 still running on two clusters: text andupload
39
![Page 43: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/43.jpg)
Varnish 4 upgrade
I Bunch of patches forward portedI VMODs now built out-of-treeI VCL code upgradesI Custom python modules reading VSM �lesforward ported
I Varnishkafka
V4 running on two clusters: misc and maps
40
![Page 44: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/44.jpg)
V4 packages
I O�cial Debian packaging:git://anonscm.debian.org/pkg-varnish/pkg-varnish.git
I WMF patches:https://github.com/wikimedia/operations-debs-varnish4/
tree/debian-wmf
I Need to co-exist with v3 packages (main vs.experimental)
I APT pinning
41
![Page 45: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/45.jpg)
VMODs
I vmod-vslp replacing our own chash VMODI vmod-netmapper forward-portedI Packaged vmod-tbf and vmod-header
42
![Page 46: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/46.jpg)
V4 VMOD porting
43
![Page 47: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/47.jpg)
V4 VMOD packaging
I Modi�cations to vmod-tbf to build out-of-treeI Header �les pathI Autotools
I vmod-header was done already, minorpackaging changes
44
![Page 48: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/48.jpg)
VCL code upgrades
I Need to support both v3 and v4 syntax (sharedcode)
I Hiera attribute to distinguish between the twoI ERB variables for straightforwardreplacements
I $req_method→ req.method vs. req.requestI $resp_obj→ resp vs. objI ...
I 42 if @varnish_version4
45
![Page 49: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/49.jpg)
varnishlog.py
I Python callbacks on VSL entries matchingcertain �lters
I Ported to new VSL API using python-varnishapi:https://github.com/xcir/python-varnishapi
I Scripts depending on it also portedI TxRequest→ BereqMethodI RxRequest→ ReqMethodI RxStatus→ BereqStatusI TxStatus→ RespStatus
46
![Page 50: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/50.jpg)
varnishkafka
I AnalyticsI C program reading VSM �les and sending datato kafka
I https://github.com/wikimedia/varnishkafkaI Lots of changes:I 6 �les changed, 612 insertions(+), 847deletions(-)
47
![Page 51: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/51.jpg)
varnishtest
I Started using it after Varnish Summit BerlinI See ./modules/varnish/�les/tests/I Mocked backend (vtc_backend)I Include test version of VCL �lesI VCL code depends heavily on the speci�cserver
48
![Page 52: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/52.jpg)
[ . . . ]
varn ish v1 −arg "−p vcc_err_unref= fa l se " −vc l +backend {backend vtc_backend {
. host = "$ { s1_addr } " ; . port = "$ { s1_port } " ;}
inc lude " / usr / share / varn ish / tests / wikimedia_misc−frontend . v c l " ;} −s ta r t
c l i e n t c1 {txreq −hdr "Host : g i t . wikimedia . org " −hdr "X−Forwarded−Proto : https "rxrespexpect resp . status == 200expect resp . http . X−Cl ient−IP == " 1 2 7 . 0 . 0 . 1 "
txreq −hdr "Host : g i t . wikimedia . org "rxresp# http −> https red i rec t through _synth , we should s t i l l get X−Cl ient−IP# ( same as in _de l i ve r )expect resp . status == 301expect resp . http . X−Cl ient−IP == " 1 2 7 . 0 . 0 . 1 "
} −run
49
![Page 53: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/53.jpg)
Future plans
50
![Page 54: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/54.jpg)
Future plans - TLS
I Outbound TLSI Add support for listening on unix domainsocket
51
![Page 55: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/55.jpg)
Future plans - backends
I Make backend routing more dynamic: eg,bypass layers on pass at the frontend
I etcd-backed director to dynamicallydepool/repool/re-weight
52
![Page 56: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/56.jpg)
Future plans - caching strategies
I Only-If-Cached to probe other cachedatacenters for objects before requestingfrom the applayer
I XKey integration to "tag" di�erent versions ofthe same content and purge them all at once(eg: desktop vs. mobile)
53
![Page 57: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/57.jpg)
Future plans - bloom �lters
Very fast and space-e�cient way to �nd out ifsomething is de�nitely not in the set
I cache-on-second-fetch: avoid caching "rare"items
I 404 �lter with the bloom set representing alllegal URLs to help against randomized URLpaths from botnets
54
![Page 58: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/58.jpg)
Conclusions
I One of the most popular CDNs in the world isbuilt in the open using FOSS
I Multi-layered Varnish setupI Currently upgrading to Varnish 4I Big plans for the future!
55
![Page 59: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/59.jpg)
Cache servers
101 bare-metal serversI 28 AmsterdamI 27 VirginiaI 26 TexasI 20 California
56
![Page 60: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/60.jpg)
edns-client-subnetimport dnsimport c l ientsubnetopt ion
def resolve ( c l i e n t _ i p ) :cso = cl ientsubnetopt ion . ClientSubnetOption ( c l i e n t _ i p )message = dns . message . make_query ( ’ en . wikipedia . org ’ , ’A ’ )message . use_edns ( options =[ cso ] )
# ns0 . wikimedia . orgr = dns . query . udp (message , ’ 208 .80 . 1 54 . 238 ’ )
for a in r . answer :print a
print " United States "resolve ( ’ 1 9 9 . 2 1 7 . 1 1 8 . 4 1 ’ )
print " I t a l y "resolve ( ’ 1 5 1 . 1 . 1 . 1 ’ )
57
![Page 61: Running Wikipedia.org - Varnishcon 2016 Amsterdam...Wikimedia Foundation I Non-pro˝t organization focusing on free, open-content, wiki-based Internet projects I No ads, no VC money](https://reader033.vdocument.in/reader033/viewer/2022051822/5fecf4af7125c703190465ee/html5/thumbnails/61.jpg)
edns-client-subnet
$ python resolve . pyUnited Statesen . wikipedia . org . 600 IN A 208 .80 . 1 53 . 224I t a l yen . wikipedia . org . 600 IN A 9 1 . 1 9 8 . 1 7 4 . 1 9 2
58