Improving Site Response TimePart 3: Varnish Configuration Step by Step

1 19.2.2012Kim Stefan Lindholm

Instructions are for Amazon Linux AMI (64-bit) and compatible systems.

2

3

Varnish Configuration - RequirementsVarnish Configuration - Requirements

1 All traffic to origin server is routed through Incapsula firewall

2 If origin server or Incapsula proxy is down, Varnish serves a cached copy for 6 hours

3 Varnish restarts automatically upon critical failure or server reboot and notifies the administrator by e-mail

4 Varnish hit rate, CPU load and memory usage can be monitored at specific URL

CACHE SIZE

4

Most files are served from CDN, so dataset should be small

VCL file: Normalization of hostname, gzip/deflate, etc.

VCL file: No caching for logged in users / administrator back-end

VCL file: Only one version of page is cached regardless of cookies

3 variants of style sheets: standard, IE8 or later, ≤IE7

ESTIMATING SIZE

• Test site has 27 pages, sized 2.5 - 5.3 kB according to Firefox

• We didn’t optimize for IE7, thus page sizes are 88 - 129 kB

• For a very static site, let’s say we’ll reach a maximum of 100 pages

• Firefox, Chrome, Opera & Safari: 100 x 5.5 kB = 0.6 MB

• IE 8 or later : 100 x 120 kB = 12 MB

• IE7 & IE6: 100 x 130 kB = 13 MB5

ESTIMATING SIZE

• Depending on the amount of unique (= browser x encoding etc.) cached pages needed, entire real dataset will probably be 10-50 MB in size

• Amazon EC2 micro instance has 613 MB of RAM and running “vmstat” or “free -m” shows that ~170 MB of that is available

• As a rule of thumb (which doesn’t always work) you can allocate 80 % of free memory to Varnish cache

• We’ll allocate 130 MB and hot dataset should still be a fraction of that

• Had we not pushed 250 MB of files to CDN, we’d need at least 1 GB of RAM

6

Installing Varnish

7

8

•Our VCL files and tools use syntax of Varnish 3.0 and don’t work with older versions

•Building Varnish from source:

•Copy VCL configuration file to /etc/varnish and try invoking Varnish: “sudo /usr/local/sbin/varnishd -V”

sudo su -yum install -y gcc make pkgconfig pcre-devel ncurses-develcd /usr/srcwget http://repo.varnish-cache.org/source/varnish-3.0.2.tar.gz -O - | tar xzcd varnish-3.0.2./configuremake && make installexit

•Running with 130 MB of memory (remove line breaks):

•Only if out of memory, try with disk:

•Stopping Varnish:

9

sudo /usr/local/sbin/varnishd -s malloc,130M -f /etc/varnish/<your_config>.vcl -T 127.0.0.1:2000 -a 0.0.0.0:80

sudo /usr/local/sbin/varnishd -s file,/<path>/<file>,3G -f /etc/varnish/<your_config>.vcl -T 127.0.0.1:2000 -a 0.0.0.0:80

sudo pkill varnishd

•Useful commands for monitoring Varnish:

•Purging main page / all pages from cache:

•Further performance tuning:

10

varnishstatvarnishhistvarnishlog

varnishadm -T localhost:2000 ban.url "^/$"varnishadm -T localhost:2000 ban.url "^/.*"

varnishtopvarnishsizes

sudo /usr/local/sbin/varnishd -s malloc,130M -u nobody -g nobody -p cli_timeout=30 -p thread_pool_add_delay=2 -p thread_pool_min=400 -p thread_pool_max=4000 -p session_linger=100 -f /etc/varnish/<your_config>.vcl -T 127.0.0.1:2000 -a 0.0.0.0:80

Installing Security.VCL

11

12

•Varnish makes a nice first line of defense against web attacks

•Security.VCL is a web application firewall similar to Apache mod_security but faster

•Edit your VCL file and add this line near the top:

# If not yet installed: sudo yum install -y makewget https://github.com/comotion/security.vcl/tarball/master -O - | tar xzcd <comotion-security-dir>/vcl/sudo makecd ..sudo ln -s $PWD/vcl/ /etc/varnish/security

include "/etc/varnish/security/main.vcl";

•Edit file vcl/config.vcl and comment out some rules:

•Finally, reload your Varnish configuration and test the firewall. Visiting these URLs must return “Error 403 Naughty, not nice!”

•example.com/exploit/foo/bar :bla •example.com/index.old•example.com/SELECT FROM•example.com/javascript:

13

#include "/etc/varnish/security/modules/robots.vcl";#include "/etc/varnish/security/modules/cloak.vcl";

Installing New Relic

14

15

•New Relic allows tracking request queue times which is a relevant metric when load testing Varnish

•Sign up for a free account at http://newrelic.com/, subscribe to a weekly performance summary and write down your license key

sudo rpm -Uvh http://yum.newrelic.com/pub/newrelic/el5/x86_64/newrelic-repo-5-3.noarch.rpmsudo yum install -y newrelic-sysmondsudo nrsysmond-config --set license_key=<your_license_key>sudo /etc/init.d/newrelic-sysmond start

•Create file /etc/varnish/newrelic.h

•Add the following to your VCL file, inside vcl_recv:

16

C{#include </etc/varnish/newrelic.h>}C

/* * Add X-Request-Start header so we can track queue times in New Relic RPM */

#include <stdio.h>#include <sys/time.h>

struct timeval detail_time;gettimeofday(&detail_time, NULL);char start[20]; sprintf(start, "t=%lu%06lu", detail_time.tv_sec, detail_time.tv_usec);VRT_SetHdr(sp, HDR_REQ, "\020X-Request-Start:", start, vrt_magic_string_end);

After a while, request queuing parameter should appear in New Relic RPM:

17

Installing Munin

18

19

• Install Munin and Varnish plugins:

•Command “munin-node-configure” lists installed plugins

sudo su -yum install -y munin-node munincd /usr/share/munin/plugins/wget https://raw.github.com/munin-monitoring/contrib/master/plugins/ varnish/varnish_allocatedwget https://raw.github.com/munin-monitoring/contrib/master/plugins/ varnish/varnish_cachehitratiowget https://raw.github.com/munin-monitoring/contrib/master/plugins/ varnish/varnish_healthy_backendswget https://raw.github.com/munin-monitoring/contrib/master/plugins/ varnish/varnish_hitratewget https://raw.github.com/munin-monitoring/contrib/master/plugins/ varnish/varnish_total_objectschmod a+x /usr/share/munin/plugins/varnish_*ln -s /usr/share/munin/plugins/varnish_* /etc/munin/plugins/exit

•Add to /etc/munin/plugin-conf.d/munin-node

•Edit e-mail settings in /etc/munin/munin.conf

•Start Munin: “sudo service munin-node start”

20

[varnish*]user root

# Uncomment to set network traffic warning at 400K# [if_*]# env.warning 400000

contact.me.command mail -s "Munin notification" admin@example.comcontact.me.always_send warning critical

Munin notifications can be tested by setting a low threshold for traffic:

21

Cloud platforms allow setting useful alerts as well, here’s Amazon CloudWatch:

22

...or sign up for a free RevealCloud account at https://app.copperegg.com/signup/free

23

Weekly E-mail from Munin

24

25

•Create file /etc/varnish/email_varnish_reports.sh

#!/bin/bash# Send Munin generated Varnish statistics by e-mail

VARNISH_LOCATION="Tokyo"REPORT_PATH=/var/www/html/munin/localhost/localhostEMAIL_RECIPIENT="admin@example.com"EMAIL_SUBJECT="Varnish Weekly Statistics"EMAIL_BODY="Weekly statistics attached."

hash mutt 2>&- || { echo -e >&2 "\nMutt not installed, aborting.\n"; exit 1; }

echo $EMAIL_BODY | mutt -s "$EMAIL_SUBJECT ($VARNISH_LOCATION)" \ -a $REPORT_PATH/varnish_cachehitratio-week.png \ -a $REPORT_PATH/varnish_hitrate-week.png \ -a $REPORT_PATH/varnish_total_objects-week.png \ -a $REPORT_PATH/varnish_allocated-week.png \ -a $REPORT_PATH/df-week.png \ -a $REPORT_PATH/threads-week.png \ -a $REPORT_PATH/cpu-week.png \ -a $REPORT_PATH/memory-week.png \ -- $EMAIL_RECIPIENT

•Edit file /etc/crontab to send a report every Monday

•Make sure Mutt is installed and restart cron daemon

26

MAILTO=admin@example.com

00 08 * * Mon root /etc/varnish/email_varnish_reports.sh

sudo yum install -y muttsudo service crond restart

Limited Browser Access to Munin Graphs

27

28

• Install lighttpd

•Edit file /etc/lighttpd/lighttpd.conf

•Start lighttpd

sudo yum install -y lighttpd

server.port = 8081server.document-root = server_root + "/html"

$HTTP["remoteip"] !~ "127.0.0.1" { url.access-deny = ( "" )}

sudo service lighttpd start

•Now you can let Varnish control access to Munin graphs. Benefit: maintain ACLs in one configuration file only.

•Edit your VCL file and add this line near the top:

•Add the following in the beginning of vcl_recv:

29

backend monitoring { .host = "127.0.0.1"; .port = "8081"; }

if (req.url ~ "^/munin" && client.ip ~ internal && (req.url ~ "\?your-secret-token" || req.http.referer ~ "(www\.)?example\.com")) { set req.backend = monitoring; return (pipe);}

Automatic Restarting

30

31

• Install daemontools and create Varnish service directory

sudo su -mkdir -p /packagecd /packagewget http://cr.yp.to/daemontools/daemontools-0.76.tar.gztar zxpf daemontools-0.76.tar.gzrm -f daemontools-0.76.tar.gzcd admin/daemontools-0.76sed -i '/extern int errno/{s/^/\/* /;s/$/ *\//;G;s/$/#include <errno.h>/;}' src/error.hpackage/installmkdir /var/servicemkdir -m 1755 /var/service/varnish

•Stop Varnish in case it’s running and create an executable script /var/service/varnish/run:

•Note that the script may get called multiple times during reboot, thus sending several e-mails

32

#!/bin/sh# Daemontools run script for starting Varnish

exec 2>&1exec echo | mail -s "Varnish in Tokyo restarting" admin@example.comexec varnishd -F -s malloc,130M -u nobody -g nobody -p cli_timeout=30 \ -p thread_pool_add_delay=2 -p thread_pool_min=400 -p thread_pool_max=4000 \ -p session_linger=100 -f /etc/varnish/varnish.tokyo.vcl -T 127.0.0.1:2000 \ -a 0.0.0.0:80

•Create a log script and add symbolic link:

•Confirm that the services are running:

• If daemontools is not running, type "sudo /command/svscanboot &". Varnish is stopped by typing "svc -d /service/varnish" and started with "svc -u /service/varnish".

33

mkdir -m 755 /var/service/varnish/logcd /var/service/varnish/logwget http://qmail.jms1.net/scripts/service-any-log-runmv service-any-log-run runchmod 755 runln -s /var/service/varnish /service/varnish

svstat /service/varnish /service/varnish/log

•Reboot the system to check that everything works fine. You might have to take two more steps. Comment out this line from file /etc/inittab:

•Create file /etc/init/svscan.conf:

•Add similar scripts /service/<your-service>/run for all services you need to manage, e.g. Munin and lighttpd.

34

#SV:12345:respawn:/command/svscanboot

start on runlevel [12345]stop on runlevel [^12345]respawnexec /command/svscanboot

35

•Finally, you can create a swap file in case Varnish needs it:

#!/bin/bash# Create swapfile if not already present. Default size is 2 GB.

if [ ${SWAP_SIZE_MEGABYTES:=2048} -eq 0 ];then echo No swap size given, skipping.else if [ -e /swapfile ];then echo /swapfile already exists, skipping. else echo Creating /swapfile of $SWAP_SIZE_MEGABYTES MB dd if=/dev/zero of=/swapfile bs=1024 count=$(($SWAP_SIZE_MEGABYTES*1024)) mkswap /swapfile fi swapon /swapfile echo Swap Status: swapon -sfi

DOWNLOADS

36

• varnish.tokyo.vcl (VCL example) - https://gist.github.com/1754248

• newrelic.h - https://gist.github.com/1817420

• email_varnish_reports.sh - https://gist.github.com/1817400

• run (daemontools) - https://gist.github.com/1818886

• create_swapfile.sh - https://gist.github.com/1817411

• daemontools log script - http://qmail.jms1.net/scripts/service-any-log-run

• Other code snippets of this presentation - https://gist.github.com/1819143

VCL Configuration File

37

# VCL configuration file for Varnish

# Define which IP addresses or hosts have access to files that are# blocked from the public internetacl internal { "localhost";}

# Define origin serversbackend web { .host = "1.2.3.4"; .port = "80"; }backend web_ssl { .host = "1.2.3.4"; .port = "443"; }

# Uncomment to support Munin graphs# backend monitoring { .host = "127.0.0.1"; .port = "8081"; }

# Uncomment to include Security.VCL module# @see: https://github.com/comotion/security.vcl# include "/etc/varnish/security/main.vcl";

# Respond to incoming requestssub vcl_recv { # Uncomment to support Munin graphs. Access is granted if visitor # is coming from a whitelisted IP address and secret token is # provided. # e.g. http://www.example.com/munin?your-secret-token # if (req.url ~ "^/munin" && client.ip ~ internal # && (req.url ~ "\?your-secret-token" # || req.http.referer ~ "(www\.)?example\.com")) { # set req.backend = monitoring; # return (pipe) ; # } # Uncomment to have New Relic track queue times # C{ # #include </etc/varnish/newrelic.h> # }C

Page 1 # Handle HTTPS connection if (server.port == 443) { set req.backend = web_ssl; } else { set req.backend = web; } if (req.restarts == 0) { if (req.http.x-forwarded-for) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; } else { set req.http.X-Forwarded-For = client.ip; } } # Normalize requests sent via curls -X mode and LWP if (req.url ~ "^http://") { set req.url = regsub(req.url, "http://[^/]*", ""); } # Normalize hostname to avoid double caching set req.http.host = regsub(req.http.host, "^example\.com$", "www.example.com"); # Uncomment to support shared hosting when testing through staging # server # set req.http.host = regsub(req.http.host, "^cache\.example\.com$", # "www.example.com");

# Use anonymous, cached pages if all backends are down if (!req.backend.healthy) { unset req.http.Cookie; }

Page 2

38

your origin server here

# Allow the backend to serve up stale content if it is # responding slowly set req.grace = 6h; # Do not cache these paths if (req.url ~ "^/status\.php$" || req.url ~ "^/administrator") { return (pass); } # Do not cache authenticated sessions if (req.http.Cookie && req.http.Cookie ~ "authtoken=") { return (pipe); } # Do not allow outside access to configuration.php if (req.url ~ "^/configuration\.php$" && !client.ip ~ internal) { # Have Varnish throw the error directly # error 404 "Page not found.";

# Use a custom error page set req.url = "/"; } # Allow purge only from internal users if (req.request == "PURGE") { if (!client.ip ~ internal) { error 405 "Not allowed."; } return (lookup); }

Page 3 # Handle compression correctly. Different browsers send # different "Accept-Encoding" headers, even though they # mostly all support the same compression mechanisms. By # consolidating these compression headers into a consistent # format, we can reduce the size of the cache and get more hits. # @see: http:// varnish.projects.linpro.no/wiki/FAQ/Compression if (req.http.Accept-Encoding) { if (req.http.Accept-Encoding ~ "gzip") { # If the browser supports it, we'll use gzip. set req.http.Accept-Encoding = "gzip"; } else if (req.http.Accept-Encoding ~ "deflate") { # Next, try deflate if it is supported. set req.http.Accept-Encoding = "deflate"; } else { # Unknown algorithm. Remove it and send unencoded. unset req.http.Accept-Encoding; } } # Always cache the following file types for all users if (req.url ~ "(?i)\.(png|gif|jpeg|jpg|ico|swf|pdf|txt|css|js|html|htm|gz|xml) (\?[a-z0-9]+)?$") { unset req.http.Cookie; } if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { /* Non-RFC2616 or CONNECT which is weird. */ return (pipe); }

Page 4

39

if (req.request != "GET" && req.request != "HEAD") { return (pass); }

# We cache requests with cookies too (e.g. Google Analytics) # Original: if (req.http.Authenticate || req.http.Authorization # || req.http.Cookie) { if (req.http.Authenticate || req.http.Authorization) { return (pass); } return (lookup);} # sub vcl_pipe {# # Note that only the first request to the backend will have# # X-Forwarded-For set. If you use X-Forwarded-For and want to# # have it set for all requests, make sure to have:# # set bereq.http.connection = "close";# # here. It is not set by default as it might break some# # broken web applications, like IIS with NTLM authentication.# return (pipe);# }

# sub vcl_pass {# return (pass);# }

# Determine the cache key when storing/retrieving a cached pagesub vcl_hash { hash_data(req.url); if (req.http.host) { hash_data(req.http.host); } else { hash_data(server.ip); }

Page 5 # Don't include cookie in hash # if (req.http.Cookie) { # hash_data(req.http.Cookie); # } return (hash);}

sub vcl_hit { if (req.request == "PURGE") { purge; error 200 "Purged."; } if (obj.ttl <= 0s) { return (pass); } return (deliver);}

sub vcl_miss { if (req.request == "PURGE") { error 404 "Not in cache."; } return (fetch);}

# Called when the requested object has been retrieved from the backend,# or the request to the backend has failed; "beresp" stands for# back-end responsesub vcl_fetch {

Page 6

40

# Don't allow static files to set cookies if (req.url ~ "(?i)\.(png|gif|jpeg|jpg|ico|swf|pdf|txt|css|js|html|htm|gz|xml) (\?[a-z0-9]+)?$") { unset beresp.http.Set-cookie; } # Allow items to be stale if needed set beresp.grace = 6h; if (beresp.ttl <= 0s) { set beresp.http.X-Cacheable = "NO:Not Cacheable"; return (hit_for_pass); } else if (req.http.Cookie ~"(UserID|_session)") { # Don't cache content for logged in users set beresp.http.X-Cacheable = "NO:Got Session"; return (hit_for_pass); } else if (beresp.http.Cache-Control ~ "private") { # Respect the Cache-Control=private header from the backend set beresp.http.X-Cacheable = "NO:Cache-Control=private"; return (hit_for_pass); } else if (beresp.ttl < 1s) { # Extend the lifetime of the object artificially set beresp.ttl = 300s; set beresp.grace = 300s; set beresp.http.X-Cacheable = "YES:Forced"; } else { # Varnish determined the object was cacheable set beresp.http.X-Cacheable = "YES"; # Uncomment to have Varnish cache objects longer than the clients # do. Cache must be purged manually when the site changes, so don't # use with frequently changing content - comments, visitor counters # etc. # @see: https://www.varnish-cache.org/trac/wiki/ VCLExampleLongerCaching

Page 7 # unset beresp.http.expires; # set beresp.ttl = 1w; # set beresp.http.magicmarker = "1"; } return (deliver);}

sub vcl_deliver { # Uncomment to add hostname to headers # set resp.http.X-Served-By = server.hostname;

# Identify which Varnish handled the request if (obj.hits > 0) { set resp.http.X-Cache = "HIT from Tokyo"; set resp.http.X-Cache-Hits = obj.hits; } else { set resp.http.X-Cache = "MISS from Tokyo"; } # Remove version number sometimes set by CMS if (resp.http.X-Content-Encoded-By) { unset resp.http.X-Content-Encoded-By; } if (resp.http.magicmarker) { # Remove the magic marker, see vcl_fetch unset resp.http.magicmarker; # By definition we have a fresh object set resp.http.Age = "0"; }

return (deliver);}

Page 8

41

your location here

sub vcl_error { # Redirect to some other URL in case of root page failure # if (req.url ~ "^/?$") { # set obj.status = 302; # set obj.http.Location = "http://backup.example.com/"; # }

# Otherwise redirect to root, which will likely be in the cache set obj.http.Content-Type = "text/html; charset=utf-8"; synthetic {"<html><head> <title>Page Unavailable</title> <style> body { background: #efefef; text-align: center; color: white; font-family: Trebuchet MS, sans-serif; } #page { width: 500px; margin: 100px auto 0; padding: 30px; background: #888888; border-radius: 14px; -moz-border-radius: 14px; -webkit-border-radius: 14px; border: 0 } a, a:link, a:visited { color: #cccccc; } .error { color: #222222; } </style></head><body onload="setTimeout(function() { window.location = '/' }, 3000)"> <div id="page"> <h1 class="title">Page Unavailable</h1> <p>The page you requested is temporarily unavailable.</p> <p>We're redirecting you to the <a href="/">homepage</a> in 3 seconds.</p> <div class="error">(Error "} + obj.status + " " + obj.response + {")</div> </div></body></html>

Page 9"}; return (deliver);}

Page 10

42

ONE MORE THING...

Incapsula need your A record and CNAME record (www) to point to their servers. This is obviously not the case when you send visitors to Varnish instead.

If your DNS settings don’t match Incapsula’s instructions, you’ll see an error message in control panel and the service might be disabled - not sure about the latter.

Quick fix is to always send visitors from Ireland and Israel to the default address as these are the locations of Incapsula Site Helper bot. If this seems too hackish, more elegant solutions probably exist.

43

GEODNS SETTINGS

44

Record Area Data

example.com Europe, Africa, Global <Varnish Ireland IP>

example.com Americas, Global <Varnish California IP>

example.com Asia, Australia, Global <Varnish Tokyo IP>

example.com Ireland, Israel, Global <Incapsula IP>

www.example.com Global example.com

www.example.com Ireland, Israel <Incapsula CNAME>

Matches are made from smallest to largest qualifying records, so Ireland takes precedence over Europe which in turn precedes global record. Geo-targeting is never 100% accurate.

DNS FAILOVER

45

• Wondered why area Global was set for so many records?

• This means that when one edge server is down, requests will be balanced to all remaining servers marked as global

• As a result, potential DDoS attack will have to take down 4 destinations instead of one. An alternative would be having only Incapsula as backup and keeping other Varnish boxes (and visitors) oblivious to regional attacks.

• When a failed edge server is back up again, it will start receiving requests as usual

RESOURCES - VARNISH

• https://www.varnish-cache.org/trac/wiki/VCLExamples

• https://www.varnish-cache.org/docs/trunk/installation/upgrade.html

• http://nwlinux.com/varnish-caching-proxy-configuration-resources/

• http://nwlinux.com/mitigating-ddos-attacks-with-varnish-proxy/

• https://github.com/comotion/security.vcl

46

RESOURCES - MONITORING

• http://newrelic.com/docs/server/server-monitor-installation-redhat-and-centos

• http://munin-monitoring.org/

• http://engineering.gomiso.com/2011/01/04/easy-monitoring-of-varnish-with-munin/

• http://waste.mandragor.org/munin_tutorial/munin.html

• http://www.copperegg.com/product/pricing/47

RESOURCES

• http://www.productionmonkeys.net/guides/web-server/varnish

• http://cr.yp.to/daemontools.html

• http://www.webperformance.com/library/tutorials/CalculateNumberOfLoadtestUsers/

48