Debugging Fastly VCL: 101

Cassandra DixonSolutions Architect | Fastly

Steven ChuobTechnical Account Manager | Fastly

Discussion

● Common Fastly VCL Debugging

● More Technical Fastly VCL

Debugging

Introduction

● Purpose of this training

● What will you learn

● Aiden’s Journey

When Contacting Fastly Support

What to provide:

● Tell us who you are

● Tell us the issue

● Be specific

● Steps to reproduce

● How did you discover it?

Aiden’s Journey through Fastly

FOR PARTICIPANTS:

ALL CODE IN THIS PRESENTATION IS BROKEN.

DO NOT USE IN PRODUCTION.

Fastly’s Shielding Service

Fastly’s Shielding Service (VCL Flow)

Origin

vcl_recv

vcl_deliver

vcl_fetch

Edge POP Shield POP

vcl_recv

vcl_deliver

vcl_fetchUser

restart

• vcl_recv - This is the first

subroutine the request goes

through

• Handles backend/shield

selections

• Modifications to hostname, url

and other req.headers

vcl_recv

• vcl_fetch - This subroutine runs

after the object has been

fetched from the origin/shield

• Does not run on a hit. Only for

miss, pass requests

• Caching logic and TTLs

vcl_fetch

● vcl_deliver - This is one of the

last subroutine that handles the

response.

● This subroutine sends the object

and its headers to a client

making a request.

● Handles modifications to the

response headers before

delivery

vcl_deliver

● vcl_log - Like vcl_deliver this is

one of the last subroutine to be

run.

● If you configure log, it is called in

this subroutine

vcl_log

• vcl_recv - This is the first subroutine the request goes

through, backend/shield selections happens in this routine.

• vcl_deliver - This is one of the last subroutine that

handles the response from the origin. This subroutine sends

the object and its headers to a client making a request.

• vcl_log - Like vcl_deliver this is one of the last subroutine

to be runned. This will send a log to a logging endpoint

after the request has been handled.

• vcl_error - This subroutine handles any error handling

when the origin responds with an error.

• vcl_miss - This subroutine runs once the object is

determined to not be in cache.

• vcl_hit - This subroutine runs once the object is

determined to be in cache.

• vcl_pass - This subroutine will either be runned after

vcl_recv or vcl_fetch.

• vcl_fetch - This subroutine runs right after the object has

been fetched from the origin and its caching TTL set (based

on what’s being sent from the origin).

Cluster Edge Cluster Node

Clustering

The more you know...

Months 1-6

● Setting up an anonymous cloud hosting origin

● Enabling Fastly’s Shielding Service

backend default { .host = “174.138.66.5”; .port = “80”; .connect_timeout = 60s; .first_byte_timeout = 60s; .between_bytes_timeout = 60s; .max_connections = 800;}

backend shield_ssl_cache_lcyxxxx_LAX { .is_shield = true; .connect_timeout = 2s; .port = "80"; .host = "23.235.47.20"; .max_connections = 1000; .share_key = "fastlyshield";} ...

Aiden’s VCL

sub vcl_recv { # custom vcl set_hostname # setting host header to match backend set req.http.host = "hostname expected by origin"; set req.backend = F_addr_xxx;

#do shield here { if (req.backend == F_addr_xxx && req.restarts == 0) { if (server.identity !~ "-LCY$" && req.http.Fastly-FF !~ "-LCY") { set req.backend = ssl_shield_lax_ca_us;} if (!req.backend.healthy) { # the shield datacenter is broken so don’t go to it set req.backend = F_addr_xxx; }}}

if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") { return(pass); } return(lookup);}

Aiden’s VCL

Aiden’s Testing

● Test URL

● Check Aiden’s VCL

● Testing Tools○ watch○ curl○ Logging

Logging

curl -svo /dev/null http://altitude-stage.global.ssl.fastly.net/shielding

< HTTP/1.1 200 OK< Server: Apache/2.4.7 (Ubuntu)< Content-Type: text/html< Fastly-Restarts: 1< Cache-Control: max-age=86400< Content-Length: 14405< Accept-Ranges: bytes< Date: Sun, 18 Jun 2017 16:22:53 GMT< Via: 1.1 varnish< Age: 0< Connection: keep-alive< X-Served-By: cache-sjc3635-SJC< X-Cache: MISS< X-Cache-Hits: 0< X-Timer: S1497802973.177423,VS0,VE171< Vary: Accept-Encoding

Test URL

● Hits to Origin multiple times for the same object

● Fastly-Restart > 0

● Served-By header shows only one POP

● Age is randomly coming back as 0.

sub vcl_recv {

# custom vcl set_hostname # setting host header to match backend set req.http.host = "hostname expected by origin"; set req.backend = F_addr_xxx;

#do shield here { if (req.backend == F_addr_xxx && req.restarts == 0) { if (server.identity !~ "-LCY$" && req.http.Fastly-FF !~ "-LCY") { set req.backend = ssl_shield_sjc_ca_us;} if (!req.backend.healthy) { # the shield datacenter is broken so don’t go to it set req.backend = F_addr_xxx; }}}

if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") { return(pass); } return(lookup);}

Reviewing: Aiden’s VCL

Fastly’s Shielding Service (VCL Flow)

Origin

vcl_recv

vcl_deliver

vcl_fetch

Edge POP Shield POP

vcl_recv

vcl_deliver

vcl_fetchUser

restart

http://altitude-stage.global.ssl.fastly.net/req.http.host == altitude-stage.global.ssl.fastly.netreq.url == / vcl_recv

req.http.host == altitude-debug-workshop.dixonkin.comreq.url == /

vcl_recv

Delivers a 500 internal domain not found to the Edge POP

vcl_deliver

vcl_fetch

A restart is triggered due to the 500 error. Cause: Edge thinks the shield is unhealthy and thus down

Origin

vcl_deliver

Easiest Solution

# OPTION 1 - Create 2 snippet # vcl_miss & vcl_pass

## Setting a hostnameif (!req.backend.is_shield) { set bereq.http.host = "hostname expected by origin";}

# OPTION 2 - Create a vcl snippet with # type “init routine” # create snippets for vcl_miss & vcl_pass

sub set_hostname { ## Setting a hostname if (!req.backend.is_shield) { set bereq.http.host = "hostname expected by origin"; }}

Better Solution

# OPTION 2 - Cont’d# Update Custom VCL file

sub vcl_miss { #FASTLY miss call set_hostname; return(fetch);}

sub vcl_pass { #FASTLY pass call set_hostname;}

sub vcl_recv { #FASTLY recv ## Setting a hostname if (!req.http.Fastly-FF) { set req.http.host = "hostname expected by origin"; }

if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") { return(pass); } return(lookup); }

What about Fastly-FF?

• Aiden creates a login page

• Set Cache-control: max-age=0

• He implemented serving stale

Month 8-10

Fastly Serve Stale

Client

Fastly Origin

Is content cached?

Yes No

Serve Stale Serve Error

Fastly Testing

● Test URL

● Testing Tools○ watch○ curl○ logging○ Fastly-Debug

curl -svo /dev/null http://altitude-stage.global.ssl.fastly.net/caching/time

< HTTP/1.1 200 OK< Server: Apache/2.4.7 (Ubuntu)< Cache-Control: max-age=0, public< Via: 1.1 varnish< Content-Length: 1533< Accept-Ranges: bytes< Date: Tue, 20 Jun 2017 21:48:21 GMT< Via: 1.1 varnish< Age: 8< Connection: keep-alive< Fastly-Debug-Digest: 6d34439e4502f525f4dee9255bb9fb1f3b4943b4a9b792c1b07f09f4d31b7c38< X-Served-By: cache-lax8635-LAX, cache-sjc3640-SJC< X-Cache: MISS, HIT< X-Cache-Hits: 0, 1< X-Timer: S1497995301.171881,VS0,VE0

Test URL

Logging

● Hits in logging

● Cache-Control: max-age=0

● Age is greater than 0.

● Includes log line with a HIT

response

● Includes curl responses;

showing Age=0 and sometimes

Age=6

Aiden’s Support Ticket

Jun 20 14:48:19 Fastly_logs

AltitudeDebugWorkshop-Staging:

2017-06-20T21:48:18+0000 SJC 8.18.217.202 "-" "-"

[20/Jun/2017:21:48:18 +0000] "GET /caching/time

HTTP/1.1" 200 1533 "" "curl/7.48.0"

recv[shield_lax_ca_us:/caching/time] hit

deliver[HIT-STALE:hits=1:age=1] HIT-STALE

sub vcl_recv { # setting host header to match backend set req.http.host = "hostname expected by origin";

# Fix encoding issues in urls set req.url = regsub(req.url, "&amp;", "&"); set req.http.tmpLog = if (req.restarts == 0, "recv", req.http.tmpLog "; recv"); set req.backend = F_addr_xxx; #do shield here ...

if (req.request != "HEAD" && req.request != "GET" && req.request != "FASTLYPURGE") { return(pass); } # Strip out the service ID for readability set req.http.tmpLog = req.http.tmpLog "[" regsub(req.backend, "^[a-zA-Z0-9].*--", "") ":" req.url "]";

return(lookup); }

Aiden’s VCL - vcl_recv

sub vcl_miss { set req.http.tmpLog = req.http.tmpLog " miss";

#--FASTLY MISS START ... # Snippet set_hostname_miss : 100 if (!req.backend.is_shield) { set bereq.http.host = "altitude-debug-workshop.dixonkin.com"; }

#--FASTLY MISS END

return(fetch); }

Aiden’s VCL - vcl_miss

sub vcl_fetch { set beresp.do_stream = true; #if (beresp.status >= 500 && beresp.status < 600) {

# /* deliver stale if the object is available */ # if (stale.exists) { # return(deliver); # } ... #set stale_if_error and stale_while_revalidate (customize these values) set beresp.stale_if_error = 86400s; set beresp.stale_while_revalidate = 10s;

if (req.http.X-Debug) { set beresp.http.Backend-Name = beresp.backend.name; }

...}

Aiden’s VCL - vcl_fetch

● Cache-control: max-age=0 is not equivalent to a pass object

● TTL of zero is cacheable- serve stale applies

Aiden: No Regrets, Lessons Learned

Solutions

OPTION 1 - Update

cache-control header on your

origin

OPTION 2 - Create a cache setting

on Fastly with a condition on the

path

OPTION 3 - Custom VCL in vcl_recv after

the #FASTLY recv macros.

Break

• New Photo Gallery released on

website

• TTL needed for static pictures

• Aiden’s origin is not sending the

cache control header for this

content.

• He adds a cache header on

Fastly

Month 14 -15

sub vcl_fetch { set beresp.do_stream = true; ... #set stale_if_error and stale_while_revalidate (customize these values) set beresp.stale_if_error = 86400s; set beresp.stale_while_revalidate = 10s;

if (req.http.X-Debug) { set beresp.http.Backend-Name = beresp.backend.name; } ... #--FASTLY FETCH BEGIN ... # priority: 10 if ( req.url == "/redirect1" ) { # Header rewrite Static Page Cache : 10 set beresp.http.Cache-Control = "max-age=3600"; } ...}

Aiden’s VCL

Fastly Testing

● Test URL

● Testing Tools○ watch○ curl○ logging○ Fastly-Debug

curl -svo /dev/null http://altitude-stage.global.ssl.fastly.net/redirect1 -H 'Fastly-Debug:1'

< HTTP/1.1 200 OK< Server: Apache/2.4.7 (Ubuntu)< Cache-Control: max-age=3600< Content-Length: 1917< Accept-Ranges: bytes< Date: Wed, 21 Jun 2017 20:24:02 GMT< Via: 1.1 varnish< Age: 1< Connection: keep-alive< CDN-Process-Log: recv[shield_lax_ca_us:/redirect1] hit deliver[HIT:hits=1:age=1]< Fastly-Debug-Path: (D cache-sjc3121-SJC 1498076643) (F cache-sjc3121-SJC 1498076642)< Fastly-Debug-TTL: (H cache-sjc3121-SJC 119.237 86400.000 1)< Fastly-Debug-Digest: cfb378e8d0413b6345893969c58d8ed23b6db487be9197c0d24d8b5833df8970< X-Served-By: cache-sjc3121-SJC< X-Cache: HIT< X-Cache-Hits: 1< X-Timer: S1498076643.983655,VS0,VE0

Curl Test

Logging

Wait - What is Fastly-Debug showing?

Fastly-Debug-Path: (D cache-sjc3121-SJC 1498076643) (F cache-sjc3121-SJC

1498076642)

Fastly-Debug-TTL: (H cache-sjc3121-SJC 119.237 86400.000 1)

The first number is the TTL remaining for the object

The third number is current age of the item in cache

The second number refers to the grace

● Object is expiring after 120 seconds

● Fastly-Debug is saying the object TTL is 120 seconds

sub vcl_fetch { set beresp.do_stream = true; ... #set stale_if_error and stale_while_revalidate (customize these values) set beresp.stale_if_error = 86400s; set beresp.stale_while_revalidate = 10s; ... #FASTLY fetch ...

if (beresp.http.Expires || beresp.http.Surrogate-Control ~ "max-age" || beresp.http.Cache-Control ~"(s-maxage|max-age)") { # keep the ttl here } else if (beresp.cacheable) { # apply the default ttl set beresp.ttl = 3701s; }

}

Aiden’s VCL

Important VCL Logic

TTL Flow

Origin

vcl_recv

vcl_deliver

vcl_fetch

Edge POP Shield POP

vcl_recv

vcl_deliver

vcl_fetchUser

vcl_recv vcl_recv

Origin TTL = 0

TTL = 120Cache-Control: max-age=3600

vcl_fetch

vcl_deliver

vcl_fetch

vcl_deliver

TTL = 3600

● There is a default TTL of 120 that is used when the object has no TTL or Cache-Control headers from the origin.

● The starting TTL is set before vcl_fetch runs on a service

Lessons Learned

Solution

A Cache-control header created

in vcl_fetch does not reset the

TTL

To set a TTL, modify the variable

beresp.ttl in vcl_fetch

Therefore create a cache setting on Fastly

to actually change and set the beresp.ttl

• A starting point for Debugging

• Streaming Logs

• Basic understanding on VCL

subroutines

Recap

Questions?

Thank youCassandra & Steven