1

Trade Me: Solving Real-World Problems With Varnish Cache

April 2016

About Trade Me

• NZ’s largest domestic website

• Auctions & Classifieds• Jobs, Property, Motors• Used to be just

www.trademe.co.nz• Now we have a public API

too

Some Stats

• Over 60 million requests/day (desktop)• Same again for API• 700+ active 3rd party apps consuming our API• 5 first party apps (iOS/Android/websites)• Over 60% of our internet traffic is photos & other static

content• We achieve ~95% hitrate for static content with Varnish

More Stats

6

Problem #1

Integrating a web app with a CMS

Problem statement

• Pages with static text were part of the main project• Text changes required a full deploy • Deploy requires involvement from several areas• Took too long to make simple changes, higher risk• Applied to several sections of the site, we wanted to split

into separate “microsites”• “Hey, we should use a CMS!”

CMS – challenges

• Needed a way of keeping the dynamic header and footer from the main site, while adding CMS functionality for creating and editing content

• CMS microsites needed to reflect changes to main site’s CSS and JS

• TL;DR problem – how to seamlessly integrate a separate app (on different servers) into the main site

Header

Footer

Content

Custom logoGreeting

Shopping cart count

Search history

Dynamic drop-down menus

TimeAdvertising

Normal page request

Load balancer

Main siteSite Content

Footer

Header

Request for CMS page

Load balancer

Varnish (ESI)

CMSCMS Content

URL rule - eg /help/*

<esi:header.aspx>

<esi:footer.aspx>

Request for CMS page

Load balancer

Main site

Varnish (ESI)

CMS

URL rule - eg /help/*

header.aspxfooter.aspx

CMS Content

<esi:header.aspx>

<esi:footer.aspx>

Header

Footer

17

Problem #2

Changing backend file structure (with no downtime)

Problem statement

• File metadata performance concerns on storage• Needed to change how image files were stored on disk :• From 100 dirs. per image size to 1000

Changing backend file structure – challenges

• Needed to change underlying file storage scheme• Changes to file upload/save code• Changes to image URLs in web application• Tidy up URLs at the same time (why not?)• Remove “_full” from Fullsize image filename• Remove directory ID from all URIs

• All without causing any user-facing errors

Basic steps

1. Agree image ID for change2. Code changes (inactive)3. VCL changes (inactive)4. [activation happens]5. Backfill all active images to the new dir structure6. Remove all old logic

Regular Expressions …are hard Even when you are staring

at them, it takes a while to process what you’re looking at.

Regex – use a tool

VCL Changes

• Extract image ID from URL (regex)• If image ID > $threshold• Rewrite URL for backend (add folder, remove _full from filename)• Log all changes to Varnishlog

VCL

We’ll need this later

VCL – new sub

Extract the image ID from the URLStore it in an HTTP header

vcl_recv

If the image ID is new, ensure we use the thousand directories

VCL

Rewrite From (client request) Rewritten To (backend request)/photoserver/thumb/12345.jpg /photoserver/thumb/

345/12345.jpg/photoserver/thumb/45/12345.jpg

/photoserver/thumb/345/12345.jpg

/photoserver/full/12345.jpg /photoserver/full/345/12345.jpg

/photoserver/45/12345_full.jpg /photoserver/full/345/12345.jpg

/photoserver/lv/0/1234500.jpg /photoserver/lv/500/1234500.jpg

Here’s some examples showing how we consistently rewrite requests for old and new formats to what we need on the backend

What’s next for us with Varnish?

• Move servers to private cloud• Investigate object storage• Investigate using Varnish API Engine for our public API• Better stats and management

Questions