Testing Varnish-Cache

Originally posted here.

Our team has many RestFul APIs written in PHP that serve information stored in a MySQL database. The content we serve supports many sections of the paper and that means we get a lot of traffic. The majority of our content is not personalized and we control how often that content is updated, which makes it attractive to cache. Before adding Varnish to our stack, we spent significant effort dealing with load spikes, especially at the database layer. The applications that were causing the most pain had a very large, complicated code base and fixing that code or even replacing that code was estimated at six or more months. We wanted a solution that we could implement quickly and didn't require a major code rewrite. We wanted a solution that would take the pressure off of our application servers and our database servers to remove the distractions of dealing with production load alerts. Varnish was the perfect solution to our problem. By adding Varnish in front of our API servers, we took a huge load off of our API and database servers.

At first we added basic caching, but as we used the software and got more experience we found different things we could do in the varnish layer. Varnish has a lot of features and capabilities. Taking advantage of the software inevitably results in a lot of configuration and tweaking. Some functionality can be achieved by editing the varnish configuration language (VCL) and sometimes you need to add a module or create your own module. Writing VCL or working with varnish modules (vmods) leads to testing, which is what I'm writing about today; how to test Varnish. This post will focus on how to test VCL and if I have time, I will add another post to talk about how to test varnish modules.

Let’s imagine a real-life feature and how we can go about testing the feature. Our javascript programmers use JQuery and sometimes turn on JQuery cache busting. When enabled, JQuery adds a timestamp e.g. “_1331829184859=” to the query string in an attempt to bust the cache. So, if I strip the query string parameter, I can prevent JQuery from busting our cache. Here's one way I could clean our URL using VCL:

The first “if” statement strips the “_1331829184859=” and the rest of the lines are there to clean the url so it isn't left with unneeded characters. The URL is reset on line 11.

How can I test this code? I could start a varnish server with a backend apache or nginx instance that logs requests, issue a variety of curl requests and then manually verify the logs, but there’s an easier way. Varnish ships with the ability to test using the testing tool varnishtest. Varnishtest gives you the ability to write VCL tests you can run on the command line or as part of your build process. Here's an example:

Line 1 is just for documentation purposes. Lines 2 - 6 define a server that will accept a request and issue a response. Since I intend to clean the URL before it get’s passed to a backend server, I added a test to verify the backend URL sent to the server with the “expect” syntax on line 4. If the value doesn’t match, the test stops and you get lots of debugging style output.

Line 7 is the syntax to add a backend definition and VCL. Line 8 is optional. I added it to illustrate you're allowed to import vmods. At line 9 I define the vcl_recv subroutine and lines 10 – 16 has the logic I need to remove the cache busting parameter.

Lines 19 – 21 define vcl_deliver to set a response header. I did this to illustrate one way to validate logic during a client request. Line 23 is where I define a client. The client's are where you issue requests with the txreq (transmit request) and receive the response with rxresp (receive response.) At line 26, I use expect to verify the test variable is as expected. Adding the "expect" in the client logic is a better way to test multiple inputs. I kept the example short, but it's trivial to add multiple txreq -url, rxresp, expect lines to test different inputs. You can also copy and paste the client c1 to create c2, c3...cN clients.

Line 28 is where I run the client logic.

Our tests are saved in files e.g. test01.vtc. Assuming you compiled and installed varnish in the standard locations, running varnishtest is this easy:

When the test fails, you get a lot of output to look at. Normally, your good tests produce very little output. However, you can run your tests in “verbose” mode (-v) to get full output. For the simple test above, a passing test in verbose mode produces 253 lines of output. A failing tests produces less lines of output, but only a few less: 189.

Varnishtest allows you some flexibility as well. If you’re building and testing tweaks to varnish, you can specify what varnishd to use with –D e.g.

You also use the -D option to your advantage and pass variables to your test, since ${varnishd} (or anything else defined with –D) will be available to your VCL when the test is compiled and run. For example, when building and compiling custmo varnish modules, you can import the library from the build directory e.g.

Conclusion

Testing with varnishtest made a huge impact when working with my VCL. I can quickly and easily add logic and test theories on my local vagrant box. There is a bit of a learning curve to get started with varnishtest, but the documentation is getting better and there are good examples to follow on the internet.