Durability, Scalability, Availability

I have been using Varnish for quite some time and have always wished that there was some way for Varnish to know to serve “Stale” pages when the upstream application servers are swamped. There is actually a feature request for this on the Varnish Trac system here. NOTE: this feature should not really be necessary unless you have underestimated the ability of your application servers to handle your traffic. However even after proper capacity planning sometimes you get well… DUGG. We all know the “digg effect” (formerly referred to as the “slashdot effect”) and it’s repercussions (500, Guru meditation, Houston we have a problem!) There are many ways to skin a cat, but none would be as simple as this (considering we have an existing varnish setup). I should note that simply getting “Dugg” or “Slashdotted” normally wouldn’t take down a site with a proper reverse proxy setup based on Varnish. If your TTL is appropriate and you are using an appropriate GRACE value (for you Squid readers: “stale-while-revalidate“) then you will probably not saturate your app servers. Unfortunately if your content is good and your UI is right then maybe, just maybe a certain percentage of your new readers will stick around. And here is where it gets scary for the app servers. Maybe just, maybe your new readers will start to navigate in ways that your cache is not used to. Maybe they will start to hit those really OLD articles that haven’t been requested in months! If you think about your sites content vs it’s popularity you it will look something like this:

No matter what you do there will always be something that falls into those “long tails” if your traffic patterns shift suddenly you can very well start to make a lot more request to your upstream servers than you (or more importantly your reverse proxy) expected.

Back to the task at hand. What can I do while I wait for the Varnish team to put this feature through? EASY… use Squid! There are so many debates over which reverse proxy is currently the fastest, which one is easier to setup or integrate with legacy apps etc. I’m certainly NOT trying to get into that! In fact I will skirt the issue entirely saying this: when the features are right and you can afford to use it then why not? NOW don’t get me wrong. Afford can mean a lot of things. Take it as you will. I personally HATE using software, ANY software when I don’t have to. In fact I try to design my stacks as small as possible. As a general rule LESS SOFTWARE IS BETTER! It means less maintenance, less quality assurance… less hastle! However there are situations like the one I described above when you are put between a rock and a hard place. I can either:

A) Swap Varnish out completely and start using squid.

B) Augment my http acceleration layer with squid.

C) Buy more application servers and avoid the issue.

I wish, I wish, I wish C was always an option. Unfortunately not all client’s can afford to simply throw more money at the problem. If I had my choice I would scale horizontally off to the…horizon. SO I now get to choose between A and B. A is what my Sysadmin gut feeling (about never using more software than necessary) is telling me to do. BUT A also has the Test Engineer in me screaming “You will have to test everything all over again!”

Sooo here is another instance where the REAL WORLD comes crashing down on good systems engineering. C is the cheapest most cost effective solution. It could be said that maintaing another piece of software over time is going to be more costly than the upfront cost of swapping out Varnish entirely. But consider this…the Varnish feature that I was mentioning earlier… has already been assigned. It is only a matter of time before someone decides to pick it up and implement it. Hell I might even go ahead and do it if I can find the time. (BTW if your reading this month’s past the publish date of this post then you should definitely check that Trac ticket and see what has become of it.)

C it is. Now I am going to have to dust off my Squid skills and install that beast again. (Of course I couldn’t get through an article about varnish and squid with out some opinion…. Setting up Squid is not the easiest thing in the world!)

I had a bit of a quandary today as I reviewed some code today and I decided to write up this little post on the experience. In essence it came down to choosing between code readability and code reuse. I am a huge fan of both and they almost never conflict in fact they almost always exist in a symbiotic state. But not this time.

This title of this post has been buzzing around in my head recently as I have been doing a lot of performance tuning on a very large and HIGHLY trafficked web site of a major print publication. Anyway I was recently reading (reference coming soon! until then dig through my delicious links) a blog post on the topic of performance tuning and why it is silly how often people micro-tuning. Think of that in the same negative respect that you think of micro-managing. It’s a waste of time and resources. The moral: even though performance tuning is normally a good thing, after a while it looses it’s value. You spend your money on the low hanging fruit (AHEM more hardware!)

Back to today. The code that I was reviewing had a few very cryptic stanzas that included a call to a function that consumed a single integer based parameter which after thorough inspection simply ended up determining if the query it finally triggers against the database is ascending or descending. Now I honestly doubt this developer (no matter how Jr) actually thought they would be increasing the performance of the application by including this integer parameter. In fact I am sure that they assumed that they were doing a good thing by reusing code (which I am a HUGE proponent of normally!) but when it came time for another developer to get into the code to augment it slightly it took exponentially more time then it would have if the original developer had simply a) created the database queries inline or b) created two separate nearly identical functions. Instead (fyi we are using an ORM that follows the ActiveRecord Pattern) the object has one single method getNextArticle(int foo) that returns both the previous AND the next article! How confusing! Anyway the best solution that required the least new untested code at this point was to wrap that function with two new functions that map to the integer values. Now we have:


/*Note that the following function has a different method signature than the original and thus can coexist due to the method overloading feature of java*/
getNextArticle(){
return this.getNextArticle(1);
}



getPreviousArticle(){
return this.getNextArticle(2);
}


I can already hear someone out there saying: “That’s silly why not re-factor the whole object so that it has:

getNextArticle()

getPreviousArticle()

wrapping some new function that is called with a String paramater like ASC or DESC.” Well to put it simply, time. Regression testing takes time and we would have to do a heck of a lot more of it on this project that unfortunately does not yet have 100% unit test coverage (to my DISDAIN) and does not have an automated testing setup. So the moral of the story is this. Even though code reuse is almost always a GOOD sign of proper software development practices sometimes it can lead to poor readability which then becomes more of a maintenance problem than the code reuse actually solves. In retrospect it would have been great to have found this code before it went through thorough user acceptance testing. However due to the accelerated nature of meeting client demands and working on unrealistic project schedules we were not able to do enough peer code reviews of code nor were we able to create a team large enough to do pair programming.

AHH how the real world always ruins principles, techniques, patterns and paradigms that work so well on paper!