How to performance profile an app is an art not covered here. Seriously if you want to start somewhere I suggest reading this article over at Zend developer zone (It’s from 2007 but it’s still relevant):

http://devzone.zend.com/article/2899-Profiling-PHP-Applications-With-xdebug

Also there is a great article if you can find it from the September 2004 edition of php|Architect… If you can’t find it on the internet email me and I will send you a PDF version of it. It is a complete and comprehensive article and should not be missed if you are working with xdebug at all.

So finally back on topic:

Installing xdebug is pretty easy assuming you have php/pecl/pear installed… If you don’t I suggest you use yum and if you don’t have that installed then just follow this tutorial: http://www.matteomattei.com/en/install-yum-and-php-pear-on-centos-5

Anyway once you have pecl installed you can run the following:

# pear install pecl/xdebug

downloading xdebug-2.0.5.tgz …
Starting to download xdebug-2.0.5.tgz (289,234 bytes)
……………………………………………………done: 289,234 bytes
67 source files, building
running: phpize
Configuring for:
PHP Api Version:         20041225
Zend Module Api No:      20050922
Zend Extension Api No:   220051025
/usr/bin/phpize: /tmp/tmpOcEvNL/xdebug-2.0.5/build/shtool: /bin/sh: bad interpreter: Permission denied
Cannot find autoconf. Please check your autoconf installation and the $PHP_AUTOCONF
environment variable is set correctly and then rerun this script.

Woops!

Looks like something is up with running /tmp/tmpOcEvNL/xdebug-2.0.5/build/shtool

I have seen this type of build error before when using other package management systems and it turns out that what I thought was true. That the /tmp directory was mounted with the noexec switch.

See here:

# mount -l | grep /tmp
simfs on /tmp type simfs (rw,noexec)
simfs on /var/tmp type simfs (rw,noexec)

So let’s remount it with the exec switch:

mount -o remount,exec /tmp

And now let’s see what mount -l says:

mount -l | grep /tmp
simfs on /tmp type simfs (rw)
simfs on /var/tmp type simfs (rw,noexec)

So we can see that now the /tmp does not have noexec listed.

Ok let’s try and install via pecl again:

# pear install pecl/xdebug
downloading xdebug-2.0.5.tgz …
Starting to download xdebug-2.0.5.tgz (289,234 bytes)
……………………………………………………done: 289,234 bytes
67 source files, building
running: phpize
Configuring for:
PHP Api Version:         20041225
Zend Module Api No:      20050922
Zend Extension Api No:   220051025
building in /var/tmp/pear-build-root/xdebug-2.0.5
running: /tmp/tmp3tHVR2/xdebug-2.0.5/configure
checking for egrep… grep -E
checking for a sed that does not truncate output… /bin/sed
checking for gcc… gcc
checking for C compiler default output file name… a.out
checking whether the C compiler works… configure: error: cannot run C compiled programs.
If you meant to cross compile, use `–host’.
See `config.log’ for more details.
ERROR: `/tmp/tmp3tHVR2/xdebug-2.0.5/configure’ failed

Well were not in the clear yet the error looks surprisingly similar… let’s just try to mount /var/tmp without the noexec switch as well:

# mount -o remount,exec /var/tmp

And again the pecl install:

# pear install pecl/xdebug
downloading xdebug-2.0.5.tgz …
Starting to download xdebug-2.0.5.tgz (289,234 bytes)
…………………………………………..done: 289,234 bytes
67 source files, building
running: phpize
Configuring for:
PHP Api Version:         20041225
Zend Module Api No:      20050922
Zend Extension Api No:   220051025
building in /var/tmp/pear-build-root/xdebug-2.0.5
running: /tmp/tmpkWolpH/xdebug-2.0.5/configure
checking for egrep… grep -E
checking for a sed that does not truncate output… /bin/sed
checking for gcc… gcc
checking for C compiler default output file name… a.out
checking whether the C compiler works… yes
checking whether we are cross compiling… no
checking for suffix of executables…

<SNIP>

Build process completed successfully
Installing ‘/var/tmp/pear-build-root/install-xdebug-2.0.5//usr/lib64/php/modules/xdebug.so’
install ok: channel://pecl.php.net/xdebug-2.0.5
You should add “extension=xdebug.so” to php.ini

#extension_dir = “/usr/local/lib/php/20060613″

#doc_root=”/usr/local/apache2/htdocs/web”

#the following was added by jesse to fix the path variables when run in cgi mode
#cgi.fix_pathinfo=0

zend_extension=”/usr/local/lib/php/20060613/xdebug.so”
#xdebug.remote_enable=1

;JESSE: the following was taken from: http://code.google.com/p/syslogr-utils/wiki/XdebugHelper
;When using Eclipse with PDT and xdebug — make sure to change your
;tools –> addons –> xdebug helper –> preferences –> idekey = ECLIPSE_DBGP
;(This is the setting that XDEBUG_SESSION_START is set to on your web browser URL)

;JESSE: I have put the following notes in from some research into profiling with webgrind:
;(they were taken from: http://www.chrisabernethy.com/php-profiling-xdebug-webgrind/ )

;NOTE: usually use cachegrind.out.%t.%p when I want one output file per script run,
;but if I want to run a script multiple times and see the aggregate numbers
;I use cachegrind.out.%s and set xdebug.profiler_append = on

;NOTE: A Nice to know, for different cachegrind files per host you can use %H in the
;profiler_output_name, according to
;http://www.xdebug.org/docs/all_settings#trace_output_name

;Always profile scripts with xdebug:
;xdebug.profiler_enable = 1

#xdebug.profiler_enable=1

;Alternatively, enable profiling with GET/POST parameter XDEBUG_PROFILE,
;e.g. http://localhost/samplepage.php?XDEBUG_PROFILE:
;xdebug.profiler_enable_trigger = 1

xdebug.profiler_enable_trigger=1

xdebug.profiler_output_dir=”/tmp/xdebug/”

;the following forces xdebug to append to the outfile rather than
;overwrite it on each exec of a script
;xdebug.profiler_append=On

;the patterns for the names of the outfiles
;xdebug.profiler_output_name = cachegrind.out.%s
xdebug.profiler_output_name = cachegrind.out.%t.%p

;the following opens debug output links in textmate
xdebug.file_link_format = “txmt://open?url=file://%f&line=%l”

xdebug.remote_host=”127.0.0.1″
xdebug.remote_port=9000
xdebug.remote_handler=”dbgp”
xdebug.remote_mode=req
xdebug.idekey=1

Now check the php-cli and see if xdebug shows up in it’s ini dump.

php -i | grep xdebug
/etc/php.d/xdebug.ini,
xdebug
xdebug support => enabled
xdebug.auto_trace => Off => Off
xdebug.collect_includes => On => On
xdebug.collect_params => 0 => 0
xdebug.collect_return => Off => Off
xdebug.collect_vars => Off => Off
xdebug.default_enable => On => On
xdebug.dump.COOKIE => no value => no value

<snip>

Looking good!

Oh and obviously restart apache… /etc/init.d/httpd restart

Sweet.

<snip>

To begin I think it is necessary to explain why things are cached and why caching has an impact on editorial workflow and functionality. Specifically this is in reference to polls. It’s going to take a bit to grasp what is happening so please read slow and bare with me.

Full web page caching or HTTP acceleration is done in part to effectively reduce the load experienced by an application and also to mask performance bottlenecks in the application to users.

Without caching, every time a user makes a request, that request eventually has some action that is performed by business logic in the application and ultimately the database. That action can actually be one or THOUSANDS of connections and queries. Those thousands of queries per user request are normally NOT unique as in one request will normally cause the same exact connections and queries to be performed as the next request for the same content. So as the number of users goes up the number of requests go up and ultimately the number of connections and queries on the db will go up (however far more dramatically).

Here are some scenarios:

1 user makes 1 request for 1 page that takes 500 queries to render = 500 queries on the database.

1 user makes 3 requests for 3 different pages each taking 500 queries to render = 1500 queries on the database.

2 users each make 3 requests for 3 different pages each taking 500 queries to render = 3000 queries.

It’s linear when you look at it in this fashion but if you think about how the database actually performs transactions you will see that the performance hit becomes EXPONENTIAL.

The database performs each transaction in a fashion that is sometimes blocking only allowing a single transaction to request a certain record at a time (rare in read only scenarios) OR the io needed to service requests can actually cause the processor to wait while the hard disks are saturated (common on high volume, data intensive applications). This wait time experienced by the processor can quickly add up to minutes experienced by an individual request. Minutes can become tens of minutes even hours as queries stack up waiting for the hard disks to return the data to the queries in the queue ahead of them.

There are many ways to solve this problem but the first is always to remove the waste.

Why perform all those non-unique calculations over and over again when you can just do them once, store the results and then return those stored results when the same non-unique request comes back again?

That is caching.

There are many levels of caching.

We can cache the query results for each query performed on the database. Every single one of the thousands of unique queries is currently kept cached automatically by the datbase for a certain period of time until it is thrown away to make room for “Hotter” queries.

We can also cache the results of the processing of the business logic (in the case of web applications this is usual some sort of html or xml output) those results can be used to produce an actual .html file on the hard disk.

This called “page level” caching.

Hotness explained (this is not unique to databases this is for most simple caching algorithms)

If a certain request is made for a certain piece of data lets assign it an importance value of 1.

If a subsequent request is made for that same data lets add 1 to that importance value so that it is now 2

If no other request is made for that data it’s value stays at 2 for some period of time.

Now let’s graph this importance or hotness value:

The x axis is some deliniation of the volume of data. Lets just presume that all your data is in a single directory and it all has random alphabetical filenames. Then X would possibly start at at AKURALEJD and end at ZNFNALHE. NOTE the filenames are RANDOMLY assigned to the data.

The Y axis is the hotness or importance value assigned to each piece of data so the Hotter the data the higher it is plotted on the Y axis.

What you will find is that SOME portion of the data is hotter than others. The plot of said hotness follows a bell curve.

So why not just cache ALL possible data ever produced by an application?

Memory is limited. We can’t possibly account for every single unique view of the data. SO we can only keep a portion of that data in the cache. There for we draw vertical lines on our graph starting at the middle of the bell curve and expanding them until the content contained in distance between them is equal to amount of available memory. After that the “tail ends” remain un-cached.

Now what happens when what’s hot changes? Well we have to make room in the cache so that this new data can be put in. How do we know what to throw out of the cache to make room? Well we introduce a lifetime or a TTL (Time To Live) on each element in the cache.

So in it’s most basic form a TTL will tell the cache to “Throw out anything older than X”

That means that as time goes on if something HAS not been requested in a longer period of time than the TTL is set to then it will eventually make it’s way out of the cache.

This aging process also allows for the content that is cached to be “refreshed” or updated periodically. SO if you add NEW data or change data in the application then those changes will eventually make their way into the cache and ultimately be seen by people making requests for cached data.

Ok so that all sounds good so why all the issues with caching?

Before we get into this we need to take an even farther step back. Let’s look at how our content is uniquely named.

When it comes to the web our content is uniquely named by it’s URL (uniform resource locator) See: http://en.wikipedia.org/wiki/Uniform_Resource_Locator

With a unique URL the cache is able to determine “what is what” or more technically what the “STATE” of the application is for that URL. The web used to be so simple.

What webservers do with URL’s:
I will briefly explain the two most common functions a web server performs. These are:

Responding to GET requests. (commonly associated with clicking a link)

Responding to POST requests. (commonly associated with pushing a submit button)

NOTE that BOTH are “REQUESTS” there for if you click a link OR click a submit button you are “REQUESTING” something from the webserver. In the case of a POST request most times you are also sending something extra to the webserver (for instance a contact form OR your answers to a poll!)

Once the webserver receives your request it will check to see if it is a GET or a POST, then it will check to see what file was requested and then if the file is of a certain type it will sometimes offload the request to some application server (for instance PHP running as a cgi process). This application server (PHP for instance) will then process the request (taking into account the information in the GET or POST, making needed database queries etc.) finally the application server returns a response back to the webserver which will in turn return the response to your browser.

That response may or may return unique information depending on what was passed in the GET or POST request to the webserver (and ultimately to the application server) NOTE: In the past ALL requests both GET and POST caused the browser to refresh the page. THAT fact alone is what defined WEB 1.0.

Enter AJAX.

Now web 2.0. We no longer have Unique URLS for every piece of content. More importantly we don’t have page refreshes upon every request. The same URL/page might look one way when you first visit it and then let’s just say (for the sake of a timely example) you submit a poll for instance. If the submission of that poll does not navigate you away from the page to a new URL then with page level caching you will never see the results of the poll as your submission will only return the same exact cached page that you first saw since the url is the same. THEREFOR you will see the UNSUBMITTED poll once again.

So then how do we deal with caching AND AJAX?

We do so in one of two ways.

1) By maintaining STATE or at least some subset of the state on the client side (within the browser’s cookies).

OR

2) By caching only small parts of a page that don’t change and allowing the other parts of a page to remain UNCACHED. This is commonly called “partial caching”.

So why not use partial caching on my project?

Partial caching is entirely application dependent. It has to be architected into the application from the start. You unfortunately cannot “just add” partial caching because someone has to take the time to determine which portions of an application CAN be cached and which portions CANNOT. Most times off the shelf web applications cannot have partial caching added on top of them. Most traditional CMS applications and legacy web applications are not Web 2.0 “savy” enough.

Drupal falls into a gray area. Some modules are smart enough. Others are not. In general you cannot use partial caching with drupal without some legwork.

SO we use the page level caching described above. Now what?

Well now we have to take into account the functionality implications of a page looking the same way no matter what every time you visit it. Sounds like it doesn’t fix anything, and it doesn’t, without some tricks.

To start we can tell the cache to NEVER respond to a POST request with cached content. That means that every time someone submits a poll then show them the results. The results are generated each time by the application server and then the unique response is sent back to the browser.

Why does the poll still NOT work on the site then?

The long and the short of it:

It does. BUT unfortunately the poll is one of those modules that was NOT written to be used with page level caching.

What’s happening is this:

The first time you visit the site (or at least before ever taking the poll) you get there by typing www.yourpage.com into your browser. That is a GET request and the homepage is retrieved from the cache. You see the poll and you ARE allowed to vote and you DO see your results. Your request is sent via a POST to the application servers and your response is returned by the application servers… NOT the cache because it was a POST request.

Then you navigate away from that page and everything is fine. Then let’s say you come back to the home page by clicking the logo in the top left. That is a GET request. The homepage is returned FROM THE CACHE this time and you do NOT see your results in the poll. You see a poll that has not been taken yet. Because that is how the homepage was cached originally.

SO then you try to take the poll again and this time it doesn’t let you submit. Why? Because the poll module in drupal was built to only allow ONE vote per person per poll. That means that you CANT submit the poll more than once.

So since you have already submitted and since the poll is cached as unsubmitted you are stuck with an unsubmittable poll.

DARN! How do we fix this?

The easy way: Make the polls allow more than one vote per person per poll. Really why is that so bad?

The hard way: Rewrite the poll module correctly to refresh itself via another Ajax call for an un-cached version of the results. This means essentially writing a new poll module.

I hope that explains why the polls are not working. Maybe you picked up a thing or two a long the way.

</snip>

$telnet master 9000

telnet: connect to address 192.168.0.1: Connection refused

however if you try the same thing from the master machine itself it works!? (assuming your name node is currently running and supposedly listening on port 9000)

This of course through me for a loop and I assumed it was everything from firewalls to bad interconnects between my machines. Strange that I never thought about the hostnames. I guess it was because all other services between them have been working fine. If anyone has any idea why hdfs has a problem with this I would love to know.

#!/usr/bin/env python

import sys

import gdata.youtube

import gdata.youtube.service

import re

racist_pattern = re.compile(’.*igger.*’, re.IGNORECASE)

#import pprint

#pp = pprint.PrettyPrinter(indent=4)

yt_service = gdata.youtube.service.YouTubeService()

#yt_service.developer_key = “”     #turns out the developer key isn’t necessary

urlpattern = ‘http://gdata.youtube.com/feeds/api/videos/%s/comments?start-index=%d&max-results=50′

for line in sys.stdin:

video_id = line.strip()

index = 1

url = urlpattern % (video_id, index)

#print url

comments = []

while url:

if index < 20:

comment_feed = yt_service.GetYouTubeVideoCommentFeed(uri=url)

#comments.extend([ comment.content.text for comment in comment_feed.entry ])

for comment in comment_feed.entry:

if racist_pattern.match(comment.content.text):

print ‘%s\t%s\n’ % (comment.author[0].name.text, comment.content.text)

#print [ 'Author: %s\t Comment: %s\n' % (comment.author[0].name.text, comment.content.text) for comment in comment_feed.entry ]

url = comment_feed.GetNextLink().href

index += 1

else:

#currently the google youtube gdata api will not support over 1000 comments

url = ‘http://gdata.youtube.com/feeds/api/videos/%s/comments?start-index=951&max-results=49′ % video_id

comment_feed = yt_service.GetYouTubeVideoCommentFeed(uri=url)

for comment in comment_feed.entry:

if racist_pattern.match(comment.content.text):

print ‘%s\t%s\n’ % (comment.author[0].name.text, comment.content.text)

#comments.extend([ comment.content.text for comment in comment_feed.entry ])

#print [ 'Author: %s\t Comment: %s\n' % (comment.author[0].name.text, comment.content.text) for comment in comment_feed.entry ]

break

No matter what you do there will always be something that falls into those “long tails” if your traffic patterns shift suddenly you can very well start to make a lot more request to your upstream servers than you (or more importantly your reverse proxy) expected.

Back to the task at hand. What can I do while I wait for the Varnish team to put this feature through? EASY… use Squid! There are so many debates over which reverse proxy is currently the fastest, which one is easier to setup or integrate with legacy apps etc. I’m certainly NOT trying to get into that! In fact I will skirt the issue entirely saying this: when the features are right and you can afford to use it then why not? NOW don’t get me wrong. Afford can mean a lot of things. Take it as you will. I personally HATE using software, ANY software when I don’t have to. In fact I try to design my stacks as small as possible. As a general rule LESS SOFTWARE IS BETTER! It means less maintenance, less quality assurance… less hastle! However there are situations like the one I described above when you are put between a rock and a hard place. I can either:

A) Swap Varnish out completely and start using squid.

B) Augment my http acceleration layer with squid.

C) Buy more application servers and avoid the issue.

I wish, I wish, I wish C was always an option. Unfortunately not all client’s can afford to simply throw more money at the problem. If I had my choice I would scale horizontally off to the…horizon. SO I now get to choose between A and B. A is what my Sysadmin gut feeling (about never using more software than necessary) is telling me to do. BUT A also has the Test Engineer in me screaming “You will have to test everything all over again!”

Sooo here is another instance where the REAL WORLD comes crashing down on good systems engineering. C is the cheapest most cost effective solution. It could be said that maintaing another piece of software over time is going to be more costly than the upfront cost of swapping out Varnish entirely. But consider this…the Varnish feature that I was mentioning earlier… has already been assigned. It is only a matter of time before someone decides to pick it up and implement it. Hell I might even go ahead and do it if I can find the time. (BTW if your reading this month’s past the publish date of this post then you should definitely check that Trac ticket and see what has become of it.)

C it is. Now I am going to have to dust off my Squid skills and install that beast again. (Of course I couldn’t get through an article about varnish and squid with out some opinion…. Setting up Squid is not the easiest thing in the world!)

This title of this post has been buzzing around in my head recently as I have been doing a lot of performance tuning on a very large and HIGHLY trafficked web site of a major print publication. Anyway I was recently reading (reference coming soon! until then dig through my delicious links) a blog post on the topic of performance tuning and why it is silly how often people micro-tuning. Think of that in the same negative respect that you think of micro-managing. It’s a waste of time and resources. The moral: even though performance tuning is normally a good thing, after a while it looses it’s value. You spend your money on the low hanging fruit (AHEM more hardware!)

Back to today. The code that I was reviewing had a few very cryptic stanzas that included a call to a function that consumed a single integer based parameter which after thorough inspection simply ended up determining if the query it finally triggers against the database is ascending or descending. Now I honestly doubt this developer (no matter how Jr) actually thought they would be increasing the performance of the application by including this integer parameter. In fact I am sure that they assumed that they were doing a good thing by reusing code (which I am a HUGE proponent of normally!) but when it came time for another developer to get into the code to augment it slightly it took exponentially more time then it would have if the original developer had simply a) created the database queries inline or b) created two separate nearly identical functions. Instead (fyi we are using an ORM that follows the ActiveRecord Pattern) the object has one single method getNextArticle(int foo) that returns both the previous AND the next article! How confusing! Anyway the best solution that required the least new untested code at this point was to wrap that function with two new functions that map to the integer values. Now we have:


/*Note that the following function has a different method signature than the original and thus can coexist due to the method overloading feature of java*/
getNextArticle(){
return this.getNextArticle(1);
}



getPreviousArticle(){
return this.getNextArticle(2);
}


I can already hear someone out there saying: “That’s silly why not re-factor the whole object so that it has:

getNextArticle()

getPreviousArticle()

wrapping some new function that is called with a String paramater like ASC or DESC.” Well to put it simply, time. Regression testing takes time and we would have to do a heck of a lot more of it on this project that unfortunately does not yet have 100% unit test coverage (to my DISDAIN) and does not have an automated testing setup. So the moral of the story is this. Even though code reuse is almost always a GOOD sign of proper software development practices sometimes it can lead to poor readability which then becomes more of a maintenance problem than the code reuse actually solves. In retrospect it would have been great to have found this code before it went through thorough user acceptance testing. However due to the accelerated nature of meeting client demands and working on unrealistic project schedules we were not able to do enough peer code reviews of code nor were we able to create a team large enough to do pair programming.

AHH how the real world always ruins principles, techniques, patterns and paradigms that work so well on paper!