i just finished working on a neat referrer tracking page that displays which pages people are visiting on my site–and from whence they came, even decoding the keywords of search engine queries. check it out!

how does referrer tracking work?

unless you’ve gone out of your way to disable this feature (usually for privacy reasons), every time you click a link on a webpage, the url of that page gets sent to the webserver of the link you clicked.

for example if you click on a search result in google, the website you arrive at “knows” the url of your google query you came from–which contains the keywords you typed into google. there is even a blog plugin which highlights these keywords to make it easier for visitors to find what they were searching for.

in order to track the sites (like google) that refer vistors to your site, a service like extreme tracking takes advantage of the fact that javascript has access to the referrer URL of the current webpage. it encodes this url as a query string in what becomes a separate request for a single pixel transparent gif located on extreme tracking’s servers.

they happily provide the gif image while logging the encoded referrer url and other details of the “hit” in their database. they use this information primarily to provide a nice, free interface (with those “lucrative” google ads), detailing information about the last 20 referrers as well as cumulative hit counts over various intervals.

if you’re interested or curious to know what keywords people might be typing into google in order to find your website, that referrer information is golden. however once you start getting more than 20 hits a day, extreme tracking’s free service loses its luster.

so what did you do exactly?

a while ago i wrote a custom apache logging program that stored the raw http request information in a database, but it turns out there are these evil referrer spamming programs (e.g. Reffy) which send fake requests to webservers with referrer urls for various viagra and texas hold’em sites. so all the data i collected was littered with hundreds of bogus requests. that was a dead end.

as it turns out, extreme tracking’s use of javascript means that hits are only logged when a webpage is rendered in a web browser, causing that little 1 pixel gif to be requested. so i modeled my second attempt on that principle. i put a line of javascript in all of my webpages which triggers an image request that happens to be a php script that logs the referrer information and returns a single pixel gif. easy as pie.

then i wrote a little bit of php to decode the search engine query strings (including google’s image search!) so that i could see what search terms were bringing people to which pages on my site. eventually i may add filters and most requested lists, but right now i’m just enjoying reloading the referrer page to get a real time picture of the requests coming to my site.

can i see the code?

sure! well, actually it’s in pretty rough shape. but why not?
15-Feb-2005 v0.1 custom referrer tracking

note to windows users: first save the tar.gz file to your desktop, then extract with winzip to view the files.