i just finished working on a neat referrer tracking page that displays which pages people are visiting on my site–and from whence they came, even decoding the keywords of search engine queries. check it out!

how does referrer tracking work?

unless you’ve gone out of your way to disable this feature (usually for privacy reasons), every time you click a link on a webpage, the url of that page gets sent to the webserver of the link you clicked.

for example if you click on a search result in google, the website you arrive at “knows” the url of your google query you came from–which contains the keywords you typed into google. there is even a blog plugin which highlights these keywords to make it easier for visitors to find what they were searching for.

in order to track the sites (like google) that refer vistors to your site, a service like extreme tracking takes advantage of the fact that javascript has access to the referrer URL of the current webpage. it encodes this url as a query string in what becomes a separate request for a single pixel transparent gif located on extreme tracking’s servers.

they happily provide the gif image while logging the encoded referrer url and other details of the “hit” in their database. they use this information primarily to provide a nice, free interface (with those “lucrative” google ads), detailing information about the last 20 referrers as well as cumulative hit counts over various intervals.

if you’re interested or curious to know what keywords people might be typing into google in order to find your website, that referrer information is golden. however once you start getting more than 20 hits a day, extreme tracking’s free service loses its luster.

so what did you do exactly?

a while ago i wrote a custom apache logging program that stored the raw http request information in a database, but it turns out there are these evil referrer spamming programs (e.g. Reffy) which send fake requests to webservers with referrer urls for various viagra and texas hold’em sites. so all the data i collected was littered with hundreds of bogus requests. that was a dead end.

as it turns out, extreme tracking’s use of javascript means that hits are only logged when a webpage is rendered in a web browser, causing that little 1 pixel gif to be requested. so i modeled my second attempt on that principle. i put a line of javascript in all of my webpages which triggers an image request that happens to be a php script that logs the referrer information and returns a single pixel gif. easy as pie.

then i wrote a little bit of php to decode the search engine query strings (including google’s image search!) so that i could see what search terms were bringing people to which pages on my site. eventually i may add filters and most requested lists, but right now i’m just enjoying reloading the referrer page to get a real time picture of the requests coming to my site.

can i see the code?

sure! well, actually it’s in pretty rough shape. but why not?
15-Feb-2005 v0.1 custom referrer tracking

note to windows users: first save the tar.gz file to your desktop, then extract with winzip to view the files.

there is one major weakness of hosting a webserver on a dhcp network: your IP address could change at any time.

sometime last night after 10pm my cable modem was given a new IP address. which means anyone trying to get to justinsomnia.org since then would have ended up looking for my site at the old IP address (24.211.162.54) and would have gotten a “site not found error”.

i went home at lunch to figure out why my site was unreachable, discovered what had happened, and updated my zoneedit dns records to reflect the new IP address (24.163.64.42). but because of the global nature of the dns system, the old IP address is probably cached on DNS servers all around the world (which may prevent me from even posting this), so it will be a day or two until the new IP address propagates.

the good news is that i had the same IP address since I set up my server last august. which means i’ve had about 5 months of continuous uptime. the bad news is that i thought my IP address would be more stable than that, considering my internet connection and webserver are always on. maybe roadrunner did some kind of across the board reorganization of their IPs?

all of roadrunner’s phone numbers are busy. ah, customer service. but i got through to their live online chat service and they said my IP address could change every night when the DHCP lease is renewed, even though that is unlikely. and then they went on to remind me that webservers are prohibited in my service agreement. gee thanks.

Tyson: Please be aware that the terms of use for the residential Road Runner service prohibit using any kind of server application. This is outlined under section 5 D of the terms of use agreement which was signed before Road Runner was installed.

If you would like to obtain a copy of this agreement or any other service agreement details, or would like to inquire about service options that will allow the use of servers, we recommend contacting the local office. You may also go to [this] website for related policies.

gotta love ISP’s that don’t understand the spirit of the internet.

after making and eating chili with jane, i came back to my place on the way to catch the tail end of a poker game, only to discover it had been postponed due to the wintry mix.

so it’s around 10pm and i settle down to work some more on the dynamic blogroll. i thought i’d make a few more changes (thanks to some suggestions from ryan) and release v0.3. of course some of the changes required that i have a later version of php, but my server is running debian gnu/linux which tends to lag in the up-to-datedness of its software packages.

thus began a long journey in which, with limited knowledge of debian, or linux for that matter, i attempt to figure out how to get php to the latest version.

now i have no problem configuring and compiling things from source, but my experience with debian has taught me that they put their files in slightly different locations (more like OpenBSD, i believe). so if i installed php from scratch, i’d have the debian version in one place and the newly compiled version in another and it would be a mess. but this was all moot because in order to install php to be used with apache, i needed apxs (APache eXtenSion tool) which i didn’t have because i didn’t compile apache from the source. so now i was going to have to install apache which would put its files all over the place and really gunk things up.

so i almost get to emailing my dad, i half composed an email to my linux hacker friend nolan, but i kept going back to google, digging around, reading php and apache install docs, when someone, somewhere mentions apt-get.org as a place to go for more up-to-date, unofficial debian packages. talk about the world’s most unassuming website. from there i find packages.dotdeb.org which is listed as having packages to easily turn your box into a LAMP server. perfect!

30 minutes later i figure out/remember how to update the list of possible package sources for apt-get (/etc/apt/sources.lst) update php and mysql, and i’m off to the races.

except it’s 1:30 in the morning. so really i’m off to bed. but at least i got php upgraded to version 4.3.10.

my favorite blog posts from each month during the past year.

january: where in the world is carmen sandiego?
one of the few memes i’ve participated in, and after traveling to killington in march, i can add vermont (and new hampshire) to the map

february: my milkshake brings all the boys to the yard
a weird mashup created in post-dance-party haze

march: on covenant: point, counter point
my response to patrick’s comment on defining marriage, sparking a flurry of responses

april: a kiln opening howto
explains what a mark hewitt kiln opening is like for those of us who make waking at 5am part of the ritual

may: hell yeah i graduated
dad’s excellent shot of me mid-stride on the way to the sils graduation ceremony–the culmination of 19 straight years of formal education

june: a politcal coming of age
musing on my response to the events of sept 11, after seeing fahrenheit 9/11 on opening day

july: breakfast of champions
recipes make excellent fodder for blog posts–more people should blog their recipes

august: so i built a server
the beginning of a project that now hosts this blog and the editplus wiki

september: on future careers
pondering what to do next, realizing it might be quite different from what i have been doing and that i might be pondering for a while

october: file this under “things not to do at 70mph”
a picture so amazing i can’t imagine i took it

november: new code
one metric of the work i did while in ghana

december: searching for strong’s
musing on what makes the perfect space for laptops, friends, and familiar strangers

if you’re reading this, then you realize that i’ve moved my blog onto a webserver in my living room. my unc web space is now essentially an html business card, inspired by anton’s recent post. the liberal use of tilde inspired by rebecky.

grilled hot dogs and burgers this evening, discovered that you can open a can of chili and sit it right on the grill if you want chili for your hot dogs. the kids played tennis and swam, i enjoyed not worrying about nothing. after a little tennis to compensate (yeah right) for mass quantities of chili dogs consumed, i traveled to raleigh to check out jackie and ryan’s new house/apartment and to celebrate their recent marriage.

had a nice time chatting and touring the neighborhood, looking forward to coming back some weekend night to take advantage of their south glenwood nightlife just a block away.

came back to chapel hill, brought recently under the weather jane some ginger ale. now i’m here, at home, in bed, typing this. the narrative has caught up with the present. good night.