Refer madness
Jul 28th, 2003 by mark
On my site I use a PHP/MySQL tool to track incoming referers. The tool, called Refer 2.01 is very slick and it gives me a sense of where you all are coming from to get to my humble little web site.
Recently I have noticed a trend, well a phenomenon actually. Back in June I posted a story about the new Apple products from the WWDC held in San Francisco. This posting included the word “panther” oddly enough. Panther is the code name for the next release of Mac OS X, release 10.3. Since that posting I have seen a surge in refer hits from google in the form of a search string for “torrent” coupled with the string “panther”. 40 in the last week, similar numbers in the weeks prior to last.
Bit Torrent is a new peer-to-peer file sharing and download program. Obviously, people are searching for copies of the Panther beta to download and use, and they want to use Bit Torrent to accomplish this download. Somehow my referer log page has enough hits on it from google with the words “torrent” and “panther” that it is now the number 1 link from google for those search arguments.
While a part of me likes that my site is garnering all these hits, I realize that the people aren’t coming to see my content. They are just trying to find a copy of Apple’s latest OS to play with, and through the power of the Internet, they are getting my refer logs instead.
I’m either going to have to turn the refer logs off for a while, or figure out how to hide those pages from google (and other search engines as well) so that searchers find what they are looking for. Or hope that some percentage of visitors to my refer log will follow the link up top to my main page and actually see my content.
Of course in writing this posting I have used the magic words, torrent and panther, several times. Maybe now the search engines will point to this posting instead of my log page.
A hits a hit, right?
Keep the thing.
You can hide your refer page by adding a file called “robots.txt” to your server root (probably the folder “public_html” if it’s anything like mine) with this info in it:
User-agent: *
Disallow: /refer
It takes Google like six to eight weeks to crawl your site and find that file, but you can request they do it sooner by going here:
http://www.google.com/remove.html#exclude_website
You have to have the robots.txt file on your server first, before making the request. This will hide your /refer subdirectory from all spiders and bots, by the way.
Good luck!