How to Track Googlebot Visits March 4, 2006
(Listen to article)
Many bloggers want to track when the Googlebot visits their weblog, for various reasons. One way is to use Google Sitemaps, to see which of your pages Google has index, as well a relative measure of pagerank for those pages. However, this does not tell you when the Googlebot visited.
If you want a more accurate accounting on Googlebot’s visits, one way is to check your web server logs. These logs tell you not just when Googlebot visited, but other bots as well - provided you know how to decipher all of the different “user agents”. But if your host provider doesn’t give you access to your web logs, you have to use other means. For example, some blogging platforms have a plugin to track Googlebot visits. For WordPress, the one I’ve been using is Fabrizio Tarizzo’s WP-Googlestats.
The WP-Googlestats plugin does not make it immediately clear that you need to manually create a mySQL table first. (I had to go to the source website to get this information.) When you create the table, don’t forget to use the exact same table name prefix as for the rest of your blog’s mySQL tables. Other than that, it’s a cinch to use from the control panel. You’ll see “Googlebot” at the far right under the “Manage” tab of WordPress. I’ve used it with WordPress 1.5.2.
Now this plugin doesn’t give you details of other bots, but it’s a start. What would be nice is if there was a comprehensive, publicly accessible list of IP addresses for all types of bots. Having such a list means being able to filter out all non-human traffic. Such a list is one of my many projects. Once I’ve put something together, the code will be posted here at codeprofessor.com.
- Posted in : Googlebot
- Author :rdash
- Please postmark:
Comments»
Thanks for the useful information.
I asked my provider where to access to the original log files. After accessing the logs, I could track googlebot behavior easily just by visual inspection. In them every line contains a “hit”, that is a request to your server from an IP, like
65.67.82.167 - - [26/3/2006:09:49:42 +1000] “GET /robots.txt HTTP/1.0″ 404 288 “-” “googlebot/2.1 (+http://www.googlebot.com/bot.html)”
This way you can track it with your own eyes.
Your article was an eye opener for me. Now I track googlebot regulary by reading the access logs. I found out that after I made the google sitemap, the googlebot is frequently visiting my site now.
Try the WP-GoogleStats plugin as well. I use that on my sites where I don’t have access to my web server logs.
To Google Success, sorry for your comment not appearing until now. Spam Karma trapped it for some reason. Glad your provider gave you access. I use log files for those domains that my host provider allows access. For everything else, I use the info above.