Mon, 16 Jul 2007 14:35:56 by Matt Hopkins
The Google Adwords system has its own crawler/bot that helps Google analyze landing page quality and is separate from the main index crawler (GoogleBot). This bot does not respect the robots.txt file in the "standard" way.
If you wanted to block all bots from accessing your site or a section of your site, you would add the following to your robots.txt file:
User-Agent: * Disallow: /
or
User-Agent: * Disallow: /folder-to-ignore/
But because many Adwords users may not use robots.txt properly and to prevent them from paying the higher charges (CPC) that would be incurred with a low quality score, the Google Adwords bot will ignore any "global" User-Agent statements.
If you want to block the Adwords Bot, you have to specify it explicitly like this:
User-agent: AdsBot-Google Disallow: /
or
User-Agent: AdsBot-Google Disallow: /folder-to-ignore/
Blocking access to the Adwords Bot is not recommended with any of your landing pages - so use this carefully.
Matt Hopkins
|