< Back to Blog

Google Adwords Bot and Robots.txt
Mon, 16 Jul 2007 14:35:56 by Matt Hopkins

The Google Adwords system has its own crawler/bot that helps Google analyze landing page quality and is separate from the main index crawler (GoogleBot).  This bot does not respect the robots.txt file in the "standard" way.

If you wanted to block all bots from accessing your site or a section of your site, you would add the following to your robots.txt file:  

User-Agent: *
Disallow: /

or

User-Agent: *
Disallow: /folder-to-ignore/

But because many Adwords users may not use robots.txt properly and to prevent them from paying the higher charges (CPC) that would be incurred with a low quality score, the Google Adwords bot will ignore any "global" User-Agent statements.

If you want to block the Adwords Bot, you have to specify it explicitly like this:

User-agent: AdsBot-Google
Disallow: /

or

User-Agent: AdsBot-Google
Disallow: /folder-to-ignore/

Blocking access to the Adwords Bot is not recommended with any of your landing pages - so use this carefully.



Matt Hopkins


Subscribe

Archives

Related Blogs
Pay Per Click Landing Pages Are Key.......
Mon, 21 Jul 2008 15:41:50 by David Thomas
Pay Per Click Trademark Storm Has Passed
Fri, 11 Jul 2008 13:44:40 by David Thomas
Google do do hardcopy
Tue, 17 Jun 2008 13:44:25 by Joe Bursell
Yahoo! get paid for Google Ads
Fri, 13 Jun 2008 10:28:01 by Joe Bursell
Load-time can now influence your Adwords quality score
Wed, 5 Mar 2008 07:31:37 by Matt Hopkins
Google Testing Adsense (PPC Ads) in Videos
Fri, 22 Feb 2008 07:32:20 by Matt Hopkins
Google's New Conversion Optimizer - The Worst Tool Ever....
Thu, 14 Feb 2008 17:00:23 by James Daniels