Crawl Errors and robots.txt

GoogleBot, MSNBot, Yahoo!Slurp ... Everything about indexing Bots, ip lists, User Agents, Crawl and robots.txt.

Moderator: Moderators

Crawl Errors and robots.txt

Postby darkone » Mon Jun 15, 2009 1:31 pm

I am looking over my crawl errors in the google webmaster tools and here is my robots.txt file:

Code: Select all
User-agent: *
Disallow: /adm/
Disallow: /ads/
Disallow: /antispam/
Disallow: /blocks/
Disallow: /cache/
Disallow: /dl_mod/
Disallow: /docs/
Disallow: /download/
Disallow: /files/
Disallow: /includes/
Disallow: /language/
Disallow: /phpbb_seo/
Disallow: /store/
Disallow: /styles/
Disallow: /umif/
Disallow: /umil/
Disallow: /common.php
Disallow: /config.php
Disallow: /mcp.php
Disallow: /memberlist.php
Disallow: /posting.php
Disallow: /report.php
Disallow: /search.php
Disallow: /ucp.php



Why are they reporting that they are having crawling errors to memberlist, ucp, ads, search and download pages when they are supposed to be restricted from them in the first place right? And does the crawl errors page does it have any effect on your rank placement in google?


Thanks

-D1-
darkone
 
Posts: 21
Joined: Thu Jul 12, 2007 3:23 pm

Advertisement

Re: Crawl Errors and robots.txt

Postby dcz » Wed Jun 17, 2009 9:58 am

Well, google reports as "error" url being disallowed by a robots.txt, so it may just be this, eg nothing ;)

++
Useful links :
SEO Forum || SEO Directory || SEO phpBB || Search
____________________

Liens Utiles :
Forum référencement || Annuaire référencement || Référencement phpBB || Recherche
dcz
Admin
Admin
 
Posts: 21219
Joined: Fri Apr 28, 2006 9:03 pm


Return to roBots

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 2 guests