English English | French French
 

Robot.txt blocking google bot

General discussions Forum for everything that does not fit elswhere in the phpbb SEO forums.

Moderator: Moderators

Post a reply

Robot.txt blocking google bot

Postby praveen » Sun Nov 06, 2011 11:30 am

Hi, I have installed SEO mod (url rewriting mod), it generate a robot.txt file which i placed on my site.

Robot.txt-
User-agent: *
Disallow: /viewtopic.php
Disallow: /viewforum.php
Disallow: /index.php?
Disallow: /posting.php
Disallow: /search.php?
Disallow: /ucp.php
Disallow: /mcp.php
Disallow: /post
Disallow: /member
Disallow: /memberlist.php
Disallow: /faq.php


Now my webmaster tool is displaying below error--
Code: Select all
Show URLs: Not found ‎(1)‎
Restricted by robots.txt ‎(182)‎
URL    Detail    Detected
http://www.devftp.com/post32.html
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/viewtopic.php?f=8&t=29
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/memberlist.php?mode=viewprofile&u=1113
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/viewtopic.php?t=304&p=355
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/viewtopic.php?t=303&p=353
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/viewtopic.php?t=302&p=352
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/viewtopic.php?f=8&t=29&p=44
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/viewtopic.php?f=8&t=303&start=0&view=print
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/viewtopic.php?f=8&t=304&start=0&view=print
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/memberlist.php?mode=email&t=302
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/viewtopic.php?f=5&t=9
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/memberlist.php?mode=email&t=303
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/post352.html
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/viewtopic.php?t=298&p=344
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/post44.html
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/viewtopic.php?f=11&t=2
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/post47.html
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/viewtopic.php?t=297&p=343
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/viewtopic.php?t=294&p=340
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/viewtopic.php?t=293&p=337
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/viewtopic.php?t=294&p=338
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/viewtopic.php?f=5&t=33
   URL restricted by robots.txt    Nov 4, 2011
http://www.devftp.com/post29.html
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/post25.html
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/post6.html
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/post345.html
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/post217.html
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/post337.html
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/post343.html
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/post41.html
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/post40.html
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/post326.html
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/viewtopic.php?t=292&p=334
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/viewtopic.php?t=292&p=335
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/viewtopic.php?f=8&t=29&start=0&view=print
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/viewtopic.php?f=8&t=293&start=0&view=print
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/viewtopic.php?f=8&t=297&start=0&view=print
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/viewtopic.php?f=7&t=296&start=0&view=print
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/viewtopic.php?f=5&t=8&start=0&view=print
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/memberlist.php?mode=email&t=296
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/viewtopic.php?f=5&t=6&p=28
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/search.php?author_id=57&sr=posts
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/viewforum.php?f=3
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/post43.html
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/post336.html
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/post342.html
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/post344.html
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/viewtopic.php?f=8&t=299&start=0&view=print
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/viewtopic.php?t=5&p=7
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/viewtopic.php?f=11&t=294&start=0&view=print
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/viewtopic.php?f=5&t=5&start=0&view=print
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/memberlist.php?mode=email&t=293
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/memberlist.php?mode=email&t=297
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/memberlist.php?mode=email&t=294
   URL restricted by robots.txt    Nov 3, 2011
http://www.devftp.com/memberlist.php?mode=email&t=292
   URL restricted by robots.txt    Nov 2, 2011
http://www.devftp.com/viewtopic.php?f=6&t=292&start=0&view=print
   URL restricted by robots.txt    Nov 2, 2011
http://www.devftp.com/viewtopic.php?f=8&t=298&start=0&view=print
   URL restricted by robots.txt    Nov 2, 2011
http://www.devftp.com/memberlist.php?mode=email&t=298
   URL restricted by robots.txt    Nov 2, 2011
http://www.devftp.com/viewtopic.php?f=5&t=33&start=0&view=print
   URL restricted by robots.txt    Nov 2, 2011
http://www.devftp.com/viewtopic.php?f=7&t=295&start=0&view=print
   URL restricted by robots.txt    Nov 2, 2011
http://www.devftp.com/memberlist.php?mode=email&t=295
   URL restricted by robots.txt    Nov 2, 2011
http://www.devftp.com/viewtopic.php?f=5&t=32&start=0&view=print
   URL restricted by robots.txt    Nov 2, 2011
http://www.devftp.com/viewtopic.php?f=5&t=5&p=14
   URL restricted by robots.txt    Nov 1, 2011
http://www.devftp.com/ucp.php?mode=resend_act
   URL restricted by robots.txt    Nov 1, 2011
http://www.devftp.com/viewtopic.php?f=3&t=11
   URL restricted by robots.txt    Nov 1, 2011
http://www.devftp.com/viewtopic.php?f=11&t=26&start=0&view=print
   URL restricted by robots.txt    Nov 1, 2011
http://www.devftp.com/viewtopic.php?f=11&t=25&start=0&view=print
   URL restricted by robots.txt    Nov 1, 2011
http://www.devftp.com/viewtopic.php?f=11&t=25
   URL restricted by robots.txt    Nov 1, 2011
http://www.devftp.com/viewtopic.php?p=14
   URL restricted by robots.txt    Nov 1, 2011
http://www.devftp.com/viewtopic.php?f=5&t=6
   URL restricted by robots.txt    Oct 31, 2011
http://www.devftp.com/viewtopic.php?t=19&p=33
   URL restricted by robots.txt    Oct 31, 2011
http://www.devftp.com/viewtopic.php?p=48
   URL restricted by robots.txt    Oct 31, 2011
http://www.devftp.com/post350.html
   URL restricted by robots.txt    Oct 31, 2011
http://www.devftp.com/viewtopic.php?t=194&p=214
   URL restricted by robots.txt    Oct 30, 2011
http://www.devftp.com/viewtopic.php?t=301&p=349
   URL restricted by robots.txt    Oct 30, 2011
http://www.devftp.com/viewtopic.php?f=3&t=219
   URL restricted by robots.txt    Oct 26, 2011
http://www.devftp.com/viewtopic.php?f=10&t=241
   URL restricted by robots.txt    Oct 26, 2011
http://www.devftp.com/viewtopic.php?t=286&p=326
   URL restricted by robots.txt    Oct 26, 2011
http://www.devftp.com/post12.html
   URL restricted by robots.txt    Oct 26, 2011
http://www.devftp.com/memberlist.php?mode=email&t=232
   URL restricted by robots.txt    Oct 25, 2011
http://www.devftp.com/viewforum.php?f=13
   URL restricted by robots.txt    Oct 25, 2011
http://www.devftp.com/index.php?sid=f42fbb0cf06de73741b8cff66cad7392
   URL restricted by robots.txt    Oct 25, 2011
http://www.devftp.com/viewtopic.php?f=3&t=220
   URL restricted by robots.txt    Oct 25, 2011
http://www.devftp.com/post347.html
   URL restricted by robots.txt    Oct 24, 2011
http://www.devftp.com/viewforum.php?f=6
   URL restricted by robots.txt    Oct 24, 2011
http://www.devftp.com/memberlist.php?mode=email&t=300
   URL restricted by robots.txt    Oct 24, 2011
http://www.devftp.com/memberlist.php?mode=email&t=291
   URL restricted by robots.txt    Oct 23, 2011
http://www.devftp.com/viewtopic.php?p=25
   URL restricted by robots.txt    Oct 23, 2011
http://www.devftp.com/viewtopic.php?p=12
   URL restricted by robots.txt    Oct 23, 2011
http://www.devftp.com/viewtopic.php?f=8&t=30
   URL restricted by robots.txt    Oct 22, 2011
http://www.devftp.com/viewtopic.php?p=47
   URL restricted by robots.txt    Oct 21, 2011
http://www.devftp.com/viewtopic.php?p=44
   URL restricted by robots.txt    Oct 21, 2011
http://www.devftp.com/ucp.php?mode=register
   URL restricted by robots.txt    Oct 21, 2011
http://www.devftp.com/viewtopic.php?t=301&p=348
   URL restricted by robots.txt    Oct 20, 2011
http://www.devftp.com/viewtopic.php?t=301&p=350
   URL restricted by robots.txt    Oct 20, 2011
http://www.devftp.com/memberlist.php?mode=email&t=301
   URL restricted by robots.txt    Oct 20, 2011
http://www.devftp.com/viewtopic.php?f=6&t=301&start=0&view=print
   URL restricted by robots.txt    Oct 20, 2011
http://www.devftp.com/viewtopic.php?t=241&p=277
   URL restricted by robots.txt    Oct 19, 2011
http://www.devftp.com/viewtopic.php?f=5&t=32
   URL restricted by robots.txt    Oct 19, 2011
http://www.devftp.com/viewtopic.php?t=225&p=255
   URL restricted by robots.txt    Oct 19, 2011


Any suggestion how to overcome this problem??
praveen
 
Posts: 12
Joined: Thu Jun 23, 2011 3:03 pm
Top

Advertisement

Re: Robot.txt blocking google bot

Postby praveen » Sun Nov 06, 2011 11:36 am

here is my site url - www.devftp.com
Also webmaster tool displaying the parameter table --
Parameter URLs monitored Last configured Effect on content What Googlebot should crawl
t 371 4:29 PM (32 minutes ago) - Let Googlebot decide (default) Edit / Reset
f 247 4:30 PM (32 minutes ago) - Let Googlebot decide (default) Edit / Reset
mode 127 4:26 PM (35 minutes ago) - No URLs Edit / Reset
view 114 4:26 PM (36 minutes ago) - No URLs Edit / Reset
p 103 4:30 PM (32 minutes ago) - Let Googlebot decide (default) Edit / Reset
start 38 4:25 PM (37 minutes ago) - No URLs Edit / Reset
u - 5:02 PM (0 minutes ago) - No URLs Edit / Delete


what should be the default setting for these parameters???
praveen
 
Posts: 12
Joined: Thu Jun 23, 2011 3:03 pm
Top

Re: Robot.txt blocking google bot

Postby Stephen26 » Thu Nov 10, 2011 6:43 pm

praveen wrote:what should be the default setting for these parameters???

They are "Let Googlebot decide (default)". :wink: You really don't have to change anything here for several reasons:

1) You already have rewritten / static URLs thanks to phpBB-SEO, so those settings for dynamic URLs are not important.
2) It can take a quite some time until Google uses them.
3) Sometimes this never happens.
4) These settings in Google webmaster tools are irrelavant for other search engines obviously.
5) Thanks to phpBB-SEO your site uses the canonical element which major search engines support.
6) Most importantly, see 1)
I install and configure phpBB-SEO (and other mods) for a fair price and very fast. If you are interested simply send me a private message.
Stephen26
phpBB SEO Team
phpBB SEO Team
 
Posts: 407
Joined: Mon May 03, 2010 7:06 am
Top
Post a reply

Return to General Discussions

 


  • Related topics
    Replies
    Views
    Last post

Who is online

Users browsing this forum: No registered users and 3 guests


 
Powered by phpBB © phpBB Group

phpBB SEO
Time : 0.112s | 16 Queries | GZIP : Off