Robots.txt file

freepkr · by **freepkr** » Wed Aug 25, 2010 10:03 am

Can someone tell me if my robots.txt file is good and what I should add or not for my wordpress blog. I should have 2 robots.txt files one for the forum and one for the wordpress blog?

User-agent: *
Disallow: /forum/viewtopic.php
Disallow: /forum/viewforum.php
Disallow: /forum/index.php?
Disallow: /forum/posting.php
Disallow: /forum/search.php?
Disallow: /forum/ucp.php
Disallow: /forum/mcp.php
Disallow: /forum/post

Disallow: /forum/member
Disallow: /forum/memberlist.php
Disallow: /forum/faq.php

Hi freepkr

That's right. I appreciate your thread. I also add another points, and that is: The robots.txt file is a good way to prevent this page from getting indexed. However, not every site can use it. The only robots.txt file that the spiders will read is the one at the top html directory of your server. This means you can only use it if you run your own domain. The spiders will look for the file in a location similar to these below:

pageresource.com/robots.txt
javascriptcity.com/robots.txt
mysite.com/robots.txt

ppcmanagementconsultancy.com

thezodiac · by **thezodiac** » Mon Sep 20, 2010 10:56 am

There can be only one robots.txt on your domain and this should be located in the root: -http://www.yourdomain.com/robots.txt

I'm using this robots.txt:

User-agent: *
Disallow: /forums/viewtopic.php?
Disallow: /forums/viewforum.php?
Disallow: /forums/index.php?
Disallow: /forums/posting.php?
Disallow: /forums/groupcp.php
Disallow: /forums/profile.php?
Disallow: /forums/memberlist.php
Disallow: /forums/search.php?
Disallow: /forums/login.php
Disallow: /forums/faq.php

Disallow: /boards/viewtopic.php?
Disallow: /boards/viewforum.php?
Disallow: /boards/index.php?
Disallow: /boards/posting.php?
Disallow: /boards/groupcp.php
Disallow: /boards/profile.php?
Disallow: /boards/memberlist.php
Disallow: /boards/search.php?
Disallow: /boards/login.php
Disallow: /boards/faq.php
Disallow: /boards/index.php?

Disallow: /index.php?
Disallow: /modules/
Disallow: /bulletin-board-lc2/
Disallow: /php-lc1/
Disallow: /galleries-photos/
Disallow: /sub_folder/

freepkr · by **freepkr** » Sun Oct 10, 2010 11:46 pm

Google Webmaster tool told me that I have some crawl errors, most of them in forum posts like this : forum/post14.html

I should change something in my robots.txt file ?

User-agent: *
Disallow: /forum/viewtopic.php
Disallow: /forum/viewforum.php
Disallow: /forum/index.php?
Disallow: /forum/posting.php
Disallow: /forum/search.php?
Disallow: /forum/ucp.php
Disallow: /forum/mcp.php
Disallow: /forum/post

Disallow: /forum/member
Disallow: /forum/memberlist.php
Disallow: /forum/faq.php

freepkr · by **freepkr** » Mon Nov 01, 2010 11:25 pm

Common somebody need an answer. It's so hard ?

katerinak955 · by **katerinak955** » Tue Dec 07, 2010 6:37 am

Hi,
Very nice explanation.
Thanks,
Katerina.

rogerf265 · by **rogerf265** » Tue Dec 28, 2010 9:58 am

Robots.txt file is a special text file that is always located in your Web server's root directory. Robots.txt file contains restrictions for Web Spiders, telling them where they have permission to search. A Robots.txt is like defining rules for search engine spiders (robots) what to follow and what not to. It should be noted that Web Robots are not required to respect Robots.txt files, but most well written Web Spiders follow the rules you define.

mmark · by **mmark** » Mon Jan 24, 2011 2:09 pm

How to Bypass Robots .txt? Anyone?

SeO · by **SeO** » Mon Jan 24, 2011 2:15 pm

mmark wrote:How to Bypass Robots .txt? Anyone?

Bypassing the robots.txt does not make much sense since the robots.txt is only read and followed on a voluntary basis. If the client does not load it, and also it can be loaded but not taken into account.

Goroman · by **Goroman** » Wed Jun 01, 2011 9:28 am

this is not good robots.txt for worpress, coz it's for phpbb3

example

Code: Select all: User-agent: * Disallow: /cgi-bin Disallow: /wp-admin Disallow: /wp-includes Disallow: /wp-content/plugins Disallow: /wp-content/cache Disallow: /wp-content/themes Disallow: /trackback Disallow: /feed Disallow: /comments Disallow: /category/*/* Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Allow: /wp-content/uploads # Google Image User-agent: Googlebot-Image Disallow: Allow: /* # Google AdSense User-agent: Mediapartners-Google* Disallow: Allow: /* # digg mirror User-agent: duggmirror Disallow: / Sitemap: http://www.example.com/sitemap.xml

Stephen26 · by **Stephen26** » Sun Jun 26, 2011 8:25 am

Why is adding Disallow: /viewtopic.php & Disallow: /viewforum.php to the robots.txt necessary when all URLs are html files? And does that improve ranking? Thanks!

Goroman · by **Goroman** » Sun Jun 26, 2011 6:10 pm

this is for google
to not index old urls

Stephen26 · by **Stephen26** » Wed Jun 29, 2011 3:54 pm

I'm not sure when to implent the Disallow: /viewtopic.php? and Disallow: /viewforum.php? in the robots.txt for an already indexed board. After 1 months all new URLs where indexed, old URLs get 301-redirected and their number in the Google index is down by about 60%. Do I have to wait until they are mostly gone or can / shall I add those lines now? I'm at month 2 btw.

Robots.txt file

Robots.txt file

Advertisement

Re: Robots.txt file

Re: Robots.txt file

Re: Robots.txt file

Re: Robots.txt file

Re: Robots.txt file

Re: Robots.txt file

Re: Robots.txt file

Re: Robots.txt file

Re: Robots.txt file

Re: Robots.txt file

Re: Robots.txt file

Re: Robots.txt file

Re: Robots.txt file

Re: Robots.txt file

Who is online

Advertisement