It is currently Fri May 25, 2012 7:52 pm

News News of roBots

Site map of roBots » Forum : roBots

GoogleBot, MSNBot, Yahoo!Slurp ... Everything about indexing Bots, ip lists, User Agents, Crawl and robots.txt.

Cheap soft

Hi!
New soft for a cheap price at our online store!
Adobe Acrobat Capture 3.0 Cluster Edition
Thanks!

Linking Removed.
Read more : Cheap soft | Views : 1340 | Replies : 1


robots.txt for my forum

Hello all, I just created a robots.txt file.

this is my robots.txt file...

http://www.m-benzforum.com/robots.txt

Please add If I need... and Please correct if I'm wrong

Thanks for your help
Read more : robots.txt for my forum | Views : 11174 | Replies : 3


where i must put my robots.txt file

i want to ask some question about robots.txt file

imagine i have installed my forum in this address

-http://subdomain.mydomain.com

where i must put my robots.txt file ?

i've put one of my robots.txt file in WWW folder of my host
and another one in subdomain folder

the robots.txt file for root folder is like this

User-agent: *
Disallow: /subdomain/folder of page

and robots.txt file for subdomain is like this

User-agent: *
Disallow: /folder of page


is it correct? is it necessary to have two robots.txt file or one of them in root folder is sufficient ?

what a bout this ...
Read more : where i must put my robots.txt file | Views : 2205 | Replies : 7


Possible for search engine bot to crawl more than 24 hours?

Anyone know if it's normal to have a search engine bot stay on your site for many hours? I have a Ask Jeeves bot that has been on my site for almost 24 hours now. is it stuck, or can this be normal for a search engine bot to stay so long? RobotStats does not even document it even though the IP's match. at first I thought it just kept visiting... but now I think it's just "parked" on my site.

any ideas?

Thanks

::edit::
Wow, okay it seems as though it finally left my site. but my question still ...
Read more : Possible for search engine bot to crawl more than 24 hours? | Views : 1328 | Replies : 3


About robots.txt

Hi, I've been trying for a while to get rid of old URL's of my site from Google. I submited links for spesific subdirectorys to be removed. but google still seems to be searching for these old URL's. or subfolders inside my old phpbb directory too.

In my robots.txt file I have a Dissallow /forum/ the old name of my phpbb. but the directory does not exist on my site... I figured this was okay since search engines usually scan robot.txt first before indexing a site right?


Why would google still be looking for a subdiectory or URL that I ...
Read more : About robots.txt | Views : 1466 | Replies : 3


 

Login  •  Register


Statistics

Total posts 41127 • Total topics 8563 • Total members 15417