Sharepoint 2007 Crawler and robots.txt

jeremyotten · Joined: 29 May 2008 Posts: 35

Ok Al I have iis6 and my Simple Rewrite MOD is 80 percent working --> thanx to SEO --> Im sure we will get to 100 percent.

Now I have a question. I have secured my forum so that only registered users can login and see the forum contents.

In sharepoint I can give in a username and password. which will be used to crawl the forum. The inly thing that I need to do is that sharepoint cannot crawl the logoff page. Because then It would be in a logon --> logoff cycle and wouldn't be able to crawl the forum.

Do I need to set certain rules in a robots.txt or in de httpd.ini .... with ISAPI_REWRITE 3.1 its called the httpd.conf and can me edited trough the windows ISAPI_REWRITE manager which is installed with the lite version.

Please assist! You would make my day to make this work.

dcz

I don't get it fully here.

You run a private forum (guest do not see any post) and you give some users (bots ?) a logged in access (through sharepoint ?) and you would like to additonally give this account an guest access, so it could see the log-out page ?

What is the point to show a page that tells you can't see anything if it's the only matter ?

jeremyotten · Joined: 29 May 2008 Posts: 35

No this in not what I ment.

The crawler of Sharepoint logs into the forum but only crawl the public forums and not the private forums....

dcz

So sharepoint is indeed a search engine, did not find it in Google ... only this : http://en.wikipedia.org/wiki/SharePoint

Generally speaking, the crawling of the private forum is problematic. phpBB3 does have advanced bot management features which allow you to grant bots the right to crawl private forums, but I do find it risky for several reasons.

First is, this is cloaking, since bots would see what guest would not, and theoretically, this could lead to being blacklisted.

Then, unless you make sure to only rely on ip to match bots, it can be a security problem. If a bot only detected upon it's user agent is set with more right than a guest, anyone faking the UA could have access to data he was not supposed to.

So, I'd personally would not go into this direction, at least until Google clearly gives us the "right" to.

++

jeremyotten · Joined: 29 May 2008 Posts: 35

Sharepoint is from Microsoft Wink

Microsoft Sharepoint Wink

its only used internally.

Sharepoint is to collabirate everything to one only it lakes the function of a good forum. so the only thing we want to do is. Make the forum secure from the outside would --> By you must login to read anything. But we also want sharepoint to index the forum...

jeremyotten · Joined: 29 May 2008 Posts: 35

I search the internet of people who have the same problem.

someone said this:

I came across this thread as I was having the same issue. Were you able to resolve it?

In my case, the problem turned out to be in the web site I was trying to crawl. The developers had the following line of code in the login page's Page_Load event: FormsAuthentication.SignOut() without checking to see if the event was caused by a post back.

Modifying the event to only sign out during the initial (i.e., non-postback) page load fixed the problem.

I know this is not about phpBB but does phpbb does anything similar?

dcz

Ok so I wasn't too far the first time, excuse my ignorance with windows and the Internet, even though it's mainly voluntary Wink

So, all your problematic goes down to :

jeremyotten · Joined: 29 May 2008 Posts: 35

Yes this is not the problem... I can login. It just indexes the pulic content.

dcz

Well, being able to log in does not mean the user account will have enough authorization to browse the whole forum ... Have you set up the Sharepoint user account authorization with enough right to see all your forums in phpBB ?

You can log in with your account and see what's the Sharepoint account can see, it's an option in the profile view when logged in as admin.

++