Allow or block search engine archive?

HB · phpBB SEO Team Joined: 16 Oct 2006 Posts: 809

Recently I was checking page results from google and noticed pages from omgili.com were ranked right beneath my own. Looking more carefully, this "aggregator" had pulled all the forum text and stripped out headers, footers, images, etc. I was not pleased.

Search engines like google and yahoo routinely cache snapshots of websites, presumably as an archival aid should the source site be offline. Nothing wrong with this in principle because (a) the website owner can opt out by adding directives like <meta name="robots" content="noarchive"> to their source pages, and (b) the snapshot is presented in its entirety, including copyright information.

From omgili's FAQ:

dcz

Well, SE's must run a cache to be able to give search results. I think the no-archive tag just prevent them to show it to users, but they still cache pages to be able to provide with search results.

Anyway, SE's cache is no big deal, as they cannot be a search result themselves, it's only a part of the service, and can, as a matter of fact help out when a website is offline for a bit. There is no competition between the source and the cache, the source is the only page showing up in results.

Now, talking about omgili.com, it's a totally different strory. And I am even more than than skeptical about it.

First, there is a copyright matter here, because they build pages with other's content, and even if they add a link to the source, still, they compete with your content, using your content !

So I guess some will find some interest into being listed there, but, IMHO, this is typically the kind of service to avoid for our website.
We do want our content to be browsed on our site, not on others ...

If only they used digest instead of full content, it would be a different story.

And I do not think you want ot fix this using a no-archive tag, as it will prevent regular SE's from doing their job fully.

Anyway, I'm not sure many people would like this much to read threads like this, without knowing anything about the source at first.

I mean, when browsing a forum, you can easily figure out many things about it if you can have a look at how it is organized, and what are the topic usually covered.
Here, You just find a thread out of nowhere, and you do not even know if the topic covered is of the kind the source forum is the best at.

I like the idea of specialized SE's for forums though, like boardtracker.com, but I don't like the idea of our content being used by other this way.

++

HB · phpBB SEO Team Joined: 16 Oct 2006 Posts: 809

dcz

Let's hope they follow their TOS.

I don't really like it to add a line in my page just for one bot, I mean, what if ten web site start doing this ? Will we need ten line in our meta ?

I'm not sure, but my feeling is, in case many pages from my web sites start showing up there, I'd just disallow the bot in the robots.txt, and if not enough ban the agent/ip on the server.

To me, it would look a bit more normal if we had to add a code to allow the copying of our content, even though I understand they would never start their service if they'd ask but ...

To start with, I suggest not to submit our site there. Again, I like the idea of specialized SE's, but here, it's a bit more than that.

++