Why are rasikas pages not indexed on major search engines?

[sais72]

Have the moderators of this forum deliberately preventing search engines such as google from indexing the site? If yes, could you please tell us why? I think it is useful for some of the the great pages and discussions on the site to be visible to others not aware of this community.

PS: Disclosure, i work for google but this question and suggestion is got nothing to do with my employment

[vasanthakokilam]

Good catch. Strange indeed. I will let our admin srkris answer that.

[VijayR]

Have the moderators of this forum deliberately preventing search engines such as google from indexing the site? If yes, could you please tell us why? I think it is useful for some of the the great pages and discussions on the site to be visible to others not aware of this community.

PS: Disclosure, i work for google but this question and suggestion is got nothing to do with my employment

I would be strongly against allowing search engine caching on this forum and think it is a bad idea in this specific context. Although there are many pages here that contain valuable information and facts, there is a lot of "personal opinion" expressed in the forums (in almost every single thread). Allowing all that to be cached by search engines without explicit user permission is simply a gross privacy violation. Period.

[srikant1987]

You don't need an ID here to view contents of any other section than the Lounge. I think all users know that. So it is up to the users to take care of privacy. Indeed, it's the case with all of Internet.

Also, earlier google used to point to some rasikas.org pages -- I think I was led here by some googling, in fact! Possibly it was too difficult to just get the Lounge alone de-indexed or something, so the entire forum was taken off ... Anyway, srkris must be knowing!

[Nick H]

I would be strongly against allowing search engine caching on this forum and think it is a bad idea in this specific context. Although there are many pages here that contain valuable information and facts, there is a lot of "personal opinion" expressed in the forums (in almost every single thread). Allowing all that to be cached by search engines without explicit user permission is simply a gross privacy violation. Period.

Sorry to be the one to break this to you, but that is what the internet is about, and this is a public forum on the internet. Exclamation Mark!

[Rsachi]

I believe rasikas.org site has been set up to prevent contents coming up on search engine indexing. I read this on the Net:
When search engines crawl a site, they first look for a robots.txt file at the domain root. If found, they read the file’s list of directives to see which directories and files, if any, are blocked from crawling. This file can be created with a robots.txt file generator. When you use a robots.txt generator Google and other search engines can then figure out which pages on your site should be excluded. In other words, the file created by a robots.txt generator is like the opposite of a sitemap, which indicates which pages to include.

[sais72]

I believe rasikas.org site has been set up to prevent contents coming up on search engine indexing.

Yes, this is clear. The question is if this was deliberate. If yes, would like to know the reasons.

I do not see a good reason for the pages to be not indexed because most of the site is accessible for casual users (unsigned or unregistered users).

[VijayR]

Sorry to be the one to break this to you, but that is what the internet is about, and this is a public forum on the internet. Exclamation Mark!

Nick, my turn now. Sorry to be the one to remind you, but here is an excerpt from the terms of use that you and I agreed to (be legally bound by) when you created an account on rasikas.org (click on the "register" link on the top right of the main webpage to see these terms).

Code: Select all

"As a user you agree to any information you have entered to being stored in a database. While this information will not be disclosed to any third party without your consent, neither “rasikas.org” nor phpBB shall be held responsible for any hacking attempt that may lead to the data being compromised."

As far as I can tell, Rasikas.org is not owned by Google (or any other company that hosts a search engine), so that would make them a third party to Rasikas. If rasikas.org decides to change its terms of use (which it is entitled to, at any time), that is another matter. But, as they currently stand, it would be a violation of the user agreement / terms of use. [EDIT: I would urge srkris, when he has some time, to consider modifying the terms of use to make this issue more clear.]

Having said all that, of course I realize that there is no such thing as "true privacy" in the Internet. Like many of us here, my personal policy is not to post anything that I would later regret or be embarrassed by, if publicly disclosed. I have absolutely no issue if any of my past posts are cached. But, that is completely orthogonal to my opinion of what constitutes a privacy violation and what does not.

[Nick H]

VijayR, I think that this term relates to our personal information that we might have registered on the site. Most forums have some similar terms. I can't see that it applies to what we post on threads that are accessible by non-members, because that is necessarily available to third parties: it's available to anyone who cares to drop by. Or, another interpretation might be that if we post on what we understand to be an open forum, then we necessarily consent to our words being available to third parties.

Is it possible, though, that being open to search engines would make it easier for the spammers to find us? That would be tedious for all of us, especially the moderators.

Like many of us here, my personal policy is not to post anything that I would later regret or be embarrassed by, if publicly disclosed.

How I wish I had been that sensible! I also wish that I had opted for anonymity --- but that is another thing that some here strongly disagree with. On this, and a couple of other forums, its too late now. I have not been sensible or anonymous --- and if I must shout in the street, people are going to hear. :$

[Rsachi]

Nick,
I think if people are willing to disclose their real names and have no ulterior agenda, that will elevate this forum to an even more dignified and honourable and attractive space for rasikas.
Privacy is an important issue for one's safety and well, privacy, but that does not mean we should hide behind pseudonyms and pseudo-coordinates. My opinion.
Rasikas.org has valuable content and insights from many competent people and I think it is a shame it does not show up in googlenath. So much so that often I do a separate search in rasikas.org for lyrics, composers, musicians and get valuable information!

[Nick H]

Added to which, sometimes Google is just better than native site search. I guess that nobody in the world is as good at search technology as they are.

Until today, I didn't realise that we were not googlable. How do people ever find us? Is it all by word of mouth? How much would the community expand if we were? Ramifications!

[vasanthakokilam]

Let us wait for srkris to write about the history behind the indexing and caching. A while back all of our content was indeed searchable through search engines. I remember writing here that Google could bring up content that was just an hour old. As Nick wrote, Google search is better than the forum search software for some tasks. I did not realize as well that our content is not indexed by Google until pointed out by sais72.

Vijay: i think the clause you are quoting is not related to the content in threads but to any personal information you have given while registering. And this is all part of the same boiler plate end user agreement that every forum that uses this software uses. May be it can use some clarification if there is such a general confusion ( and if so it afflicts a whole lot of forums ).

Be that as it may, there is a difference between indexing and caching. sais72's question is carefully worded indeed and restricted to indexing.
Your post #3 refers to caching. sais72 can clarify if google can be instructed to just index and not cache. ( not that I am advocating one way or the other but it will be good to know ).

( On the legal front regarding caching, Google lawyers hopefully have looked at it thousand ways to make sure their caching of copyrighted content and subsequent dispersal of such content is legally tenable everywhere in the world. 'Instapaper' like personal caching programs while awesome do indeed operate on a legal grey area but so far no one has challenged them in a serious way. But these are all different questions and topics and off syllabus for this thread )

[srikant1987]

Vijay, the quote mentions information you "have entered" when signing up; that obviously means the profile information, because you can't make any posts before the process of signing up is completed.

Only the fields in profile information you choose to make public are made available, and only to other rasikas.org members.

[VijayR]

VK and Srikant: I disagree that the terms of use only refer only to information that you have entered when signing up. I read the entire thing again and, while some may interpret it that way, nowhere does it explicitly mention that. It just says "any information you entered". Anything else is inference and, hence, open to interpretation. In fact, that sentence is part of a paragraph that starts with "You agree not to post any abusive, ..." and continue to talk about posts and topics. So, my interpretation is that the "information you entered" can refer to anything you ever enter on the site.

The point about index vs. cache is well taken (by the way, it is entirely to possible to only index and not cache through appropriate directives in robots.txt or through HTML <meta> headers). In any case, regardless of whether we (or more precisely, srkris) enable indexing and/or caching, I think we should update the terms of use to be a little more explicit.

[Nick H]

What you post on the site is publicly available: it would be a nonsense for a forum management to say "we will not make your publicly-available post available to anyone else." It is available to everyone else. What difference if google gives a helping hand?

This is, of course, the edge of minefield (or a whirpool? Or both?) in which we see Facebook becoming the new word for internet, and, along with Google, trying to own not only the net itself, but every single piece of creative work made, not to mention the unscrupulous who will take what they wish and pass it off as their own. Ours is a relatively private party, due to the speciality of our interest, but there is nothing that any forum management can do to protect itself or its members from such cut and paste antics.

[rajeshnat]

Vk
Just like you even I was not aware that google indexing which was once enabled is disabled. I think it is perfectly ok to index all contents here in google. OUr search can never match google. Let us hear from Srkris, especially why he had disabled the once enabled feature.

[eesha]

indexing has been disabled for rasikas.org using robots.txt .

You can see that at http://www.rasikas.org/robots.txt

[vasanthakokilam]

I do not administer this, we will let srkris look into this, but...
I see
Disallow: /forum_flux/

That forum_flux was the old fourm directory. If the whole site is disabled, then I would expect that line to be just
Disallow: /

[balakk]

Perhaps the simple reason is it attracts undesirable attention.
Spammers for one; a site prominent on Google is a great target for comment spammers.
It's probably a lot of work for a single web admin to control spam.
Second, it may be an attempt at crowd control and managing the influx of new blood.
I'm sure in the season we get tons of new visitors - a site featuring prominently on google can attract far more numbers of users. Increasing the user base might be a good idea for a commercial site; for a community site with negligible(no?) sources of income, it's just additional burden.
It may also attract specimens from norse mythology, i'm told.

For rasikas who are disappointed with the built-in search and lack of google guidance, try searching on bing.
Use the search term and "site:rasikas.org".
Example to search for harimau:

http://www.bing.com/search?q=harimau+site%3Arasikas.org

It returns reasonably relevant results. Happy Christmas and New Year folks!

[Nick H]

Searching for harimau returns reasonably relevant results

]

I'm sure it does!

[rshankar]

Searching for harimau returns reasonably relevant results

Shouldn't it be 'unreasonably'?

[srkris]

It is true that the site is not well indexed by google, bing seems to have it indexed though.

[mahavishnu]

Srkris, is that by design? Perhaps it is time to open the forum to be indexed by google...

[sais72]

Hi srkris and other moderators,

I looked a bit more into this. When Googlebot requests a forum page, the server returns a page different from what normal users see. Googlebot gets the message "You are not authorised to read this forum.".

You can see this yourself using the Fetch as Google function in Webmaster Tools:

http://support.google.com/webmasters/bi ... wer=158587

The server returns this message for both Googlebot and Googlebot-Mobile.

You can also see the issue by changing the user-agent of your browser or using a tool that fetches pages using different user-agents, such as web-sniffer.net:

Fetch with a Googlebot user-agent:
http://web-sniffer.net/?uak=9&url=http% ... 6t%3D20570

Fetch with an IE 6 user-agent:
http://web-sniffer.net/?uak=1&url=http% ... 6t%3D20570

Treating Googlebot differently from users like this is considered cloaking, which is a violation of Google’s Webmaster Guidelines. Please see this Help Center article about cloaking:

http://support.google.com/webmasters/bi ... swer=66355

To fix this, make sure that your server and the forum’s content management system do not treat Googlebot differently from how they treat normal users.

Hope this helps,