I was in my forums "who's online" area and noticed google there.
No big deal google is always there but this time G is as a guest and not as "google" further i have threads that are locked out for most people.
But when i check where google is i see "Viewing No Permissions Page".
Does that mean google is spidering my threads i dont want them to?
Thanks.
It means they are getting an error stating they are unable to view that page, they are not able to view the page content. Pretty much if you logout of your forums and view your site, that's about all spiders will see also.
Thanks KuJoe.
I was a little concerned as i have information etc that i certainly dont want to be listed.
Thanks again.
If you're really worried, you can use a robots.txt file to tell them where not to index. Pretty much all reputable search engines follow the robots.txt file.
MyBB's file:
http://community.mybboard.net/robots.txt
It also saves you bandwidth by saving them from viewing pages which are of no use to them anyway.
rh1n0 Wrote:If you're really worried, you can use a robots.txt file to tell them where not to index. Pretty much all reputable search engines follow the robots.txt file.
MyBB's file: http://community.mybboard.net/robots.txt
It also saves you bandwidth by saving them from viewing pages which are of no use to them anyway.
Great call rh1n0 i didnt even stop to think about a robots.txt file.
Thanks again.
Hi all,
interesting
.
But noob question, what to do with robots.txt ?
Thanks.
All you do is place the robots.txt file in your root directory and it tells the search engines where they can and can not crawl.
This however does not work for bad bots and scrapers etc.
You need to edit your .htaccess file to do that.
I have a page on one of my sites that goes into .htaccess a little bit.
http://www.damink.com.au/htaccess/ is the link if you want to check it out.
Hope that helps a bit.
Thanks DAMINK,
For now, i'm just testing mybb. But if it's the board i choose you gave me the key to
learn this kind of stuff
.
thanks again.
DAMINK Wrote:All you do is place the robots.txt file in your root directory and it tells the search engines where they can and can not crawl.
This however does not work for bad bots and scrapers etc.
You need to edit your .htaccess file to do that.
I have a page on one of my sites that goes into .htaccess a little bit.
http://www.damink.com.au/htaccess/ is the link if you want to check it out.
Hope that helps a bit.
You've got quite a list there
Some which I don't even have
Also, within class_session.php you can assign a different group id to use for recognized spiders, and thence give spiders different permissions than guests.