I am running a new site (it's still only in development), but I have noticed that the googlebot keeps on ending up in my calendar and just spams add notification in my calendar.php. It is only indexing useless error pages (as it obviously does not have permission to add new notifications or events). It never seems to index anything else.
It's not a big worry yet, but I just wanted to clear this up for future reference: How can I prevent googlebot from doing this?
Find out the I.P address of the GoogleBot and block the ip address using .htaccess
This is the code
----------------------------
Order Deny,Allow
Deny from IP ADDRESS HERE
-----------------------------
Replace the "IP ADDRESS HERE" with the ip address of the Google Bots
Use robots.txt to block it from accessing the calendar. Alternatively you can also add a meta no robots / no index / no follow tag to calendar pages (templates).
Ok cool thanx. I'll go with the robots.txt.
Although, for the .htaccess solution, won't this block googlebot from the entire mybb forums? I still want it to index actual content.
Thank you for the helpful replies
You're not supposed to block Google with .htaccess. It's not easy either - Google has tons of IPs and IPs can change. At the same time you might also be blocking features like Google Translate which some of your users may rely upon without you knowing it.
If you want to block search engines, use robots.txt, that's what it was made for.
And if you don't want to block the search engine entirely, just disallow unregistered users (or bots, if you have a separate group) to access the calendar or add calendar events.
(2011-03-29, 08:55 AM)linguist Wrote: [ -> ]And if you don't want to block the search engine entirely, just disallow unregistered users (or bots, if you have a separate group) to access the calendar or add calendar events.
Ok thanks. This seems sounds to me like it will be the simplest yet most efficient solution. Not exactly sure how I'll do it yet, but when I get there I'm sure I will discover a way. Will it require some coding or can I do it from admin CP?
Update: Cool I found that you can block guest users from the calendar in admin CP. Thanks a lot.