Running 1.6.1
new installation
www.bunnyboard.org/forum
Is it possible to block search engines from crawling an individual forum within Mybb?
The specific requirement is that one of my users writes a blog in a sub forum and doesn't want it crawled so that she can talk about family, friends, work etc without it all appearing on google etc
Obviously I would only want to block off that area - and would still want the rest of my Mybb to be crawled as normal.
Having researched this before asking I've seen advice about creating a robots.txt in my server's root directory, but this would block off the whole forum not just the blog bit.
Hope this makes sense!
Thanks
In robots.txt, you need to define nofollow for the directories/pages you want to block crawler access to, ie:
User-agent: *
Disallow: /directory/
Disallow: /forum-43/
More info here:
http://www.robotstxt.org/orig.html#format
Google also has a nifty little robots.txt creation tool, fast and easy too use
.
Thanks for the reply Fizz!
The address of the specific blog is:
http://www.bunnyboard.org/forum/forumdisplay.php?fid=50
So if I'm understanding this right - to block off this person's blog, my robots.txt file would look like this?
User-agent: *
Disallow: /forum/
Disallow: /forum-50/
Cheers
Mostly, though I was just giving examples. Disallow: /forum/ completely disallows that directory and all sub-directories, so you'll probably want to delete that line.
As for the specific forum I'm not sure, try Disallow: /forum/forumdisplay.php?fid=50 and let me know if that works (i honestly don't know xD)
OK mate - thanks for pointing me in the right direction!
I'll leave this query open for now in case anyone else can help me with the specifics
Cheers!
The problem is that while this will block the forum, it will still allow the threads themselves to be indexed. You'd need a plugin that sets a meta robots noindex tag for any thread within this forum. The alternative would be to make the forum inaccessible to guests, thereby making it a semi-private blog.
(2011-02-06, 11:22 AM)frostschutz Wrote: [ -> ]The problem is that while this will block the forum, it will still allow the threads themselves to be indexed. You'd need a plugin that sets a meta robots noindex tag for any thread within this forum. The alternative would be to make the forum inaccessible to guests, thereby making it a semi-private blog.
So if I remove the forum from Guest viewing through:
adminCP - forums & posts - options - permissions
this would also keep search engines out?
Cheers
It's been suggested that removing access to areas of MyBB from Guests also means those areas will not be crawled by Search Engines...
Are there any support team staff around who can confirm this for me?
Thanks folks
Aww sorry to bump this from page 4 guys but I really do need the assurance of a definitive answer
In Google Webmaster Tools, there's a "Fetch as GoogleBot" feature. You can have it fetch a page and show you what GoogleBot sees there. If guests don't have permissions to read, and if Google is treated as a guest, a no permissions page is what GoogleBot should see.