This goes against SEO but I want to keep some site areas (subforums and custom pages) private and not reachable by search engines.
I made a search in mods.php page but didn't get anything. Isn't there a plugin for this?
You can protect subforums using MyBB's user/groups/permission system.
For custom pages you might have to add custom permission checks.
Friendly search engines can be blocked via robots.txt
(2017-05-17, 10:40 AM)frostschutz Wrote: [ -> ]You can protect subforums using MyBB's user/groups/permission system.
For custom pages you might have to add custom permission checks.
Friendly search engines can be blocked via robots.txt
I do use permissions, but I often see Google or Yahoo bots that are browsing those forums or specific discussions
Adding a disallow rule to robots.txt won't drop the content from Google's index. You will have to add a meta noindex tag first.
This is what you would need to add to your robots,txt for every forum you want excluded from crawling:
User-agent: *
Disallow: /forumdisplay.php?fid=7
Disallow: /forumdisplay.php?fid=8
(2017-05-17, 10:46 AM)ShadowOne Wrote: [ -> ] (2017-05-17, 10:40 AM)frostschutz Wrote: [ -> ]You can protect subforums using MyBB's user/groups/permission system.
For custom pages you might have to add custom permission checks.
Friendly search engines can be blocked via robots.txt
I do use permissions, but I often see Google or Yahoo bots that are browsing those forums or specific discussions
They're treated as guests, just with a human-friendly name. They can't access those forums and threads, but you'll be able to see which threads or forums they're attempting to browse because
you have access to them. You don't need to do anything at all.
I created a Bots group for search engines and set group permissions accordingly. Bad bots get added to the Banned group.
Bots generally should see what's available to guests.
(2017-05-17, 03:23 PM)Ashley1 Wrote: [ -> ]Adding a disallow rule to robots.txt won't drop the content from Google's index. You will have to add a meta noindex tag first.
This is what you would need to add to your robots,txt for every forum you want excluded from crawling:
User-agent: *
Disallow: /forumdisplay.php?fid=7
Disallow: /forumdisplay.php?fid=8
This looks the most technical solution, but reading the reply I quoted below I'm now thinking this may not be necessary at all.
(2017-05-17, 03:45 PM)Nathan Malcolm Wrote: [ -> ]They're treated as guests, just with a human-friendly name. They can't access those forums and threads, but you'll be able to see which threads or forums they're attempting to browse because you have access to them. You don't need to do anything at all.
Makes sense, then I should be fine.
Friendly bots obey robots.txt. Be wary, though, that listing a URL in robots.txt suggests that such a URL exists, and non-friendly bots either don't read robots.txt or ignore its contents.
(2017-05-18, 06:48 PM)laie_techie Wrote: [ -> ]Friendly bots obey robots.txt. Be wary, though, that listing a URL in robots.txt suggests that such a URL exists, and non-friendly bots either don't read robots.txt or ignore its contents.
My guess is that major bots are non friednly bots, and I need a protection from them
