MyBB Community Forums

Full Version: About Crawling
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
Hey,I have enabled a plugin to hide thread content from guests,but can google crawl in spite of this?
I changed the group permissions of Google;to registered but is this enough for google bot to crawl the pages correctly?
It depends on plugin as well, does it treats bots as Guest or it treats it as a user.
Which plugin are you using ?
(2016-04-02, 11:59 AM)sterg Wrote: [ -> ]Hey,I have enabled a plugin to hide thread content from guests,but can google crawl in spite of this?
I changed the group permissions of Google;to registered but is this enough for google bot to crawl the pages correctly?

You should test the pages in webmaster tools - there is a function called fetch as googlebot, then you will be able to see what the return code is when google tries to crawl the page.
(2016-04-02, 12:26 PM)WallBB Wrote: [ -> ]It depends on plugin as well, does it treats bots as Guest or it treats it as a user.
Which plugin are you using ?

I use this plugin: http://community.mybb.com/mods.php?action=view&pid=313
Also add disallow rule in your robots.txt

this will block all search engines from crawling your forum...

User-agent: *
Disallow: /

Disable only one search engine (Baiduspider)

User-agent: Baiduspider
Disallow: /
(2016-04-02, 01:14 PM)darkhk3r Wrote: [ -> ]Also add disallow rule in your robots.txt

this will block all search engines from crawling your forum...

User-agent: *
Disallow: /

Disable only one search engine (Baiduspider)

User-agent: Baiduspider
Disallow: /
Why would I wanna do that?
What's the purpose of hiding content from guests and then trying to get spiders to crawl the content? That type of content will never appear in the index, because users will be redirected to a no permissions page.