(2019-09-03, 10:36 AM)noyle Wrote: [ -> ] (2019-09-03, 08:18 AM)Crazycat Wrote: [ -> ]The only solution I can see is to exclude profile links one by one.
Peharps having a task which will generate the robots.txt and exclude all users profile links could be the simpliest way.
Only do it with a robots.txt file is not enough, at least for Google, for not just future content links but links already existing in Google search results.
Read this help document to see how Google will index if a robots.txt file exists:
Quote:You should not use robots.txt as a means to hide your web pages from Google Search results. This is because, if other pages point to your page with descriptive text, your page could still be indexed without visiting the page. If you want to block your page from search results, use another method such as password protection or a noindex directive.
OP also would like to read carefully about the Google Robots FAQs page, especially the If I block Google from crawling a page using a robots.txt disallow directive, will it disappear from search results? and How long will it take for changes in my robots.txt file to affect my search results? section.
I noticed WordPress made a decision to remove the Disallow: / directive in robots.txt file from WordPress 5.3, announced recently.
In addition, OP may create an extra user group for Spiders/Bots visiting to limit what they can see. To assign a different user group other than Guests or review current settings to bots, go to AdminCP > Configuration > Spiders / Bots.
So if I were to place in a new group, making it only view the forum/threads and not profiles would suffice?
Thanks
(2019-09-03, 10:36 AM)noyle Wrote: [ -> ]In addition, OP may create an extra user group for Spiders/Bots visiting to limit what they can see. To assign a different user group other than Guests or review current settings to bots, go to AdminCP > Configuration > Spiders / Bots.
Not really sure of how Google will react: it will see the links and get an unauthorized page, so it may indexes them but not with a relevant content.
Or tag the site as gray one because of different links leading to the same page...
The way Google is working/indexing is really obscur. Personaly, I don't care about SEO and indexation, my forums are really specific and I don't try to have a lot of users, I just manage help communities
(2019-09-03, 11:24 AM)makpaolo Wrote: [ -> ]So if I were to place in a new group, making it only view the forum/threads and not profiles would suffice?
Thanks
Yep. This is what I mean. You may want to read my comment below.
(2019-09-03, 11:27 AM)Crazycat Wrote: [ -> ]Not really sure of how Google will react: it will see the links and get an unauthorized page, so it may indexes them but not with a relevant content.
Or tag the site as gray one because of different links leading to the same page...
The way Google is working/indexing is really obscur. Personaly, I don't care about SEO and indexation, my forums are really specific and I don't try to have a lot of users, I just manage help communities
You're right.
Besides Google, there are a lot of other search engines and spiders or bots. Not all of them are doing in the same way, or not all of them are good bots regarding complying to
robots.txt. So, we can't for sure whether they'll index a page. The only way is to limit what they can see and this is the forum owner's responsibility.
For MyBB, we can set the permissions to off for unwanted content accessing by user groups. It's also applicable to do it on the http server, but that seems overkilled.