Hi guys,
I am quite sure I had a robots.txt file, but now it is nowhere to be found. I looked for it because Webmaster tools finds some unreachable pages on my site (although they seem to work pretty well...). For example, yesterday I got several alleged 404 errors caused by a long list of attachments (which, I repeat, look fine to me). So, I went to look for the robots.txt, but it is not there
Should I create one? Would it help?
Any chance of getting some insight?
Thanks, I did look at that thread, but wasn't sure what to choose...
I'll start with what frostshutz says and see if the errors decrease in number. Then will reassess...
One more question, should I add "disallow: /forum/attachment"? I get some errors (although the attachments ARE there...), even though nobody seems to have it.
And since we are at it... should I add the address to the sitemap WITHOUT the "www"?
(2017-10-01, 01:58 PM)Arka Wrote: [ -> ]Thanks, I did look at that thread, but wasn't sure what to choose...
I'll start with what frostshutz says and see if the errors decrease in number. Then will reassess...
One more question, should I add "disallow: /forum/attachment"? I get some errors (although the attachments ARE there...), even though nobody seems to have it.
And since we are at it... should I add the address to the sitemap WITHOUT the "www"?
Your forum is accessible from both www. and without the www. You should set a htacess redirect so that it's one or the other. Then whichever way you chose that's how you set the sitemap reference in robots.txt.
Note, that the errors you see in webmaster tools won't go away by adding lines to robots.txt. In fact the errors will stay there longer because you're blocking access. The better way to handle errors is with meta noindex set on the pages.
(2017-10-01, 02:45 PM)Ashley1 Wrote: [ -> ]Your forum is accessible from both www. and without the www. You should set a htacess redirect so that it's one or the other. Then whichever way you chose that's how you set the sitemap reference in robots.txt.
Note, that the errors you see in webmaster tools won't go away by adding lines to robots.txt. In fact the errors will stay there longer because you're blocking access. The better way to handle errors is with meta noindex set on the pages.
Thanks Ashley,
(why am I not getting notifications?)
Regarding the errors, I am not sure WHY I am getting them... The pages exist fine.
How to go about telling myBB that I do not wish to have "user" or "attachments" indexed? I am not aware of any setting that would do that.
Regarding "www" and "not www", I have been reading here and there and couldn't find a consensus on what to do. I suppose there would need to be a redirect so that, say, "non www" go to "www", but to make sure I don't do anything daft I need some kind of tutorial for myBB.
The user you refer to sounds like the member profile of the google seo plugin. So your URls look like:
/User-John
If you want to stop indexing these, then you have to go into the member_profile template of your theme and add into the head section:
<meta name="robots" content="noindex" />