I am using ubersuggest and they are saying they are preparing upcoming Google SEO algorithm update in Summer then I got the error "1600 pageshave blocked meta tag robots or X-Robots-Tag HTTP header". I am not sure if this is related to upcoming Google SEO stuffs they say, but there were no such issue just before today in their SEO audit system. It looks like all my pages generated from MYBB are getting this error.
How do I fix this? I think this can be perhaps critical issue.
Thank you very much.
(2021-05-29, 03:37 AM)online0227 Wrote: [ -> ]"1600 pageshave blocked meta tag robots ..."
This means you have 1600 pages either blocked with robots.txt rules, or maybe 1600 pages with the meta tag noindex.
Crawl-delay: 60
User-agent: *
Disallow: /captcha.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /printthread.php
Disallow: /task.php
Disallow: /archive/
Disallow: /online.php
Disallow: /calendar.php
Disallow: /reputation.php
Disallow: /*nextoldest*
Disallow: /*nextnewest*
Disallow: /*datecut*
Disallow: /*lastpost*
Disallow: /*markread*
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /sendthread.php*
Disallow: /usercp.php*
Disallow: /usercp2.php*
Disallow: /newreply.php*
Disallow: /newthread.php*
Disallow: /editpost.php
Disallow: /private.php
Disallow: /search.php
Disallow: /memberlist.php
Disallow: /showteam.php
Disallow: /stats.php
Disallow: /showratings.php
Disallow: /syndication.php*
Disallow: /misc.php
Disallow: /*sortby*
Disallow: /*action=emailuser*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*year=*
Disallow: /*action=weekview*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*sort=*
Disallow: /*order=*
Disallow: /*mode=*
Disallow: /*datecut=*
Disallow: /usuario-*?action=emailuser
Disallow: /ratethread.php
Disallow: /attachment.php
User-agent: MediaPartners-Google
Allow: /
User-agent: dotbot
Disallow: /
User-agent: 008
Disallow: /
Sitemap:
https://www.mydomain.com/sitemap-index.xml
Exactly why my pages are blocking robots.txt? Is that caused by lots of "Disallow" things in my robots.txt?
And I think I shouldn't be including meta tag noindex in headerinclude because it will block SEO crawl...? I want Google to show my page in public.
Sometimes it's useful to put block rules into robots.txt and/or to put meta noindex on some pages. The fact that you have 1600 pages showing as blocked in a report is not necessarily a bad thing.
Your content is mainly your threads and forums, but mainly your threads. So the most important page(s) not to have blocked is showthread.php
(2021-05-29, 01:39 PM)Ashley1 Wrote: [ -> ]Sometimes it's useful to put block rules into robots.txt and/or to put meta noindex on some pages. The fact that you have 1600 pages showing as blocked in a report is not necessarily a bad thing.
Your content is mainly your threads and forums, but mainly your threads. So the most important page(s) not to have blocked is showthread.php
According to showthread.php, my SEO audit says followings have blocked meta tag robots or X-Robots-Tag HTTP header:
https://www.mydomain.com/showthread.php?mode=linear&tid=14&pid=19
https://www.mydomain.com/Thread-Hi-guys--1245?action=lastpost
https://www.mydomain.com/attachment.php?aid=25
https://www.mydomain.com/showthread.php?mode=threaded&tid=146&pid=336
https://www.mydomain.com/Thread-DISCORD-SUGGESTION-Welcome-Channel--32?action=nextnewest
* My mybb uses converted url that redirects pages with "showhread.php" to "Thread-titleOfThread--threadID" (Google SEO plugin does this convert).
Strangely there is no standard form of access to showthread... all comes with strange parameters after "?" such as "action=lastpost" and "mode=linear&tid=14&pid=19", and "action=nextnewest".
I went through reading a list of url that my SEO audit worries about, but
none of them have standard url for showing thread like "Thread-titleOfThread--threadID".
And some other list that my SEO audit says followings have blocked meta tag robots or X-Robots-Tag HTTP header:
https://www.mydomain.com/archive/index.php/forum-189.html
https://www.mydomain.com/showthread.php?mode=linear&tid=27&pid=44
https://www.mydomain.com/printthread.php?tid=135
https://www.mydomain.com/ratethread.php?tid=164&rating=2&my_post_key=6e5b8ba23ecc53e4066b55c52b138ef3
All of them have strange parameter after "?"
except urls like "/archive/blah blah blah...."
So does this mean that my mybb engine is working correctly so that I don't need to fix anything?
Thank you very much.
(2021-05-29, 01:51 PM)online0227 Wrote: [ -> ]So does this mean that my mybb engine is working correctly so that I don't need to fix anything?
Yes. At a quick read through your robots.txt file and the lists of URLs your SEO audit reports are being blocked, nothing seems seriously problematic to me. The URLs that are being blocked should be blocked, and the rules that are blocking them are appropriate. There do seem to be a few redundant rules, but they shouldn't cause problems.
Showthread.php and its pages, if your thread goes into multiple pages, should not be blocked. i.e they should be crawled and indexed, which appears to be the case.
(2021-05-29, 02:09 PM)Laird Wrote: [ -> ] (2021-05-29, 01:51 PM)online0227 Wrote: [ -> ]So does this mean that my mybb engine is working correctly so that I don't need to fix anything?
Yes. At a quick read through your robots.txt file and the lists of URLs your SEO audit reports are being blocked, nothing seems seriously problematic to me. The URLs that are being blocked should be blocked, and the rules that are blocking them are appropriate. There do seem to be a few redundant rules, but they shouldn't cause problems.
Thank you very much!
(2021-05-29, 02:20 PM)Ashley1 Wrote: [ -> ]Showthread.php and its pages, if your thread goes into multiple pages, should not be blocked. i.e they should be crawled and indexed, which appears to be the case.
I don't think pages in each threads are blocking SEO audit bots, but only threads with "?action=lastpost" querystring are blocking SEO audit bot, I guess. Because...
I went through reading those 1600 pages urls but none of them have blocked proper urls such as:
"/Thread-titleOfThread--919" // 1st page
"/Thread-titleOfThread--919?page=2" // 2nd page
I haven't seen urls like above 2 in their SEO audit reports but only threads that comes with "?action=lastpost" are in their report.
So I hope that I would be fine...
Thank you very much
What might be more useful is to add a canonical tag, which I believe (or more, assume) Google SEO does. That would tell Google all the other various showthread.php URLs with pagination and display type etc are all the same resource, and then the robots.txt rules (for those at least) would be redundant as Google would already know not to index them.