MyBB Community Forums

Hi,

After 18 months of building my website and learning to manage my VPS my site is now live but not open to registrations yet.  I created my sitemap manually, configured my htaccess and added a few lines to httpd.conf to stop bad robots.  I have Google and Bing Webmaster Tools accounts.

Google successfully crawled my website https://classified-top-secret.uk and indexed it but Bing is having problems.  I joined a webmaster forum and nothing has been successful despite good advice and changing robots.txt and ht access.

It is very frustrating as Bing only gives sweeping 'generic' failure codes which provide very little assistance in identifying the problem.  Both shodan and megindex (which I consider bad bots) have ignored my robots.txt file and crawled my site and yet bingbot is being so perdantic.

I am running out of options.  The fact that both bingbot and msnbot are showing in my apache access logs (accessing robot.txt and sitemap.xml but not crawling) would suggest that I do not have a firewall block.  I am just wondering if there is any MyBB setting or a plugin which could be causing this.

I am using MyBB 1.8 with Stop Forum Spam API Key. The plugins I use are:

AN Only Registered Links;
Announcement;
AutoMedia;
Cookie Law;
Disposable Email Blacklist;
DVZ Secure Content;
IP2Location IP-Country Blocker (registration only)
MyBB 2FA
Private Message Log
Report Private Messages
MyBB SafeLink
Strict Username
ThreadLimiter

Thanks for any help.

Talen

---

Just a quick note to say I deactivated all plugins and the problem remains.

First time looking at this issue.. probably 'em bots don't have enough permission to look through things?

I have ruled out firewall and modsecurity by disabling them briefly. I have contacted my VPS host and been assured there is no network blocking of bingbots IP addresses. The strange thing is Seo Analyzer in Bing Webmaster Tools fetches my site without problem and yet Fetch as Bingbot fails. I have even removed robots.txt and htaccess and the problem remains.

I was reading today that bingbot does not fully support SNI causing fetch problems. I am going to put my forum on a dedicated IP tomorrow to see if this solves the issue.

Just to update anyone following this thread. I have found the problem and it relates to https.

I removed the permanent redirect directive in htaccess (recommended in this MyBB thread). I was able to use Fetch as Bingbot successfully on the http version of my site but not the https version.

I could navigate my http site but not my https site (links produced 404s). The thread mentioned suggests altering all the http to https in the hard code.

I decided to try a permanent redirect to https in my virtual server settings in httpd.conf but bingbot was not having any of it. Coincidentally I found that MyBB's SEO setting did not work with this change producing 404s.

I have come to the conclusion that Bingbot will not entertain redirects. So I have 2 choices -

Have a non-https site except for registration and logging in.

or

Keep my site as https and be thankful that googlebot is not so perdantic.

I think I am going to go for the latter.

Well, the way I'd look at it now is:

Do you want Google ranking or Bing ranking more?

I would go with the former. And since HTTPS is going to be a factor soon, I would stick with HTTPS.

I completely agree with you.

After a night of reading I made a slight modification to the one suggested by adding a permanent redirect (301)

RewriteEngine On
RewriteCond %{SERVER_PORT} 80 
RewriteRule ^(.*)$ https://mywebsite.com/$1 [R=301,L]

(2015-05-20, 10:50 PM)talen_j Wrote: [ -> ]I completely agree with you.

After a night of reading I made a slight modification to the one suggested by adding a permanent redirect (301)

RewriteEngine On
RewriteCond %{SERVER_PORT} 80 
RewriteRule ^(.*)$ https://mywebsite.com/$1 [R=301,L]

Interesting. I've been having the same issues and Bing claims our site is blocking them. Of course, it isn't. I assumed it was related to https. But our https redirect is, and always has been, 301. Based on this thread it seems that when you changed from 302 to 301 your issue with Bing was resolved?

Yes, because HTTP 301 is Moved Permanently.