MyBB Community Forums

Hello everyone,

I'm trying to implement Lijit Search on my relatively new site (month old) and their staff sent me an email saying I need to remove:

Disallow: /

from robots.txt.

As it is now, they apparently can't index anything. I wonder if Google is having problems too, because despite the site appearing in Google under "site:myurl.com", none of the pages or forum have been indexed. In addition, my Adsense ads still show public service ads and they've been up for like two weeks now. And I'm not violating Google TOS so I'm wondering if Adsense's inability to recognize my content is also related to this robots.txt.

Do I need to delete this line? Or can I flip a button in the Admin CP somewhere? Why would MyBB restrict crawlers by default?

Thank you for your help,

Taylor

Disallow: / means that no bot/crawler will index any part of your site, so if you want it to index pages, then yes, you need to remove it.

Why is that included in the MyBB code by default? I mean, doesn't everyone want their pages indexed?

Sorry if that's a newbie question, but the answer does not seem obvious.

Robots.txt isn't included in MyBB by default. It was added there by someone who has access to your site.

No one has access to my site but me...perhaps I've been hacked?

And looking through my server now, I actually can't find this robots.txt file that Lijit said I needed to look in to delete that Disallow line.

This is very confusing. I'm guessing they saw that line and assumed I had a robots.txt file to contain that line.

The site uses Wordpress for the front page, but I've already checked with the theme developer and they don't package robots.txt or a Disallow line by default either.

Thank you.

The robots.txt that Lijit is talking about is in the wordpress directory. Just delete it, shouldn't do any harm.

Your robots.txt is here

Link - http://gaycarfans.com/robots.txt

According to the guys at Wordpress.org, I just needed to make a simple change to my privacy settings in the Wordpress control panel.

I think it's fixed now.

Thanks for your help!