MyBB Community Forums

Hello!

What is the best robots.txt for MyBB?

Posting in other sections to void your support denial isn't allowed.

I didn't ask for support.. I just ask what is the best robots.txt for MyBB.

That sounds like a support question though.

(2012-01-07, 04:37 AM)weBex Wrote: [ -> ]That sounds like a support question though.

Question removed. Thanks for the support.

Hello,

MyBB doesn't come with any robots.txt file in the package but after seeing some reports over my google webmaster tools account I've created a robots.txt file with:

User-agent: *
Disallow: /private.php
Disallow: /member.php?action=emailuser
Disallow: /search.php

Supreme Robots.txt file that i use after intensively tests done on my forum this will stop the most unwanted content and duplicated content from appearing in google

User-agent: *
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /sendthread.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /archive
Disallow: /online.php
Disallow: /calendar.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /*nextoldest*
Disallow: /*nextnewest*
Disallow: /*datecut*
Disallow: /*lastpost*
Disallow: /*markread*

if you wanna see what useless things google indexes from your site go to google.com type -> site:yoursite.com

This is my robots.txt:

User-Agent: *
Disallow: /moderation.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /sendthread.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /editpost.php
Disallow: /private.php
Disallow: /search.php
Disallow: /memberlist.php
Disallow: /showteam.php


User-agent: dotbot 
Disallow: /

User-agent: 008
Disallow: /

Is it good?

this is my site map it comes with google seo pack http://www.mydesiadda.com/robots.txt

I noticed some errors in Webmaster Panel in this kind of URLs:
j<strong></strong>tellFriend(3009,'')
j<strong></strong>printNews(2855)

How can I block them from crawlers?