MyBB Community Forums

Currently everything is allowed
User-agent: *
Disallow:

What are the unnecessary things that i can restrict apart from these listed below

User-agent: *
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php

Whatever you don't want to be indexed with google etc....

If you're using Google SEO, restrict /user-* . Your users don't need to appear in Google, and it just facilitates unnecessary indexing and a route for spamming, IMO.

robots.txt Tips:

1. Find robot User-agent names in your Web log
2. Always follow the capitalization of the agent names and the file and directories. If you disallow /IMAGES the robots will spider your /images folder
3. Put your most specific directives first, and your more inclusive ones (with wildcards) last

Make sure you don't include unnecessary spaces or #comments in the file because those might mess up the way the lines are interpreted by Google

If u any thing not want index in search engine then notify in robots.txt

I would recommend disallowing the index of the calender. For some reason, Google LOVES to crawl my calender. It's pointless for most forums unless you do a lot of events.

Block robots from accessing member profiles.