Hi,
Can someone give me a robots.txt for a fresh mybb forum, not customized, I need to block the junk from appearing on google.
Thanks
User-Agent: *
Disallow: /moderation.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /sendthread.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /editpost.php
Disallow: /private.php
Disallow: /task.php
Disallow: /search.php
Disallow: /stats.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /showteam.php
Disallow: /showratings.php
Disallow: /syndication.php
Add/remove what you need.
(2017-06-07, 12:32 AM)Wage Wrote: [ -> ]User-Agent: *
Disallow: /moderation.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /sendthread.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /editpost.php
Disallow: /private.php
Disallow: /task.php
Disallow: /search.php
Disallow: /stats.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /showteam.php
Disallow: /showratings.php
Disallow: /syndication.php
Add/remove what you need.
Thanks, I'll use this, I wouldn't have search.php disallowed though, its useful for bots to search the forums for things to index.
Blocking with robots.txt won't remove the "junk" that's already in the index. You need meta noindex or a 301 redirect for that. And if there happens to be external links pointing to any of those URLs in the robots.txt, they will still be added to the index.
(2017-06-07, 02:17 PM)Ashley1 Wrote: [ -> ]Blocking with robots.txt won't remove the "junk" that's already in the index. You need meta noindex or a 301 redirect for that. And if there happens to be external links pointing to any of those URLs in the robots.txt, they will still be added to the index.
Thanks for the infomation!
(2017-06-07, 04:02 PM)Mr.Kurd Wrote: [ -> ]I use this:
Robot
Red Security
Pretty much the same, most forums use this.