Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Not Solved robots.txt for MyBB
#11
Not Solved
You can check HF's robots.txt
User-Agent: *
Disallow: /moderation.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /sendthread.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /editpost.php
Disallow: /private.php
Disallow: /search.php
Disallow: /refer.php

Disallow: /stats.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /showteam.php

Disallow: /showratings.php



User-agent: dotbot 
Disallow: /

User-agent: 008
Disallow: /
[Image: Kewlz.jpg]

^^ Click to check my rank. Big Grin
Reply
#12
Not Solved
Use robots.txt to tell Google where your sitemap is.

sitemap: /sitemap.xml
[Image: hdoE.png]
m1ne.net - coming soon
Reply
#13
Not Solved
And you can check your robots.txt here:
http://tool.motoricerca.info/robots-checker.phtml

One thing I found out is that robots.txt are only read by search engines from the main domain root. So I guess it is useless to have it in the subdomain. Just make sure your robots.txt covers all your subdomains on the main robots.txt.


Reply
#14
Not Solved
Does it work in directories (ie. yoursite.com/forums for instance) or just in the root? Go a site where the forums have to be in a directory so this question is important to me.
I fold for team 52482. Do you fold?
Reply
#15
Not Solved
It needs to be in the domain root, but you can simply change 'Disallow: /sendthread.php' to 'Disallow: /forums/sendthread.php' (etc)
Reply
#16
Not Solved
My robots.txt:

Sitemap: http://your sitemap here/sitemap-index.xml
User-Agent: *
Disallow: /forum/syndication.php*
Disallow: /forum/captcha.php
Disallow: /forum/editpost.php
Disallow: /forum/forumdisplay.php*
Disallow: /forum/memberlist.php
Disallow: /forum/misc.php
Disallow: /forum/modcp.php
Disallow: /forum/moderation.php
Disallow: /forum/newreply.php*
Disallow: /forum/newthread.php*
Disallow: /forum/online.php
Disallow: /forum/portal.php*
Disallow: /forum/printthread.php*
Disallow: /forum/private.php
Disallow: /forum/ratethread.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendthread.php*
Disallow: /forum/showteam.php
Disallow: /forum/*sortby*
Disallow: /forum/task.php
Disallow: /forum/user*
Disallow: /forum/usercp.php*
Disallow: /forum/usercp2.php*
Disallow: /forum/calendar.php
Disallow: /forum/*action=emailuser*
Disallow: /forum/*action=nextnewest*
Disallow: /forum/*action=nextoldest*
Disallow: /forum/*year=*
Disallow: /forum/*action=weekview*
Disallow: /forum/*action=nextnewest*
Disallow: /forum/*action=nextoldest*
Disallow: /forum/*sort=*
Disallow: /forum/*order=*
Disallow: /forum/*mode=*
Disallow: /forum/*datecut=*
Disallow:/ *next
Disallow:/ *print
Disallow:/ *reply
Disallow:/ *post
Disallow:/ *action
Disallow:/ *user
Disallow:/ *=
Disallow: /forum/archive/
Allow: /

User-agent: MediaPartners-Google
Allow: /
Reply
#17
Not Solved
Many crawlers do not obey robots.txt so don’t expect it to be a silver bullet.

Block or remove pages using a robots.txt file
My forum has a higher purpose.
Reply
#18
Not Solved
@up, I think he cares only about Google.
Reply
#19
Not Solved
Completely disagree. Most of the search engines obey robots.txt rules.
Reply
#20
Not Solved
Most of the major crawlers obey robots.txt, Google, Bing, Yahoo, Baidu and many others.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)