Quote:Do you guys have a recommended robots.txt file as a starting point for MyBB ??
If so please post your file, I think we need to add one so our boards are crawled more efficiently.
Please guys help us again with this topic now that MyBB have search engine friendly URLs. Maybe now we must have 2 version of robots.txt depending of the option of short urls activated/deactivated status?
User-agent: *
Disallow: /admin
Disallow: /image
Disallow: /install
Disallow: /jscripts
Disallow: /calendar
Disallow: /attachment.php
Disallow: /ratethread.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /search.php
Disallow: /newreply.php
Disallow: /private.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /forumdisplay.php
Disallow: /editpost.php
Disallow: /showthread.php
Disallow: /sendthread.php
Disallow: /printthread.php
It is mine.
I hope that this useful to you.
Also, you could add this:
Disallow: /archive
12-14-2008 edit:
User-agent: *
Disallow: /image
Disallow: /jscripts
Disallow: /calendar
Disallow: /attachment.php
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /private.php
Disallow: /printthread.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /search.php
Disallow: /sendthread.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php
Why would you disallow showthread.php?? Would that not mean they can't read threads??
You probably don't want to specify your admin directory there. First reason: there aren't any public links to the admin folder (unless you have put them there yourself), so bots won't find it. Second reason: people who view your robots.txt will find your admin directory.
Yeah, you're right Denis. Thanks for clarify that.
Now, do you discourage the use of robots.txt? Why the community forum doesn't have one? :o
I'm not sure of what files or directories should I disallow :/
Something like this should work:
User-Agent: *
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /misc.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /sendthread.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php
If your forum isn't in the root directory, you'll have to modify the txt accordingly.
We used to have one here, but it disappeared some time ago
Thank you very much
Shouldn't we also disallow member.php to avoid the bot trying to register :o
What about adding a default robots.txt in the next MyBB package?
Just like a htaccess file is being included now for SEO URLs.
The bot visits the registration page loads for me, it can't actually register...
(2008-12-12, 02:22 AM)EviLito Wrote: [ -> ]Shouldn't we also disallow member.php to avoid the bot trying to register :o
Isn't that what CAPTCHA is supposed to stop?!?...
I don't think a default robots.txt should be included - mainly because the developers will need to think of what "default" is, and everybody will differ to this opinion...