(2012-06-09, 09:27 AM)CAwesome Wrote: [ -> ]It needs to be in the domain root, but you can simply change 'Disallow: /sendthread.php' to 'Disallow: /forums/sendthread.php' (etc)
Okay, that makes sense.
Now could I use more than one sitemap (cause of the entry page, plus two different script it's impossible to use just one) and link both in the file? I know I can just submit them at Google Webmaster Tools for Google, but what about other sites?
(2012-06-10, 01:07 AM)Puppyite Wrote: [ -> ]But don’t listen to me, do whatever floats your boat, I’m outta here. 
...What? We did listen, you just made a general point that wasn't correct in all cases.
Many crawlers do obey robots.txt.
Bots might not, but all reputable search engines will. If they didn't, webmasters around the world would be talking about them.
Question about robots.txt
If this is my file located in the root directory:
Quote:User-Agent: *
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /memberlist.php
Disallow: /editpost.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /sendthread.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /calendar.php
Disallow: /*action=emailuser*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*year=*
Disallow: /*action=weekview*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*sort=*
Disallow: /*order=*
Disallow: /*mode=*
Disallow: /*datecut=*
Allow: /
and my forum is in a subdirectory will it still work? My forum is in
www.mysite.com/forum/
Do I need to add /forum in front of all those?
I don't know, but I think it's better to add /forum.
My robots.txt
here and it's works perfectly.
I'm also suggest to use google seo plugin.
[quote='PsuedoK' pid='993441' dateline='1365847514']
Question about robots.txt
If this is my file located in the root directory:
Quote:Do I need to add /forum in front of all those?
Yes u need /forums/
My Robots
Thanks for the tips.
Currently this the robots.txt I am using:
User-agent: *
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /member.php?action=emailuser
Disallow: /member.php?action=login
Disallow: /member.php?action=logout
Disallow: /member.php?action=lostpw
Disallow: /member.php?action=register
Disallow: /memberlist.php
Disallow: /misc.php?action=markread
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendthread.php
Disallow: /showteam.php
Disallow: /stats.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php
PS: I've added an entry to disallow attachment.php to stop Google from trying to download attachments.
Just building on all the previous posters... I got the following robots.txt, uploaded to /public_html (not /public_html/forum/) to prevent unnecessary information getting out to spam bots and to prevent bandwidth hogging... (to be verified).
robots.txt for
http://crypto.country/forum : be warned: I have no idea what I'm doing and if anyone sees problems with this robots file... Do let us know please?
User-agent: *
Disallow: /forum/member.php?action=emailuser
Disallow: /forum/member.php?action=register
Disallow: /forum/member.php?action=login
Disallow: /forum/member.php?action=logout
Disallow: /forum/member.php?action=lostpw
Disallow: /forum/member.php?action=register
Disallow: /forum/misc.php?action=markread
Disallow: /forum/captcha.php
Disallow: /forum/editpost.php
Disallow: /forum/modcp.php
Disallow: /forum/moderation.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/printthread.php
Disallow: /forum/printthread.php*
Disallow: /forum/private.php
Disallow: /forum/ratethread.php
Disallow: /forum/report.php
Disallow: /forum/sendthread.php
Disallow: /forum/task.php
Disallow: /forum/usercp.php
Disallow: /forum/usercp2.php
Disallow: /forum/archive
Disallow: /forum/online.php
Disallow: /forum/calendar.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/memberlist.php
Disallow: /forum/misc.php
Disallow: /forum/online.php
Disallow: /forum/reputation.php
Disallow: /forum/showteam.php
Disallow: /forum/archive/
Disallow: /forum/attachment.php
Disallow: /forum/portal.php*
Disallow: /forum/*nextoldest*
Disallow: /forum/*nextnewest*
Disallow: /forum/*datecut*
Disallow: /forum/*lastpost*
Disallow: /forum/*markread*
Disallow: /forum/syndication.php*
Disallow: /forum/forumdisplay.php*
Disallow: /forum/*sortby*
Disallow: /forum/*action=emailuser*
Disallow: /forum/*action=nextnewest*
Disallow: /forum/*action=nextoldest*
Disallow: /forum/*year=*
Disallow: /forum/*action=weekview*
Disallow: /forum/*action=nextnewest*
Disallow: /forum/*action=nextoldest*
Disallow: /forum/*sort=*
Disallow: /forum/*order=*
Disallow: /forum/*mode=*
Disallow: /forum/*datecut=*
Disallow:/ *next
Disallow:/ *print
Disallow:/ *reply
Disallow:/ *post
Disallow:/ *action
Disallow:/ *user
Allow: /
User-agent: MediaPartners-Google
Allow: /
User-agent: Googlebot
Allow: /?*
User-agent: Baiduspider
Allow: /?*
User-Agent: Yahoo! Slurp
Crawl-delay: 2
Allow: /?*
User-agent: ia_archiver
Allow:
User-agent: YandexBot
Disallow: /?*
User-agent: ichiro
Disallow: /?*
User-agent: sogou spider
Disallow: /?*
User-agent: Sosospider
Disallow: /?*
User-agent: YoudaoBot
Disallow: /?*
User-agent: YetiBot
Disallow: /?*
User-agent: bingbot
Crawl-delay: 2
Disallow: /?*
User-agent: rdfbot
Disallow: /?*
User-agent: Seznambot
Request-rate: 1/2s
Disallow: /?*
Peace!
Devvie
twitter.com/devnullius