2010-08-19, 06:45 AM
Hello,
I was wondering is it a good thing to use "robots.txt" on my forum, I see a lot of bots, Yahoo, Google,etc browsing the donation page, registering,etc, I don't have a huge problem with that, but it gets indexed and will turn up somewhere on a search engine.
1: Is "robots.txt" correct, or should it be "robots.php" or something else.
2: http://thediscussionzone.com/robots.txt - Is that how it should be? I found this on a guide somewhere and just wanted to get a confirmination that it's correct.
Thank you.
I was wondering is it a good thing to use "robots.txt" on my forum, I see a lot of bots, Yahoo, Google,etc browsing the donation page, registering,etc, I don't have a huge problem with that, but it gets indexed and will turn up somewhere on a search engine.
1: Is "robots.txt" correct, or should it be "robots.php" or something else.
2: http://thediscussionzone.com/robots.txt - Is that how it should be? I found this on a guide somewhere and just wanted to get a confirmination that it's correct.
Thank you.
