MyBB Community Forums

(2010-01-24, 08:46 AM)Devilson Wrote: [ -> ]Is there any point to having a robots.txt

Yes. The bigger searchengines respect the robots.txt and will not crawl there. It's a waste of bandwidth and cpu to let them crawl those pages. Calendars for example are links after links after links, so bots keep crawling and crawling. There is no interesting content in calendars anyway, so why would you even let it crawl there and cause load on your server.

Well in that case i guess i will have to make one.

Looks good to me , Have fun

Thanks You guys for the Support , so i uploading this robot for my forums , thanks one again