MyBB Community Forums

Sorry to borrow you again about a new thread about googlebot and robot.txt bu I had to. I could not continue my thread I started more as a year ago, its closed for more as a year ago (LOL).

I want to find out if a robot.txt file realy helps to lowing my bandwidth.
Now it is for the forum as follow.

Quote:Advanced Search
Date Apache Email Ftp DirectAdmin Other Total Sent Emails
2008 07 01 11.7 MB 67.6 KB 0.00 KB 0.00 KB 0.00 KB 11.8 MB 45
2008 07 02 405.9 MB 75.1 KB 0.00 KB 0.00 KB 0.00 KB 406.0 MB 49
2008 07 03 388.3 MB 2.88 MB 0.85 KB 0.00 KB 0.00 KB 391.1 MB 1130
2008 07 04 518.0 MB 79.6 KB 56.2 KB 0.00 KB 0.00 KB 518.1 MB 47
2008 07 05 356.3 MB 0.00 KB 0.00 KB 0.00 KB 0.00 KB 356.3 MB 0
total 1.64 GB 3.09 MB 57.0 KB 0.00 KB 0.00 KB 1.64 GB 1271

I create this robot.txt file

Quote:<meta name="robots" content="index, nofollow">

and add it in the root.

I hope that robot visit my page and only the index. So my page name is in the search engines. All threads and post the robot do not may folow.
Is this code OK en would this help when you have a bandwidth usage just like I have.

The toal usage of database is 89 mb (from 3000) 17 mb is from the software and images. Time online since april 2007.

http://www.ikf-forum.eu for more visitting details.

that will prevent SE from giving your site any rank!
if you don't care and you just want to prevent SE bots from access your site
put this code inside robots.txt file and upload it to your root folder

User-agent: *  
Disallow: /

and in your headerinclude template add at the top of it the following

<meta name="robots" content="noindex, nofollow" />

I only want that those spider visit my indexpage where I have add the key-words.
I think it is not neccesary that my whole site will be folowed (89 mb) every day by 2 or more spiders.

the home page will not appear in alot of search result and will not give you a good rank.
becuase the home page contain only the whole site navigation.
but the entire pages contain the content of your forum. so it'll give you more visitors..
and it's your choice.. if you still want to make the spiders index only your home page.
remove the robots.txt and remove the meta tage I just give to you above and do the following steps..
edit global.php (in mybb root folder)
find

// Set up some of the default templates

add after

$mybb->settings['cur_file'] = basename($_SERVER['PHP_SELF']);
if ($mybb->settings['cur_file'] != 'index.php'){
$noindex = '<meta name="robots" content="noindex, nofollow" />';
}
else{
$noindex ='';
}

then add at the top of your headerinclude template {$noindex}

It should be a "robots.txt" file. Here's mine, I like to keep them out of my calendar since they like getting stuck in it:

# keep robots out of places that have no interesting content
# these places are still public and accessable. just not good
# places for search engines to hit

User-agent: *
Disallow: /forum/calendar.php

User-agent: WebReaper
Disallow: /

Does anyone have a basic robot file to show me not sure I am doing it right

thanks

Umm, see above?

# keep robots out of places that have no interesting content
# these places are still public and accessable. just not good
# places for search engines to hit

User-agent: *
Disallow: /forum/calendar.php

User-agent: WebReaper
Disallow: /


this a good template ? word for word ...Ya ?

Well, it's slightly tailored to my setup. You can check out some more info on robots.txt and work it out from that.

Thats a start for me ....thanks