Hey there.
Was just looking at your robots.txt file:
User-Agent: *
Disallow: /moderation.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /sendthread.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /postings.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /editpost.php
Disallow: /private.php
Disallow: /search.php
Is there any specific reason why you disallowed so many files and such? Also, how come you didn't disallow the admin directory?
EGS Wrote:Is there any specific reason why you disallowed so many files and such? Also, how come you didn't disallow the admin directory?
Then people could just read the robots.txt and find the admin directory.
Also there are no links from the front end to the Admin CP from the bot's point of view so they shouldn't spider it anyway.
Most of the functions in those scripts can't even be accessed by a spider because it's not logged in.
Also, there's not really any unique content on any of those pages. No point in a spider wasting bandwidth and appearing in your online list to download a useless page.
Relooking at this file, what is usercp2.php anyway?
It handles subscriptions and favourites (adding, removing them)
Can i also put the same above mentioned lines in my robot.txt, will it ban only search engine from accessing it ? will it be a problem to user of forum ??
Alternatively, you could add to the head of all pages you want hidden:
<meta name="robots" content="noindex, nofollow, noarchive" />
The benefit being, not only will robots not spider them, but it won't reveal them to naughty people either.
Tiger Wrote:will it be a problem to user of forum ??
No, browsers do not parse the robots.txt file.
Disallow: /postings.php
postings.php? I don't see any postings.php o.0
I guess the file should be updated. Originally moderation.php was postings.php