MyBB

N.Malcolm · 07-19-2010, 12:57 PM

Robots.txt : http://newsforums.biz/robots.txt

As you can see, it says disallow newreply.php.

But Google has indexed it.

http://www.google.co.uk/search?q=site:ne...4&filter=0

What am I doing wrong?

**Polarbear541** · 07-19-2010, 01:04 PM

It may take a while for Google to index it. You could always use the URL removal tool.

haytoch · 07-19-2010, 01:07 PM

http://www.google.co.uk/search?q=site:ne...4&filter=0
lake i see here google dosen't indexed it
take a look on the image:
http://img836.imageshack.us/img836/8991/google.jpg

N.Malcolm · 07-19-2010, 01:15 PM

@haytoch - See image:

http://img843.imageshack.us/img843/3508/74373823.jpg

faviouz · 07-19-2010, 01:18 PM

newreply.php isn't indexed by Google here either.

No idea why this is happening. Maybe because you're the owner?

**Polarbear541** · 07-19-2010, 01:19 PM

Cache?

N.Malcolm · 07-19-2010, 01:44 PM

@Polar - I have already tried that. I can only see it on google if I click on :

Code:

In order to show you the most relevant results, we have omitted some entries very similar to the 8 already displayed.

If you like, you can repeat the search with the omitted results included.

But it still means that it's indexed, which I don't want.

**Polarbear541** · 07-19-2010, 02:08 PM

As I said you can get a URL removal service in Google Webmaster tools if you're really that desperate. If not then it will just disappear in time.

frostschutz · (This post was last modified: 07-19-2010 03:22 PM by frostschutz.)

You can go to Google Webmaster Tools, and check with their utility if your robots.txt actually does disallow newreply.php URLs. It's blocked only if that says it's blocked.

If it's indeed blocked, you can have use the URL removal utility with yoursite/newreply.php to get all newreply URLs removed. This will only work if it's either blocked or 404. Sites that are not blocked and give 200 OK will stay indexed.

EDIT:

Your robots.txt seems to be interpreted as HTML since it contains this code at the end:

Code:

<!-- www.000webhost.com Analytics Code -->

<script type="text/javascript" src="http://analytics.hosting24.com/count.php"></script>

<noscript><a href="http://www.hosting24.com/"><img src="http://analytics.hosting24.com/count.php" alt="web hosting" /></a></noscript>

<!-- End Of Analytics Code -->

This could break your entire robots.txt. Verify with the Google Webmaster Tools whether it's actually accepted or not.

N.Malcolm · 07-19-2010, 04:56 PM

I just had the Analytics Code removed.

@Polar - I can't find the URL removal service. Could you provide a link?