Would you recommend to disallow the archive, frostschutz?
(2010-06-14, 06:17 PM)faviouz Wrote: [ -> ]Would you recommend to disallow the archive, frostschutz?
It duplicates the contain, so i'd assume no. (But i don't have a master in SEO
)
(2010-06-14, 06:04 PM)frostschutz Wrote: [ -> ]@Chrissy,
You have a newline problem in that robots.txt - you should find out why and prevent it from happening again. Superfluous newlines can cause any number of problems.
I have what?? Do you mean the empty lines between everything?
Yes, that's what he means. Download a fresh copy of robots.txt and upload it again.
I don't really have a recommendation for that.
You can disallow it - then already indexed archive pages will be removed from Googles index (could hurt you if Google so far preferred indexing the archive over the fully styled threads).
You can allow it - but then you should also set canonical - and let Google crawl it (could hurt you since the amount of time Google spends on a page is limited), and have it bend archive links to the full versions; it would also mean that Google can follow outside links, if someone decides to link to the archive instead of the full threads.
You could also switch it around entirely - disallow the forum itself and allow only the archive to limit indexing to the archive only. Some forum systems work like that, believing that simple formatting is better for Google so it can concentrate on content, and ignore eye candy.
In the end it's more a choice thing than actual recommendation. What you definitely should avoid is double content - so either allow only one or the other, or set canonical.
On my own sites, I currently don't disallow archive. I'd rather remove the archive entirely (for users too) since it does not really serve much of a purpose. I don't really like the archive and its index.php/parameter style URLs.
So, i've noticed that the robots.txt of googleSEO prohibits lots of directories.
It would be beneficial for site indexing but what about adsense-bot???
We need to re-write a robots.txt so that adsense can see all but google-bot can't see all
