Thought of this when I woke up today and I think it would be neat if like WordPress in the installation there was a checkbox wether they wanted to appear in search engines or not.
Deny Google bot's IP by using htaccess.
(2011-11-10, 03:54 PM)crazy4cs Wrote: [ -> ]Deny Google bot's IP by using htaccess.
Robots.txt exists for a reason. No need for it to be core.
Yes, that is what I said.
(2011-11-10, 04:07 PM)Aristotle Wrote: [ -> ] (2011-11-10, 03:54 PM)crazy4cs Wrote: [ -> ]Deny Google bot's IP by using htaccess.
Robots.txt exists for a reason. No need for it to be core.
Why it shouldn't be as a core feature? It would help newbies.
I don't think much webmasters would want to hinder search engines from exploring their sites. The others who want, may use htaccess (robots) method.
I don't think this would be a core
You can use robots.txt or ban search engine bot's IP addresses but i don't know why you should like it. you lost many traffic by doing it.
New users who don't know .htaccess or robots.txt would use it. +1