2015-06-15, 04:43 PM
According to Bing Webmaster tools bingbot only takes note of the first directive and ignores the others so if you use disallow and then allow, the allow directive will be ignored. It makes sense as if you tell bingbot what you do not allow to be indexed it stands to reason that everything else is allowed.
"if you wanna see what useless things google indexes from your site go to google.com type -> site:yoursite.com" - T0p3a
I did this and was amazed at the links Google indexed despite using the Google SEO plugin. I spent an hour deleting these links using the 'remove url tool' but googlebot has indexed a lot more (forgot to add new directives to robots.txt doh)
"if you wanna see what useless things google indexes from your site go to google.com type -> site:yoursite.com" - T0p3a
I did this and was amazed at the links Google indexed despite using the Google SEO plugin. I spent an hour deleting these links using the 'remove url tool' but googlebot has indexed a lot more (forgot to add new directives to robots.txt doh)