I've got some restricted threads that I'd like Google to be able to index. Anyway I can allow all access to Google, Baidu, Bing etc...?
admin panel >> configuration >> Spiders / Bots --> edit >> User Group
you can edit each bot and add to required group (eg. member group)
(2013-05-23, 06:42 PM).m. Wrote: [ -> ]admin panel >> configuration >> Spiders / Bots --> edit >> User Group
you can edit each bot and add to required group (eg. member group)
Yes, but be aware that it is good Google SEO practice to show the G bot the same that you'll be showing the visitor. What this means is that if someone clicks on a search result to a page that is restricted (but indexed by the bot), you are providing a bad user experience, which is highly frowned upon by Google and which can potentially increase your bounce rate. You can get penalized for this if your site is under manual review, by the way. Be careful.
(2013-05-23, 08:01 PM)Tato Wrote: [ -> ] (2013-05-23, 06:42 PM).m. Wrote: [ -> ]admin panel >> configuration >> Spiders / Bots --> edit >> User Group
you can edit each bot and add to required group (eg. member group)
Yes, but be aware that it is good Google SEO practice to show the G bot the same that you'll be showing the visitor. What this means is that if someone clicks on a search result to a page that is restricted (but indexed by the bot), you are providing a bad user experience, which is highly frowned upon by Google and which can potentially increase your bounce rate. You can get penalized for this if your site is under manual review, by the way. Be careful.
You can easily create a "bot" group and give it the same permissions as guests and assign any spider to it without any red flags being set. I do the same, only so i can style the bot username differently
(2013-05-23, 11:51 PM)pavemen Wrote: [ -> ] (2013-05-23, 08:01 PM)Tato Wrote: [ -> ] (2013-05-23, 06:42 PM).m. Wrote: [ -> ]admin panel >> configuration >> Spiders / Bots --> edit >> User Group
you can edit each bot and add to required group (eg. member group)
Yes, but be aware that it is good Google SEO practice to show the G bot the same that you'll be showing the visitor. What this means is that if someone clicks on a search result to a page that is restricted (but indexed by the bot), you are providing a bad user experience, which is highly frowned upon by Google and which can potentially increase your bounce rate. You can get penalized for this if your site is under manual review, by the way. Be careful.
You can easily create a "bot" group and give it the same permissions as guests and assign any spider to it without any red flags being set. I do the same, only so i can style the bot username differently
So it sounds like this wouldn't work for me. It sounds like you're not giving them the permissions to read the threads only accessible my certain groups.
Regardless, how to add them to a certain group when they don't have a user account? Or is .m.'s solution the one to go with?
then you will be violating google's policies as the spider has different permissions from guest.
(2013-05-24, 04:10 PM)pavemen Wrote: [ -> ]then you will be violating google's policies as the spider has different permissions from guest.
Okay... Well, the spiders are returning a bunch of Soft 404 errors because they can't access those threads. Does that affect rank?
(2013-05-23, 11:51 PM)pavemen Wrote: [ -> ] (2013-05-23, 08:01 PM)Tato Wrote: [ -> ] (2013-05-23, 06:42 PM).m. Wrote: [ -> ]admin panel >> configuration >> Spiders / Bots --> edit >> User Group
you can edit each bot and add to required group (eg. member group)
Yes, but be aware that it is good Google SEO practice to show the G bot the same that you'll be showing the visitor. What this means is that if someone clicks on a search result to a page that is restricted (but indexed by the bot), you are providing a bad user experience, which is highly frowned upon by Google and which can potentially increase your bounce rate. You can get penalized for this if your site is under manual review, by the way. Be careful.
You can easily create a "bot" group and give it the same permissions as guests and assign any spider to it without any red flags being set. I do the same, only so i can style the bot username differently
Absolutely, have the bot as a Guest. What I was referring to was having the bots in the group of registered users that can view pages. Thus, "locked" pages appears on search results but then when you click to the page, it says you have to register.
I've seen a lot of forums do this by the way, but it doesn't mean it is right and you can be penalized for this (I've seen a couple of forums that did when they were manually reviewed although these forums were quite blackhatted). Google are particularly emphasizing on their whole "positive experience", which sometimes works more in their favor than in our favor, despite the fact that we're the ones allowing them to spider us and have good search results instead of garbage search results.
(2013-05-24, 04:52 PM)Gizbeat Wrote: [ -> ] (2013-05-24, 04:10 PM)pavemen Wrote: [ -> ]then you will be violating google's policies as the spider has different permissions from guest.
Okay... Well, the spiders are returning a bunch of Soft 404 errors because they can't access those threads. Does that affect rank?
No. You will get a lot of soft 404s from locked pages. I even get them from the bot trying to crawl a post-reply page
So long as your forum is not a locked forum that the bot can't crawl, you'll be OK.
Essentially, the more freely accessible your pages are, the better your overall SEO. The only problem is that you'll then get a lot of lurkers that don't register and waste your forum resources, so Google is quite happy to leave the dirty clothes to us.