why is robots.txt restricting threads

When I check with google webmaster tools in the crawl errors section it says
Restricted by robots.txt ‎(1,791)‎ |

when you look at the list most of the entries are thread topics with URL restricted by robots.txt next to the entry.

does this not mean that the threads are not being indexed by google so not appearing in search results, and if so is there some way I can change that.

You can view your robots doc at yourforumurl/robots.txt

Gassy wrote:You can view your robots doc at yourforumurl/robots.txt

I have looked at it and cannot see why it would restrict threads.

I think because the permissions of the categories & forums.

Gassy wrote:I think because the permissions of the categories & forums.

that was going to be my next question.
some categories have guest access turned off. I presume that then will restrict google crawlers then.

I have looked at my sitemap.xml and some section do not appear in teh list, they seem to be the ones that are restricted to guests

If you'll set the permissions for a category that the guests can't view topics, this means that the google crawlers (or other search engines) doesn't have access to put your topic links to google. Can you verify this?

Gassy wrote:If you'll set the permissions for a category that the guests can't view topics, this means that the google crawlers (or other search engines) doesn't have access to put your topic links to google. Can you verify this?

I think that you may well be right there, i have enabled the access and will find out when or if they are now crawled.

Flappypaddles wrote:

Gassy wrote:If you'll set the permissions for a category that the guests can't view topics, this means that the google crawlers (or other search engines) doesn't have access to put your topic links to google. Can you verify this?

I think that you may well be right there, i have enabled the access and will find out when or if they are now crawled.

Yeah, after you'll gave the permission to view your topic content, google will index your pages soon.

Hello,

Robot.txt cannot be accessed to users. It is possible to get Google to index your forum, you just need to submit your site to their submit page and they will soon be able to index your forum. You can also do is help index it by posting links on popular sites such as Twitter, Facebook, and other forums. (Don't spam though )