The forum of the forums
Welcome to the Official Support Forum of Forumotion!

To take full advantage of everything offered by our forum, please log in if you are already a member, or join our community if you've not yet.



Create a free forum like this one.

why is robots.txt restricting threads

View previous topic View next topic Go down

In progress why is robots.txt restricting threads

Post by Flappypaddles on January 8th 2012, 11:36 am

When I check with google webmaster tools in the crawl errors section it says
Restricted by robots.txt ‎(1,791)‎ |

when you look at the list most of the entries are thread topics with URL restricted by robots.txt next to the entry.

does this not mean that the threads are not being indexed by google so not appearing in search results, and if so is there some way I can change that.

Flappypaddles
Forumember

Posts : 249
Reputation : 1
Language : english

http://inaflap.forumotion.co.uk

Back to top Go down

In progress Re: why is robots.txt restricting threads

Post by Guest on January 8th 2012, 1:28 pm

You can view your robots doc at yourforumurl/robots.txt

Guest
Guest


Back to top Go down

In progress Re: why is robots.txt restricting threads

Post by Flappypaddles on January 8th 2012, 2:16 pm

Gassy wrote:You can view your robots doc at yourforumurl/robots.txt

I have looked at it and cannot see why it would restrict threads.

Flappypaddles
Forumember

Posts : 249
Reputation : 1
Language : english

http://inaflap.forumotion.co.uk

Back to top Go down

In progress Re: why is robots.txt restricting threads

Post by Guest on January 8th 2012, 2:20 pm

I think because the permissions of the categories & forums. Wink

Guest
Guest


Back to top Go down

In progress Re: why is robots.txt restricting threads

Post by Flappypaddles on January 8th 2012, 2:30 pm

Gassy wrote:I think because the permissions of the categories & forums. Wink

that was going to be my next question.
some categories have guest access turned off. I presume that then will restrict google crawlers then.

I have looked at my sitemap.xml and some section do not appear in teh list, they seem to be the ones that are restricted to guests


Flappypaddles
Forumember

Posts : 249
Reputation : 1
Language : english

http://inaflap.forumotion.co.uk

Back to top Go down

In progress Re: why is robots.txt restricting threads

Post by Guest on January 8th 2012, 3:18 pm

If you'll set the permissions for a category that the guests can't view topics, this means that the google crawlers (or other search engines) doesn't have access to put your topic links to google. Can you verify this? Wink

Guest
Guest


Back to top Go down

In progress Re: why is robots.txt restricting threads

Post by Flappypaddles on January 8th 2012, 5:43 pm

Gassy wrote:If you'll set the permissions for a category that the guests can't view topics, this means that the google crawlers (or other search engines) doesn't have access to put your topic links to google. Can you verify this? Wink

I think that you may well be right there, i have enabled the access and will find out when or if they are now crawled.

Flappypaddles
Forumember

Posts : 249
Reputation : 1
Language : english

http://inaflap.forumotion.co.uk

Back to top Go down

In progress Re: why is robots.txt restricting threads

Post by Guest on January 8th 2012, 8:07 pm

@Flappypaddles wrote:
Gassy wrote:If you'll set the permissions for a category that the guests can't view topics, this means that the google crawlers (or other search engines) doesn't have access to put your topic links to google. Can you verify this? Wink

I think that you may well be right there, i have enabled the access and will find out when or if they are now crawled.
Yeah, after you'll gave the permission to view your topic content, google will index your pages soon. Wink

Guest
Guest


Back to top Go down

In progress Re: why is robots.txt restricting threads

Post by MrMario on January 9th 2012, 6:52 pm

Hello,

Robot.txt cannot be accessed to users. It is possible to get Google to index your forum, you just need to submit your site to their submit page and they will soon be able to index your forum. You can also do is help index it by posting links on popular sites such as Twitter, Facebook, and other forums. (Don't spam though Wink )

MrMario
Helper
Helper

Male Posts : 22186
Reputation : 1830
Language : test

http://test.com

Back to top Go down

View previous topic View next topic Back to top


 
Permissions in this forum:
You cannot reply to topics in this forum