Twitter not allowed by robot.txt
3 posters
Page 1 of 1
Twitter not allowed by robot.txt
Hello,
I was trying configure the Twitter Cards in my forum when i notice that the Twitter don't have access to the forumotion foruns.
The Twitter bot, used by Twitter to search the content of the websites (like the Google or Bing bots) is not allowed to crawl the forums... That is a big error that could have a big impact in the SEO of the Forumotion forums, as one of the majors social networks is not able to crawl the websites (Twitter is a major used social network,so this have some impact in the SEO there)
Is possible test that using the Twitter cards validator tools:
https://dev.twitter.com/docs/cards/validation/validator
If you enter a forumotion forum domain there (for instance, https://help.forumotion.com/forum ) it gives the error:
That said, the Twitter bot isn't allowed to crawl the forumotion forums becouse isn't allowed in the robots.txt. This error happend in all the forumotion forums...
Checking the robots.txt of the Forumotion forums, i notice that the Twitterbot isn't specify.
For instance: https://help.forumotion.com/robots.txt
For the twitterbot be allowed to crawl the forums he have to be declared:
https://dev.twitter.com/docs/cards/troubleshooting#twitterbot
As is possible to see in the robots.txt of this website, he isn't there, so its being blocked...
Hope that some admin check this ASAP.
Twitter, as i said before, is a social network used by many users and forum owners (including by Forumotion itself), so if the Twitter bot is not allowed to crawl, will have impact...
Thanks
I was trying configure the Twitter Cards in my forum when i notice that the Twitter don't have access to the forumotion foruns.
The Twitter bot, used by Twitter to search the content of the websites (like the Google or Bing bots) is not allowed to crawl the forums... That is a big error that could have a big impact in the SEO of the Forumotion forums, as one of the majors social networks is not able to crawl the websites (Twitter is a major used social network,so this have some impact in the SEO there)
Is possible test that using the Twitter cards validator tools:
https://dev.twitter.com/docs/cards/validation/validator
If you enter a forumotion forum domain there (for instance, https://help.forumotion.com/forum ) it gives the error:
- Code:
Fetching the page failed because it's denied by robots.txt.
That said, the Twitter bot isn't allowed to crawl the forumotion forums becouse isn't allowed in the robots.txt. This error happend in all the forumotion forums...
Checking the robots.txt of the Forumotion forums, i notice that the Twitterbot isn't specify.
For instance: https://help.forumotion.com/robots.txt
For the twitterbot be allowed to crawl the forums he have to be declared:
https://dev.twitter.com/docs/cards/troubleshooting#twitterbot
- Code:
User-agent: Twitterbot
Disallow:
As is possible to see in the robots.txt of this website, he isn't there, so its being blocked...
Hope that some admin check this ASAP.
Twitter, as i said before, is a social network used by many users and forum owners (including by Forumotion itself), so if the Twitter bot is not allowed to crawl, will have impact...
Thanks
Last edited by tugatech on September 5th 2014, 8:18 pm; edited 1 time in total
tugatech- Forumember
- Posts : 189
Reputation : 8
Language : Portuguese
Location : Portugal
Re: Twitter not allowed by robot.txt
Bump..
Hope someone from the Forumotion team check this.
Hope someone from the Forumotion team check this.
tugatech- Forumember
- Posts : 189
Reputation : 8
Language : Portuguese
Location : Portugal
Re: Twitter not allowed by robot.txt
And... BUMP....
I this this should desearve a little of attention. Is something that impact the SEO of all the forums....
I this this should desearve a little of attention. Is something that impact the SEO of all the forums....
tugatech- Forumember
- Posts : 189
Reputation : 8
Language : Portuguese
Location : Portugal
Re: Twitter not allowed by robot.txt
I get the feeling that this is for sites that have a Twitter page.
Is there a meta tag that you need to add in the administration panel?
Is there a meta tag that you need to add in the administration panel?
Lost Founder's Password |Forum's Utilities |Report a Forum |General Rules |FAQ |Tricks & Tips
You need one post to send a PM.
You need one post to send a PM.
When your topic has been solved, ensure you mark the topic solved.
Never post your email in public.
Re: Twitter not allowed by robot.txt
no...
This is a limitation by the "robots.txt" file applied in ALL the forums... In all the forums we have the same default "robots.txt" file, and its the same in all of them. Isn't related with something in the configuration or in the administration panel....
This file is used by the search engine and social networks bots (like the Google Bot or Bing bot).
For instance: https://help.forumotion.com/robots.txt
This is not related with the "twitter page", meta tags or any function of the forums. Is something related with the forumotion side...
The "robots.txt" from Forumotion is blocking (or better, not allowing) the Twitter bot from access the forums... So, for twitter, is like the bot can access any forumotion website, what is a BIG ERROR! Twitter is a major social network and the bots of them should have access to the forums...
And is the same in ALL the forumotion forums, becouse all of them have the exact same "robots.txt" file.
The first reply in this topic explain all in detail.
This is a limitation by the "robots.txt" file applied in ALL the forums... In all the forums we have the same default "robots.txt" file, and its the same in all of them. Isn't related with something in the configuration or in the administration panel....
This file is used by the search engine and social networks bots (like the Google Bot or Bing bot).
For instance: https://help.forumotion.com/robots.txt
This is not related with the "twitter page", meta tags or any function of the forums. Is something related with the forumotion side...
The "robots.txt" from Forumotion is blocking (or better, not allowing) the Twitter bot from access the forums... So, for twitter, is like the bot can access any forumotion website, what is a BIG ERROR! Twitter is a major social network and the bots of them should have access to the forums...
And is the same in ALL the forumotion forums, becouse all of them have the exact same "robots.txt" file.
The first reply in this topic explain all in detail.
tugatech- Forumember
- Posts : 189
Reputation : 8
Language : Portuguese
Location : Portugal
Re: Twitter not allowed by robot.txt
Bump... (again)
tugatech- Forumember
- Posts : 189
Reputation : 8
Language : Portuguese
Location : Portugal
Re: Twitter not allowed by robot.txt
I will ask our Pro Admin to checkout this topic.
Lost Founder's Password |Forum's Utilities |Report a Forum |General Rules |FAQ |Tricks & Tips
You need one post to send a PM.
You need one post to send a PM.
When your topic has been solved, ensure you mark the topic solved.
Never post your email in public.
Re: Twitter not allowed by robot.txt
Hi,
We did a test on the forum http://web-coffee.forumactif.org/ by inserting Twitter properties Cards
in the template overall_header:
"{PAGE_DESCRIPTION}" /> just in that section you should replace it a personal information
Forumotion example:
Now, Pass you forum to the validator:
https://dev.twitter.com/docs/cards/validation/validator
You will see that the Twitterbot has access . You just made mistake by inserting your information into Twitter Card
Be careful to use web-coffee.forumactif.org and not http://web-coffee.forumactif.org/
Look at the pictures:
Regards
We did a test on the forum http://web-coffee.forumactif.org/ by inserting Twitter properties Cards
in the template overall_header:
- Code:
<!-- BEGIN Twitter Cards -->
<meta property="twitter:card" content="summary" />
<meta property="twitter:site" content="@Ck_eMrkg" />
<meta property="twitter:title" content="{SITENAME_TITLE}{PAGE_TITLE}" />
<meta property="twitter:description" content="{PAGE_DESCRIPTION}" />
<meta property="twitter:image"
content="http://i58.servimg.com/u/f58/18/61/85/79/logo-w10.jpg" />
<meta property="twitter:url" content="{switch_canonical_url.CANONICAL_URL}"
/> <!-- END Twitter Cards --
"{PAGE_DESCRIPTION}" /> just in that section you should replace it a personal information
Forumotion example:
- Code:
<meta property="twitter:description" content="Support Forum of Forumotion
Users : {PAGE_TITLE}." /
Now, Pass you forum to the validator:
https://dev.twitter.com/docs/cards/validation/validator
You will see that the Twitterbot has access . You just made mistake by inserting your information into Twitter Card
Be careful to use web-coffee.forumactif.org and not http://web-coffee.forumactif.org/
Look at the pictures:
Regards
Shadow- Manager
- Posts : 16217
Reputation : 1831
Language : French, English
Re: Twitter not allowed by robot.txt
Hello Buttercup,
Thanks for the info.
Thats right. If we don't input the "http" it works, but only for the main page. The subpages still appear as having being denied by the robots.txt.
For instance:
http://web-coffee.forumactif.org/t27-la-machine-a-remonter-le-temps-du-web
Note: as before, i have tested with and without the "http". Also tested with a slash (/) in the end or not.
Thanks for the info.
Thats right. If we don't input the "http" it works, but only for the main page. The subpages still appear as having being denied by the robots.txt.
For instance:
http://web-coffee.forumactif.org/t27-la-machine-a-remonter-le-temps-du-web
Note: as before, i have tested with and without the "http". Also tested with a slash (/) in the end or not.
tugatech- Forumember
- Posts : 189
Reputation : 8
Language : Portuguese
Location : Portugal
Re: Twitter not allowed by robot.txt
Did you press the request approval button on Twitter?
Lost Founder's Password |Forum's Utilities |Report a Forum |General Rules |FAQ |Tricks & Tips
You need one post to send a PM.
You need one post to send a PM.
When your topic has been solved, ensure you mark the topic solved.
Never post your email in public.
Re: Twitter not allowed by robot.txt
SLGray wrote:Did you press the request approval button on Twitter?
Hello,
This is not related with the aprove...
Check the last message.
Thanks
tugatech- Forumember
- Posts : 189
Reputation : 8
Language : Portuguese
Location : Portugal
Re: Twitter not allowed by robot.txt
You might have to add the code to different templates so it will crawl the whole forum.
Lost Founder's Password |Forum's Utilities |Report a Forum |General Rules |FAQ |Tricks & Tips
You need one post to send a PM.
You need one post to send a PM.
When your topic has been solved, ensure you mark the topic solved.
Never post your email in public.
Similar topics
» Buy Twitter Followers | Buy Bulk Twitter Followers
» twitter
» Twitter
» Twitter Buttons
» Twitter Connect
» Twitter Buttons
» Twitter Connect
Page 1 of 1
Permissions in this forum:
You cannot reply to topics in this forum