The forum of the forums
Would you like to react to this message? Create an account in a few clicks or log in to continue.

Twitter not allowed by robot.txt

3 posters

Go down

In progress Twitter not allowed by robot.txt

Post by tugatech August 29th 2014, 5:33 pm

Hello,

I was trying configure the Twitter Cards in my forum when i notice that the Twitter don't have access to the forumotion foruns.
The Twitter bot, used by Twitter to search the content of the websites (like the Google or Bing bots) is not allowed to crawl the forums... That is a big error that could have a big impact in the SEO of the Forumotion forums, as one of the majors social networks is not able to crawl the websites (Twitter is a major used social network,so this have some impact in the SEO there)

Is possible test that using the Twitter cards validator tools:
https://dev.twitter.com/docs/cards/validation/validator

If you enter a forumotion forum domain there (for instance, https://help.forumotion.com/forum ) it gives the error:
Code:
Fetching the page failed because it's denied by robots.txt.

That said, the Twitter bot isn't allowed to crawl the forumotion forums becouse isn't allowed in the robots.txt. This error happend in all the forumotion forums...

Checking the robots.txt of the Forumotion forums, i notice that the Twitterbot isn't specify. 
For instance: https://help.forumotion.com/robots.txt

For the twitterbot be allowed to crawl the forums he have to be declared:
https://dev.twitter.com/docs/cards/troubleshooting#twitterbot
Code:
User-agent: Twitterbot
Disallow:

As is possible to see in the robots.txt of this website, he isn't there, so its being blocked...

Hope that some admin check this ASAP.
Twitter, as i said before, is a social network used by many users and forum owners (including by Forumotion itself), so if the Twitter bot is not allowed to crawl, will have impact...

Thanks


Last edited by tugatech on September 5th 2014, 8:18 pm; edited 1 time in total
tugatech
tugatech
Forumember

Male Posts : 189
Reputation : 8
Language : Portuguese
Location : Portugal

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by tugatech August 31st 2014, 2:57 am

Bump..

Hope someone from the Forumotion team check this. Wink
tugatech
tugatech
Forumember

Male Posts : 189
Reputation : 8
Language : Portuguese
Location : Portugal

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by tugatech September 2nd 2014, 8:33 pm

And... BUMP....

I this this should desearve a little of attention. Is something that impact the SEO of all the forums....
tugatech
tugatech
Forumember

Male Posts : 189
Reputation : 8
Language : Portuguese
Location : Portugal

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by SLGray September 2nd 2014, 8:37 pm

I get the feeling that this is for sites that have a Twitter page.

Is there a meta tag that you need to add in the administration panel?


Twitter not allowed by robot.txt Slgray10

When your topic has been solved, ensure you mark the topic solved.
Never post your email in public.
SLGray
SLGray
Administrator
Administrator

Male Posts : 51453
Reputation : 3519
Language : English
Location : United States

https://forumsclub.com/gc/128-link-directory/

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by tugatech September 2nd 2014, 10:09 pm

no...
This is a limitation by the "robots.txt" file applied in ALL the forums... In all the forums we have the same default "robots.txt" file, and its the same in all of them. Isn't related with something in the configuration or in the administration panel....

This file is used by the search engine and social networks bots (like the Google Bot or Bing bot).
For instance: https://help.forumotion.com/robots.txt

This is not related with the "twitter page", meta tags or any function of the forums. Is something related with the forumotion side...

The "robots.txt" from Forumotion is blocking (or better, not allowing) the Twitter bot from access the forums... So, for twitter, is like the bot can access any forumotion website, what is a BIG ERROR! Twitter is a major social network and the bots of them should have access to the forums...

And is the same in ALL the forumotion forums, becouse all of them have the exact same "robots.txt" file.
The first reply in this topic explain all in detail. Smile
tugatech
tugatech
Forumember

Male Posts : 189
Reputation : 8
Language : Portuguese
Location : Portugal

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by tugatech September 4th 2014, 1:23 pm

Bump... (again)
tugatech
tugatech
Forumember

Male Posts : 189
Reputation : 8
Language : Portuguese
Location : Portugal

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by SLGray September 4th 2014, 8:08 pm

I will ask our Pro Admin to checkout this topic.


Twitter not allowed by robot.txt Slgray10

When your topic has been solved, ensure you mark the topic solved.
Never post your email in public.
SLGray
SLGray
Administrator
Administrator

Male Posts : 51453
Reputation : 3519
Language : English
Location : United States

https://forumsclub.com/gc/128-link-directory/

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by Shadow September 5th 2014, 10:39 am

Hi,

We did a test on the forum  http://web-coffee.forumactif.org/ by inserting Twitter properties Cards
in the template overall_header:

Code:
<!-- BEGIN Twitter Cards -->
<meta property="twitter:card" content="summary" />
<meta property="twitter:site" content="@Ck_eMrkg" />
<meta property="twitter:title" content="{SITENAME_TITLE}{PAGE_TITLE}" />
<meta property="twitter:description" content="{PAGE_DESCRIPTION}" />
<meta property="twitter:image"
content="http://i58.servimg.com/u/f58/18/61/85/79/logo-w10.jpg" />
<meta property="twitter:url" content="{switch_canonical_url.CANONICAL_URL}"
/>  <!-- END Twitter Cards --

"{PAGE_DESCRIPTION}" /> just in that section you should replace it a personal information
Forumotion example:

Code:
<meta property="twitter:description" content="Support Forum of Forumotion
Users : {PAGE_TITLE}." /

Now, Pass you forum to the validator:
https://dev.twitter.com/docs/cards/validation/validator
You will see that the Twitterbot has access . You just made mistake by inserting your information into Twitter Card


Be careful to use web-coffee.forumactif.org and not http://web-coffee.forumactif.org/

Look at the pictures:

Twitter not allowed by robot.txt 05-09-13



Twitter not allowed by robot.txt 05-09-14


Regards
Shadow
Shadow
Manager
Manager

Male Posts : 16217
Reputation : 1831
Language : French, English

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by tugatech September 5th 2014, 8:17 pm

Hello Buttercup,

Thanks for the info.
Thats right. If we don't input the "http" it works, but only for the main page. The subpages still appear as having being denied by the robots.txt.

For instance:
http://web-coffee.forumactif.org/t27-la-machine-a-remonter-le-temps-du-web

Twitter not allowed by robot.txt Tugatech-2014-09-05_19.16.27

Note: as before, i have tested with and without the "http". Also tested with a slash (/) in the end or not.
tugatech
tugatech
Forumember

Male Posts : 189
Reputation : 8
Language : Portuguese
Location : Portugal

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by SLGray September 6th 2014, 2:39 am

Did you press the request approval button on Twitter?


Twitter not allowed by robot.txt Slgray10

When your topic has been solved, ensure you mark the topic solved.
Never post your email in public.
SLGray
SLGray
Administrator
Administrator

Male Posts : 51453
Reputation : 3519
Language : English
Location : United States

https://forumsclub.com/gc/128-link-directory/

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by tugatech September 6th 2014, 6:13 pm

SLGray wrote:Did you press the request approval button on Twitter?

Hello,

This is not related with the aprove...
Check the last message. Wink

Thanks
tugatech
tugatech
Forumember

Male Posts : 189
Reputation : 8
Language : Portuguese
Location : Portugal

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by SLGray September 6th 2014, 7:39 pm

You might have to add the code to different templates so it will crawl the whole forum.


Twitter not allowed by robot.txt Slgray10

When your topic has been solved, ensure you mark the topic solved.
Never post your email in public.
SLGray
SLGray
Administrator
Administrator

Male Posts : 51453
Reputation : 3519
Language : English
Location : United States

https://forumsclub.com/gc/128-link-directory/

Back to top Go down

Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum