The forum of the forums
Welcome to the Official Support Forum of Forumotion!

To take full advantage of everything offered by our forum, please log in if you are already a member, or join our community if you've not yet.



Create a free forum like this one.

Twitter not allowed by robot.txt

View previous topic View next topic Go down

In progress Twitter not allowed by robot.txt

Post by tugatech on August 29th 2014, 5:33 pm

Hello,

I was trying configure the Twitter Cards in my forum when i notice that the Twitter don't have access to the forumotion foruns.
The Twitter bot, used by Twitter to search the content of the websites (like the Google or Bing bots) is not allowed to crawl the forums... That is a big error that could have a big impact in the SEO of the Forumotion forums, as one of the majors social networks is not able to crawl the websites (Twitter is a major used social network,so this have some impact in the SEO there)

Is possible test that using the Twitter cards validator tools:
https://dev.twitter.com/docs/cards/validation/validator

If you enter a forumotion forum domain there (for instance,http://help.forumotion.com/forum ) it gives the error:
Code:
Fetching the page failed because it's denied by robots.txt.

That said, the Twitter bot isn't allowed to crawl the forumotion forums becouse isn't allowed in the robots.txt. This error happend in all the forumotion forums...

Checking the robots.txt of the Forumotion forums, i notice that the Twitterbot isn't specify.
For instance:http://help.forumotion.com/robots.txt

For the twitterbot be allowed to crawl the forums he have to be declared:
https://dev.twitter.com/docs/cards/troubleshooting#twitterbot
Code:
User-agent: Twitterbot
Disallow:

As is possible to see in the robots.txt of this website, he isn't there, so its being blocked...

Hope that some admin check this ASAP.
Twitter, as i said before, is a social network used by many users and forum owners (including by Forumotion itself), so if the Twitter bot is not allowed to crawl, will have impact...

Thanks


Last edited by tugatech on September 5th 2014, 8:18 pm; edited 1 time in total

tugatech
Forumember

Male Posts : 175
Reputation : 8
Language : Portuguese
Location : Portugal

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by tugatech on August 31st 2014, 2:57 am

Bump..

Hope someone from the Forumotion team check this. Wink

tugatech
Forumember

Male Posts : 175
Reputation : 8
Language : Portuguese
Location : Portugal

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by tugatech on September 2nd 2014, 8:33 pm

And... BUMP....

I this this should desearve a little of attention. Is something that impact the SEO of all the forums....

tugatech
Forumember

Male Posts : 175
Reputation : 8
Language : Portuguese
Location : Portugal

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by SLGray on September 2nd 2014, 8:37 pm

I get the feeling that this is for sites that have a Twitter page.

Is there a meta tag that you need to add in the administration panel?


When your topic has been solved, ensure you mark the topic solved.
Never post your email in public.


SLGray
Administrator
Administrator

Male Posts : 36659
Reputation : 2443
Language : English
Location : United States

http://ztwds.forumotion.com/

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by tugatech on September 2nd 2014, 10:09 pm

no...
This is a limitation by the "robots.txt" file applied in ALL the forums... In all the forums we have the same default "robots.txt" file, and its the same in all of them. Isn't related with something in the configuration or in theadministration panel....

This file is used by the search engine and social networks bots (like the Google Bot or Bing bot).
For instance: http://help.forumotion.com/robots.txt

This is not related with the "twitter page", meta tags or any function of the forums. Is something related with the forumotion side...

The "robots.txt" from Forumotion is blocking (or better, not allowing) the Twitter bot from access the forums... So, for twitter, is like the bot can access any forumotion website, what is a BIG ERROR! Twitter is a major social network and the bots of them should have access to the forums...

And is the same in ALL the forumotion forums, becouse all of them have the exact same "robots.txt" file.
The first reply in this topic explain all in detail. Smile

tugatech
Forumember

Male Posts : 175
Reputation : 8
Language : Portuguese
Location : Portugal

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by tugatech on September 4th 2014, 1:23 pm

Bump... (again)

tugatech
Forumember

Male Posts : 175
Reputation : 8
Language : Portuguese
Location : Portugal

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by SLGray on September 4th 2014, 8:08 pm

I will ask our Pro Admin to checkout this topic.


When your topic has been solved, ensure you mark the topic solved.
Never post your email in public.


SLGray
Administrator
Administrator

Male Posts : 36659
Reputation : 2443
Language : English
Location : United States

http://ztwds.forumotion.com/

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by Leviosa on September 5th 2014, 10:39 am

Hi,

We did a test on the forum http://web-coffee.forumactif.org/ by inserting Twitter properties Cards
in the template overall_header:

Code:
<!-- BEGIN Twitter Cards -->
<meta property="twitter:card" content="summary" />
<meta property="twitter:site" content="@Ck_eMrkg" />
<meta property="twitter:title" content="{SITENAME_TITLE}{PAGE_TITLE}" />
<meta property="twitter:description" content="{PAGE_DESCRIPTION}" />
<meta property="twitter:image"
content="http://i58.servimg.com/u/f58/18/61/85/79/logo-w10.jpg" />
<meta property="twitter:url" content="{switch_canonical_url.CANONICAL_URL}"
/> <!-- END Twitter Cards --

"{PAGE_DESCRIPTION}" /> just in that section you should replace it a personal information
Forumotion example:

Code:
<meta property="twitter:description" content="Support Forum of Forumotion
Users : {PAGE_TITLE}." /

Now, Pass you forum to the validator:
https://dev.twitter.com/docs/cards/validation/validator
You will see that the Twitterbot has access . You just made mistake by inserting your information into Twitter Card


Be careful to use web-coffee.forumactif.org and not http://web-coffee.forumactif.org/

Look at the pictures:








Regards


No help without your forum url
No support via private message



Leviosa
Administrator
Administrator

Female Posts : 15587
Reputation : 1589
Language : French, English

http://help.forumotion.com

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by tugatech on September 5th 2014, 8:17 pm

Hello Buttercup,

Thanks for the info.
Thats right. If we don't input the "http" it works, but only for the main page. The subpages still appear as having being denied by the robots.txt.

For instance:
http://web-coffee.forumactif.org/t27-la-machine-a-remonter-le-temps-du-web



Note: as before, i have tested with and without the "http". Also tested with a slash (/) in the end or not.

tugatech
Forumember

Male Posts : 175
Reputation : 8
Language : Portuguese
Location : Portugal

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by SLGray on September 6th 2014, 2:39 am

Did you press the request approval button on Twitter?


When your topic has been solved, ensure you mark the topic solved.
Never post your email in public.


SLGray
Administrator
Administrator

Male Posts : 36659
Reputation : 2443
Language : English
Location : United States

http://ztwds.forumotion.com/

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by tugatech on September 6th 2014, 6:13 pm

@SLGray wrote:Did you press the request approval button on Twitter?

Hello,

This is not related with the aprove...
Check the last message. Wink

Thanks

tugatech
Forumember

Male Posts : 175
Reputation : 8
Language : Portuguese
Location : Portugal

Back to top Go down

In progress Re: Twitter not allowed by robot.txt

Post by SLGray on September 6th 2014, 7:39 pm

You might have to add the code to different templates so it will crawl the whole forum.


When your topic has been solved, ensure you mark the topic solved.
Never post your email in public.


SLGray
Administrator
Administrator

Male Posts : 36659
Reputation : 2443
Language : English
Location : United States

http://ztwds.forumotion.com/

Back to top Go down

View previous topic View next topic Back to top


 
Permissions in this forum:
You cannot reply to topics in this forum