The forum of the forums
Would you like to react to this message? Create an account in a few clicks or log in to continue.

Robots on my forum

4 posters

Go down

Solved Robots on my forum

Post by hiwwe September 20th 2014, 10:49 am

今天收到百度的信如下:

http://www.wwe911.com/  :网站对百度进行了全站封禁2014年09月20日

亲爱的网站管理员:

经检测,网站 http://www.wwe911.com/ 对百度进行了robots全站封禁,Baiduspider无法抓取网站的网页。请您在Robots工具中查看详情。

检测时间:2014-09-20 03:00:00

結果我一看 http://www.wwe911.com/robots.txt 文件里竟然有“User-agent: Baiduspider”不讓百度抓取全站內容!

你們應該趕緊的去掉這個啊,讓百度能正常收錄我的論壇內容啊!
avatar
hiwwe
Forumember

Posts : 31
Reputation : 1
Language : cn

Back to top Go down

Solved Re: Robots on my forum

Post by hiwwe September 20th 2014, 10:51 am


网站robots.txt分析结果
位置 路径 访问者 说明
第8行 /abuse Mediapartners-Google Mediapartners-Google* * 不允许抓取
第9行 /admgt/ Mediapartners-Google Mediapartners-Google* * 不允许抓取
第10行 /contact Mediapartners-Google Mediapartners-Google* * 不允许抓取
第11行 /donate Mediapartners-Google Mediapartners-Google* * 不允许抓取
第12行 /go/ Mediapartners-Google Mediapartners-Google* * 不允许抓取
第13行 /login Mediapartners-Google Mediapartners-Google* * 不允许抓取
第14行 /modcp Mediapartners-Google Mediapartners-Google* * 不允许抓取
第15行 /post Mediapartners-Google Mediapartners-Google* * 不允许抓取
第16行 /privmsg Mediapartners-Google Mediapartners-Google* * 不允许抓取
第17行 /spa/ Mediapartners-Google Mediapartners-Google* * 不允许抓取
第18行 /sta/ Mediapartners-Google Mediapartners-Google* * 不允许抓取
第721行 / (所有目录和文件) 008 Accoona aipbot aipbot* aipbot/1.0 Alexa Alexa Bitlybot Alexibot AltaVista Intranet V2.0 AVS EVAL search@freeit.com AltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.net AltaVista Intranet V2.0 evreka.com crawler@evreka.com AltaVista V2.0B crawler@evreka.com Anonymous ApocalXExplorerBot appie Aqua_Products Argus/1.1 Artabus Ask Jeeves asterias atSpider attentio AV Fetch 1.0 AVSearch-3.0(AltaVista/AVC) AWS Cloud Based b2w b2w/0.1 BackDoorBot BackDoorBot/1.0 Baiduspider becomebot BecomeBot BigBrother BIGLOTRON (BETA 2;GNU/Linux) BizInformation Black Hole Black.Hole BlackWidow BlowFish BlowFish/1.0 BoardPulse boitho.com-dc Bookmark search tool bot/1.0 BotALot Bot mailto:craftbot@yahoo.com BotRightHere BrandProtect BuiltBotTough Bullseye Bullseye/1.0 BunnySlippers CazoodleBot Cegbfeieh cfetch cfetch/1.0 CheeseBot CherryPicker CherryPicker /1.0 CherryPickerElite/1.0 CherryPickerSE/1.0 ChinaClaw Collage cometrics-bot complex_network_group convera ConveraCrawler ConveraCrawler/0.2 ConveraCrawler/0.9d Convera Internet Spider V6.x ConveraMultiMediaCrawler/0.1 Copernic CopyRightCheck cosmos Crescent Crescent Internet ToolPak HTTP OLE Control v.1.0 Crescent Internet ToolPak HTTPOLE Control v.1.0 Curl Custo CydralSpider Deepnet Explorer default.ida DigExt DISCo discobot DISCoFinder DISCo Pump DISCo Pump 3.0 DISCo Pump 3.1 DISCo Pump 3.2 DittoSpyder DOC dotbot DotBot DotBot/1.1 Download Demon Download Demon/3.2.0.8 Download Demon/3.5.0.11 Download Ninja Download Wonder DSurf Dulance bot dumbot eCatch eCatch/3.0 echo! EchO!/2.0 EirGrabber EliteSys Entry EmailCollector Email Extractor EmailSiphon EmailSmartz EmailWolf Enterprise_Search Enterprise_Search/1.0 EroCrawler es ESIRover e-SocietyRobot Exabot Exabot/2.0 Exabot-Images Express WebPictures Express WebPictures (www.express-soft.com) ExtractorPro EyeNetIE FairAd Client Fairshare Fasterfox Fetch Flaming AttackBot Flamingo_SearchEngine FlashGet FlashGet WebWasher 3.2 Foobot FreeFind FreeWebMonitoring SiteChecker/0.1 FrontPage FrontPage [NC,OR] FurlBot Gaisbot Gaisbot/3.0 GetBot GetRight GetRight/2.11 GetRight/3.1 GetRight/3.2 GetRight/3.3 GetRight/3.3.3 GetRight/3.3.4 GetRight/4.0.0 GetRight/4.1.0 GetRight/4.1.1 GetRight/4.1.2 GetRight/4.2 GetRight/4.2b (Portuguxeas) GetRight/4.2c GetRight/4.3 GetRight/4.5 GetRight/4.5a GetRight/4.5b GetRight/4.5b1 GetRight/4.5b2 GetRight/4.5b3 GetRight/4.5b6 GetRight/4.5b7 GetRight/4.5c GetRight/4.5d GetRight/4.5e GetRight/5.0beta1 GetRight/5.0beta2 GetUrl GetWeb! Gigabot Go-Ahead-Got-It Go!Zilla Go!Zilla 3.3 (www.gozilla.com) Go!Zilla 3.5 (www.gozilla.com) Go!Zilla (www.gozilla.com) GrabNet Grafula grub grub-client Hackertarget.com Harvest Harvest/1.5 Hatena Antenna HavIndex heritrix hloader HMView httplib httrack HTTrack HTTrack 3.0 HTTrack 3.0x HTTrack [NC,OR] humanlinks ichiro IconSurf Igentia Image Collector Image Stripper Image Sucker Indy Library Indy Library [NC,OR] InfoNaviRobot InfoSpiders InterGET Internet Explore Internet Ninja Internet Ninja 4.0 Internet Ninja 5.0 Internet Ninja 6.0 InternetSupervision IRLbot Iron Iron33/1.0.2 Jeeves JennyBot Jetbot Jetbot/1.0 JetCar Jobo JOC Web Spider kalooga KDD Exploror Kenjin Spider Kenjin.Spider Keyword Density Keyword.Density Keyword Density/0.9 larbin Larbin larbin_2.6.2 (kabura@sushi.com) larbin_2.6.2 kabura@sushi.com larbin_2.6.2 (larbin2.6.2@unspecified.mail) larbin_2.6.2 larbin2.6.2@unspecified.mail larbin_2.6.2 larbin@correa.org larbin_2.6.2 listonATccDOTgatechDOTedu larbin_2.6.2 (listonATccDOTgatechDOTedu) larbin_2.6.2 (vitalbox1@hotmail.com) larbin_2.6.2 vitalbox1@hotmail.com larbin (samualt9@bigfoot.com) larbin samualt9@bigfoot.com LBot LeechFTP LexiBot libWeb/clsHTTP libWeb/clsHTTPDisallow: / libwww LightningDownload Linguee LinkedIn LinkextractorPro Linknzbot Linknzbot* Linknzbot 2004 LinkScan LinkScan/8.1a Unix LinkScan/8.1a.Unix LinkScan/8.1a Unix Disallow: / linksmanager LinksManager LinksManager.com_bot LinkWalker LjSEEK LNSpiderguy looksmart LWP LWP* lwp-trivial lwp-trivial/1.34 magpie-crawler Mail Sweeper Marketwirebot Mass Downloader Mass Downloader/2.2 Mata Hari Mata.Hari MetagerBot MetaURI Microsoft.URL Microsoft URL Control Microsoft URL Control* Microsoft.URL.Control Microsoft URL Control - 5.01.4511 Microsoft URL Control - 6.00.8169 Microsoft URL Control - 6.01.9782 MIDown tool MIIxpc MIIxpc/4.2 Missigua Locator Mister PiX Mister.PiX Mister Pix II 2.01 Mister Pix II 2.02a Mister PiX version.dll MJ12bot MLBot moget moget/2.1 mozilla Mozilla Mozilla/2.0 (compatible; Ask Jeeves) mozilla/3 mozilla/4 Mozilla/4.0 (compatible; BullsEye; Windows 95) Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000) Mozilla/4.0 (compatible; MSIE 4.0; Windows 95) Mozilla/4.0 (compatible; MSIE 4.0; Windows 98) Mozilla/4.0 (compatible; MSIE 4.0; Windows ME) Mozilla/4.0 (compatible; MSIE 4.0; Windows NT) Mozilla/4.0 (compatible; MSIE 4.0; Windows XP) mozilla/5 MRSPUTNIK MSIECrawler MSRBOT MS Search 4.0 Robot MS Search 5.0 Robot munky naver Naverbot NaverBot NaverBot-1.0 Navroad NearSite NetAnts NetAnts/1.10 NetAnts/1.23 NetAnts/1.24 NetAnts/1.25 NetAttache NetAttache Light 1.1 Netcraft Web Server Survey NetMechanic NetSpider Net Vampire Net Vampire/3.0 NetZIP NetZip-Downloader NetZip-Downloader/1.0.62 (Win32; Dec 7 1998) NetZip Downloader 1.0 Win32(Nov 12 1998) NetZippy+(http://www.innerprise.net/usp-spider.asp) NetZippy+(http:/www.innerprise.net/usp-spider.asp) NICErsPRO NimbleCrawler NPbot NPBot NPBot/3 Nutch Nutch* NutchCVS/0.06-dev NutchCVS/0.7.1 NutchOrg oBot Ocelli Octopus Offline Explorer Offline.Explorer Offline Explorer/1.2 Offline Explorer/1.4 Offline Explorer/1.6 Offline Explorer/1.7 Offline Explorer/1.9 Offline Explorer/2.0 Offline Explorer/2.1 Offline Explorer/2.3 Offline Explorer/2.4 Offline Explorer/2.5 Offline Navigator OmniExplorer_Bot oneriot Openbot Openfind Openfind data gathere Openfind data gatherer Oracle Ultra Search OutfoxBot/0.5 PageGrabber Papa Foto pavuk PBWF pcBrowser penthesilea PerMan PGBot PhpDig Pingdom GIGRIB (http://www.pingdom.com) postrank ProPowerBot ProPowerBot/2.14 ProWebWalker psbot psycheclone Psycheclone Python-urllib QuepasaCreep QueryN Metasearch QueryN.Metasearch radian6 comment reader radian6 Feedfetcher Radiation Retriever Radiation Retriever 1.1 RB2B-bot RealDownload RealDownload/4.0.0.40 RealDownload/4.0.0.41 RealDownload/4.0.0.42 ReGet RepoMonkey RepoMonkey Bait & Tackle/v1.01 RepoMonkey Bait & Tackle RepoMonkey Bait & Tackle/v1.01 research-spider RMA Robozilla Roverbot RufusBot sbider Scooter/1.0 Scooter/1.0 scooter@pa.dec.com Scooter/1.1 (custom) Scooter/2.0 G.R.A.B. V1.1.0 Scooter/2.0 G.R.A.B. X2.0 Scooter2_Mercator_x-x.0 Scooter-3.0.EU Scooter-3.0.FS Scooter-3.0.HD Scooter-3.0QI Scooter-3.0.VNS Scooter-3.2 Scooter-3.2.BT Scooter-3.2.DIL Scooter-3.2.EX Scooter-3.2.JT Scooter-3.2.NIV Scooter-3.2.SF0 Scooter-3.2.snippet Scooter/3.3 Scooter-3.3dev Scooter/3.3.QA.pczukor Scooter/3.3_SF Scooter/3.3.vscooter Scooter-ARS-1.1 Scooter-ARS-1.1-ih Scooter_bh0-3.0.3 Scooter_trk3-3.0.3 scooter-venus-3.0.vns Scooter-W3-1.0 Scooter-W3.1.2 Scrubby SearchDaimon.com-dc searchpreview seekbot Seekbot Seekbot/1.0 SEOprofiler Shai'Hulud Shim-Crawler ShopWiki ShopWiki/1.0 SightupBot SiteBot SiteSnagger Slurp China SlySearch SmartDownload SmartDownload/1.2.76 (Win32; Apr 1 1999) SmartDownload/1.2.77 (Win32; Aug 17 1999) SmartDownload/1.2.77 (Win32; Feb 1 2000) SmartDownload/1.2.77 (Win32; Jun 19 2001) Snapbot Snappy Softlayer Server Sogou web spider sootle sosospider SpankBot spanner Speedy SpiderBot Sqworm Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux ssearcher100 Stanford Stanford Comp Sci suggybot SuperBot SuperBot/2.6 SuperBot/3.0 (Win32) SuperBot/3.1 (Win32) SuperHTTP SuperHTTP/1.0 Surfbot SurveyBot suzuran Szukacz Szukacz/1.4 tAkeOut Teleport TeleportPro Teleport Pro Teleport Pro/1.29 Teleport Pro/1.29.1590 Teleport Pro/1.29.1634 Teleport Pro/1.29.1718 Teleport Pro/1.29.1820 Teleport Pro/1.29.1847 Telesoft Templeton Teoma The Intraformant The.Intraformant TheNomad TightTwatBot Titan toCrawl toCrawl/UrlDispatcher True_Robot True_Robot/1.0 turingos TurnitinBot TurnitinBot/1.5 Tweetmeme TwengaBot Twiceler URL Control UrlDispatcher ://URLFAN URL_Spider_Pro URLy Warning URLy.Warning VCI VCI WebViewer VCI WebViewer Win32 vobsub VoidEYE vscooter w3mir WatchDog/3.0 WebAuto WebAuto/3.40 (Win98; I) WebBandit WebBandit/3.50 WebCapture WebCapture 2.0 WebCatcher webcopier WebCopier WebCopier v.2.2 WebCopier v2.5 WebCopier v2.6 WebCopier v2.7a WebCopier v2.8 WebCopier v3.0 WebCopier v3.0.1 WebCopier v3.2 WebCopier v3.2a webcopy WebCopy webcrawl.net WebEmailExtrac WebEMailExtrac.* WebEnhancer WebFetch webfetch/2.1.0 WebFetcher WebGo IS Web Image Collector Web.Image.Collector WebLeacher WebmasterWorld Extractor WebmasterWorldForumBot webmirror WebMirror WebReaper Web Reaper WebReaper [info@webreaper.net] WebReaper v9.1 - www.otway.com/webreaper WebReaper v9.7 - www.webreaper.net WebReaper v9.8 - www.webreaper.net WebReaper vWebReaper v7.3 - www,otway.com/webreaper WebReaper [webreaper@otway.com] WebSauger WebSauger 1.20b WebSauger 1.20j WebSauger 1.20k website extractor Website eXtractor Website eXtractor (http:/www.asona.org) Website Quester Website.Quester Website Quester - www.asona.org Website Quester - www.esalesbiz.com/extra/ Webster Pro Webster.Pro WebStripper WebStripper/2.02 WebStripper/2.03 WebStripper/2.10 WebStripper/2.12 WebStripper/2.13 WebStripper/2.15 WebStripper/2.16 WebStripper/2.19 Web Sucker webvac WebVac WebVulnCrawl WebVulnScan WebWalk WebWasher WebWhacker WebZip WebZIP WebZIP/2.75 (http://www.spidersoft.com) WebZIP/2.75 (http:/www.spidersoft.com) WebZIP/3.65 (http://www.spidersoft.com) WebZIP/3.65 (http:/www.spidersoft.com) WebZIP/3.80 (http://www.spidersoft.com) WebZIP/3.80 (http:/www.spidersoft.com) WebZip/4.0 WebZIP/4.0 (http://www.spidersoft.com) WebZIP/4.0 (http:/www.spidersoft.com) WebZIP/4.1 (http://www.spidersoft.com) WebZIP/4.1 (http:/www.spidersoft.com) WebZIP/4.21 WebZIP/4.21 (http://www.spidersoft.com) WebZIP/4.21 (http:/www.spidersoft.com) WebZIP/5.0 WebZIP/5.0 (http://www.spidersoft.com) WebZIP/5.0 (http:/www.spidersoft.com) WebZIP/5.0 PR1 (http://www.spidersoft.com) WebZIP/5.0 PR1 (http:/www.spidersoft.com) wget wGet Wget Wget/1.10.2 Wget/1.5.2 Wget/1.5.3 Wget/1.6 Wget/1.7 Wget/1.8 Wget/1.8.1 Wget/1.8.1+cvs Wget/1.8.2 Wget/1.9-beta whitevector crawler Whitevector+Crawler Widow WikioFeedBot wikiwix-bot-3.0 Willow WinHTTrack Wise-Guys woozweb-monitoring woriobot WWW-Collector WWW-Collector-E WWWOFFLE Xaldon WebSpider Xaldon WebSpider 2.5.b3 Xenu Xenu Link Sleuth Xenu's Xenu's Link Sleuth 1.1c xGet Yahoo-MMCrawler YahooSeeker/CafeKelsa Yeti YodaoBot YRSPider Zao Zealbot Zeus Zeus 11389 Webster Pro V2.9 Win32 Zeus 11652 Webster Pro V2.9 Win32 Zeus 18018 Webster Pro V2.9 Win32 Zeus 26378 Webster Pro V2.9 Win32 Zeus 30747 Webster Pro V2.9 Win32 Zeus 32297 Webster Pro V2.9 Win32 Zeus 39206 Webster Pro V2.9 Win32 Zeus 41641 Webster Pro V2.9 Win32 Zeus 44238 Webster Pro V2.9 Win32 Zeus 51070 Webster Pro V2.9 Win32 Zeus 51674 Webster Pro V2.9 Win32 Zeus 51837 Webster Pro V2.9 Win32 Zeus 63567 Webster Pro V2.9 Win32 Zeus 6694 Webster Pro V2.9 Win32 Zeus 71129 Webster Pro V2.9 Win32 Zeus 82016 Webster Pro V2.9 Win32 Zeus 82900 Webster Pro V2.9 Win32 Zeus 84842 Webster Pro V2.9 Win32 Zeus 90872 Webster Pro V2.9 Win32 Zeus 94934 Webster Pro V2.9 Win32 Zeus 95245 Webster Pro V2.9 Win32 Zeus 95351 Webster Pro V2.9 Win32 Zeus 97371 Webster Pro V2.9 Win32 Zeus Link Scout ZyBorg 不允许抓取
robots.txt错误提示
位置 文件内容 错误原因
第2行 Disallow: 路径必须以/开始
第5行 Disallow: 路径必须以/开始
avatar
hiwwe
Forumember

Posts : 31
Reputation : 1
Language : cn

Back to top Go down

Solved Re: Robots on my forum

Post by Base September 20th 2014, 11:52 am

Hello,

This is an English support forum. Please translate your message into English so that we can help you. Thanks.
Base
Base
Forumaster

Male Posts : 10386
Reputation : 1695
Language : English and French
Location : United Kingdom, England

http://forumotionhub.net

Back to top Go down

Solved Re: Robots on my forum

Post by hiwwe September 20th 2014, 12:14 pm

Base wrote:Hello,

This is an English support forum. Please translate your message into English so that we can help you. Thanks.

The robots.txt file should have "User-agent: Baiduspider" do not let Baidu grasp the contents of the entire station!

Today I received Baidu letter as follows:
Http://www.wwe911.com/ : Web site the station the ban in 2014 09 months 20 days of Baidu
Dear webmaster:
After testing, site http://www.wwe911.com/ of robots total station Baidu were banned, Baiduspider cannot capture website Webpage. Please see the details in the Robots tool.
The detection time: 2014-09-20 03:00:00
The results I see http://www.wwe911.com/robots.txt file should have "User-agent: Baiduspider" do not let Baidu grasp the contents of the entire station!
You should hurry to get rid of this ah, let Baidu can be properly included the contents of the forum I ah!
avatar
hiwwe
Forumember

Posts : 31
Reputation : 1
Language : cn

Back to top Go down

Solved Re: Robots on my forum

Post by hiwwe September 20th 2014, 12:17 pm

Base wrote:Hello,

This is an English support forum. Please translate your message into English so that we can help you. Thanks.

Site robots.txt analysis results
A location path visitors that
Eighth lines of /abuse Mediapartners-Google Mediapartners-Google* * is not allowed to crawl
Ninth lines of /admgt/ Mediapartners-Google Mediapartners-Google* * is not allowed to crawl
Tenth lines of /contact Mediapartners-Google Mediapartners-Google* * is not allowed to crawl
Eleventh lines of /donate Mediapartners-Google Mediapartners-Google* * is not allowed to crawl
Twelfth lines of /go/ Mediapartners-Google Mediapartners-Google* * is not allowed to crawl
Thirteenth lines of /login Mediapartners-Google Mediapartners-Google* * is not allowed to crawl
Fourteenth lines of /modcp Mediapartners-Google Mediapartners-Google* * is not allowed to crawl
Fifteenth lines of /post Mediapartners-Google Mediapartners-Google* * is not allowed to crawl
Sixteenth lines of /privmsg Mediapartners-Google Mediapartners-Google* * is not allowed to crawl
Seventeenth lines of /spa/ Mediapartners-Google Mediapartners-Google* * is not allowed to crawl
Eighteenth lines of /sta/ Mediapartners-Google Mediapartners-Google* * is not allowed to crawl
The 721st line / (all the directories and files) 008 Accoona aipbot aipbot* aipbot/1.0 Alexa Alexa Bitlybot Alexibot AltaVista Intranet V2.0 AVS EVAL search@freeit.com AltaVista Intranet V2.0 Compaq Altavista Eval sveand@altavista.net AltaVista Intranet V2.0 evreka.com crawler@evreka.com AltaVista V2.0B crawler@evreka.com Anonymous ApocalXExplorerBot appie Aqua_Products Argus/1.1 Artabus Ask Jeeves asterias atSpider attentio AV Fetch 1.0 AVSearch-3.0(AltaVista/AVC) AWS Cloud Based b2w b2w/0.1 BackDoorBot BackDoorBot/1.0 Baiduspider becomebot BecomeBot BigBrother BIGLOTRON (BETA 2;GNU/Linux) BizInformation Black Hole Black.Hole BlackWidow BlowFish BlowFish/1.0 BoardPulse boitho.com-dc Bookmark search tool bot/1.0 BotALot Bot mailto:craftbot@yahoo.com BotRightHere BrandProtect BuiltBotTough Bullseye Bullseye/1.0 BunnySlippers CazoodleBot Cegbfeieh cfetch cfetch/1.0 CheeseBot CherryPicker CherryPicker /1.0 CherryPickerElite/1.0 CherryPickerSE/1.0 ChinaClaw Collage cometrics-bot complex_network_group convera ConveraCrawler ConveraCrawler/0.2 ConveraCrawler/0.9d Convera Internet Spider V6.x ConveraMultiMediaCrawler/0.1 Copernic CopyRightCheck cosmos Crescent Crescent Internet ToolPak HTTP OLE Control v.1.0 Crescent Internet ToolPak HTTPOLE Control v.1.0 Curl Custo CydralSpider Deepnet Explorer default.ida DigExt DISCo discobot DISCoFinder DISCo Pump DISCo Pump 3.0 DISCo Pump 3.1 DISCo Pump 3.2 DittoSpyder DOC dotbot DotBot DotBot/1.1 Download Demon Download Demon/3.2.0.8 Download Demon/3.5.0.11 Download Ninja Download Wonder DSurf Dulance bot dumbot eCatch eCatch/3.0 echo! EchO!/2.0 EirGrabber EliteSys Entry EmailCollector Email Extractor EmailSiphon EmailSmartz EmailWolf Enterprise_Search Enterprise_Search/1.0 EroCrawler es ESIRover e-SocietyRobot Exabot Exabot/2.0 Exabot-Images Express WebPictures Express WebPictures (www.express-soft.com) ExtractorPro EyeNetIE FairAd Client Fairshare Fasterfox Fetch Flaming AttackBot Flamingo_SearchEngine FlashGet FlashGet WebWasher 3.2 Foobot FreeFind FreeWebMonitoring SiteChecker/0.1 FrontPage FrontPage [NC,OR] FurlBot Gaisbot Gaisbot/3.0 GetBot GetRight GetRight/2.11 GetRight/3.1 GetRight/3.2 GetRight/3.3 GetRight/3.3.3 GetRight/3.3.4 GetRight/4.0.0 GetRight/4.1.0 GetRight/4.1.1 GetRight/4.1.2 GetRight/4.2 GetRight/4.2b (Portuguxeas) GetRight/4.2c GetRight/4.3 GetRight/4.5 GetRight/4.5a GetRight/4.5b GetRight/4.5b1 GetRight/4.5b2 GetRight/4.5b3 GetRight/4.5b6 GetRight/4.5b7 GetRight/4.5c GetRight/4.5d GetRight/4.5e GetRight/5.0beta1 GetRight/5.0beta2 GetUrl GetWeb! Gigabot Go-Ahead-Got-It Go!Zilla Go!Zilla 3.3 (www.gozilla.com) Go!Zilla 3.5 (www.gozilla.com) Go!Zilla (www.gozilla.com) GrabNet Grafula grub grub-client Hackertarget.com Harvest Harvest/1.5 Hatena Antenna HavIndex heritrix hloader HMView httplib httrack HTTrack HTTrack 3.0 HTTrack 3.0x HTTrack [NC,OR] humanlinks ichiro IconSurf Igentia Image Collector Image Stripper Image Sucker Indy Library Indy Library [NC,OR] InfoNaviRobot InfoSpiders InterGET Internet Explore Internet Ninja Internet Ninja 4.0 Internet Ninja 5.0 Internet Ninja 6.0 InternetSupervision IRLbot Iron Iron33/1.0.2 Jeeves JennyBot Jetbot Jetbot/1.0 JetCar Jobo JOC Web Spider kalooga KDD Exploror Kenjin Spider Kenjin.Spider Keyword Density Keyword.Density Keyword Density/0.9 larbin Larbin larbin_2.6.2 (kabura@sushi.com) larbin_2.6.2 kabura@sushi.com larbin_2.6.2 (larbin2.6.2@unspecified.mail) larbin_2.6.2 larbin2.6.2@unspecified.mail larbin_2.6.2 larbin@correa.org larbin_2.6.2 listonATccDOTgatechDOTedu larbin_2.6.2 (listonATccDOTgatechDOTedu) larbin_2.6.2 (vitalbox1@hotmail.com) larbin_2.6.2 vitalbox1@hotmail.com larbin (samualt9@bigfoot.com) larbin samualt9@bigfoot.com LBot LeechFTP LexiBot libWeb/clsHTTP libWeb/clsHTTPDisallow: / libwww LightningDownload Linguee LinkedIn LinkextractorPro Linknzbot Linknzbot* Linknzbot 2004 LinkScan LinkScan/8.1a Unix LinkScan/8.1a.Unix LinkScan/8.1a Unix Disallow: / linksmanager LinksManager LinksManager.com_bot LinkWalker LjSEEK LNSpiderguy looksmart LWP LWP* lwp-trivial lwp-trivial/1.34 magpie-crawler Mail Sweeper Marketwirebot Mass Downloader Mass Downloader/2.2 Mata Hari Mata.Hari MetagerBot MetaURI Microsoft.URL Microsoft URL Control Microsoft URL Control* Microsoft.URL.Control Microsoft URL Control - 5.01.4511 Microsoft URL Control - 6.00.8169 Microsoft URL Control - 6.01.9782 MIDown tool MIIxpc MIIxpc/4.2 Missigua Locator Mister PiX Mister.PiX Mister Pix II 2.01 Mister Pix II 2.02a Mister PiX version.dll MJ12bot MLBot moget moget/2.1 mozilla Mozilla Mozilla/2.0 (compatible; Ask Jeeves) mozilla/3 mozilla/4 Mozilla/4.0 (compatible; BullsEye; Windows 95) Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000) Mozilla/4.0 (compatible; MSIE 4.0; Windows 95) Mozilla/4.0 (compatible; MSIE 4.0; Windows 98) Mozilla/4.0 (compatible; MSIE 4.0; Windows ME) Mozilla/4.0 (compatible; MSIE 4.0; Windows NT) Mozilla/4.0 (compatible; MSIE 4.0; Windows XP) mozilla/5 MRSPUTNIK MSIECrawler MSRBOT MS Search 4.0 Robot MS Search 5.0 Robot munky naver Naverbot NaverBot NaverBot-1.0 Navroad NearSite NetAnts NetAnts/1.10 NetAnts/1.23 NetAnts/1.24 NetAnts/1.25 NetAttache NetAttache Light 1.1 Netcraft Web Server Survey NetMechanic NetSpider Net Vampire Net Vampire/3.0 NetZIP NetZip-Downloader NetZip-Downloader/1.0.62 (Win32; Dec 7 1998) NetZip Downloader 1.0 Win32(Nov 12 1998) NetZippy+(http://www.innerprise.net/usp-spider.asp) NetZippy+(http:/www.innerprise.net/usp-spider.asp) NICErsPRO NimbleCrawler NPbot NPBot NPBot/3 Nutch Nutch* NutchCVS/0.06-dev NutchCVS/0.7.1 NutchOrg oBot Ocelli Octopus Offline Explorer Offline.Explorer Offline Explorer/1.2 Offline Explorer/1.4 Offline Explorer/1.6 Offline Explorer/1.7 Offline Explorer/1.9 Offline Explorer/2.0 Offline Explorer/2.1 Offline Explorer/2.3 Offline Explorer/2.4 Offline Explorer/2.5 Offline Navigator OmniExplorer_Bot oneriot Openbot Openfind Openfind data gathere Openfind data gatherer Oracle Ultra Search OutfoxBot/0.5 PageGrabber Papa Foto pavuk PBWF pcBrowser penthesilea PerMan PGBot PhpDig Pingdom GIGRIB (http://www.pingdom.com) postrank ProPowerBot ProPowerBot/2.14 ProWebWalker psbot psycheclone Psycheclone Python-urllib QuepasaCreep QueryN Metasearch QueryN.Metasearch radian6 comment reader radian6 Feedfetcher Radiation Retriever Radiation Retriever 1.1 RB2B-bot RealDownload RealDownload/4.0.0.40 RealDownload/4.0.0.41 RealDownload/4.0.0.42 ReGet RepoMonkey RepoMonkey Bait & Tackle/v1.01 RepoMonkey Bait & Tackle RepoMonkey Bait & Tackle/v1.01 research-spider RMA Robozilla Roverbot RufusBot sbider Scooter/1.0 Scooter/1.0 scooter@pa.dec.com Scooter/1.1 (custom) Scooter/2.0 G.R.A.B. V1.1.0 Scooter/2.0 G.R.A.B. X2.0 Scooter2_Mercator_x-x.0 Scooter-3.0.EU Scooter-3.0.FS Scooter-3.0.HD Scooter-3.0QI Scooter-3.0.VNS Scooter-3.2 Scooter-3.2.BT Scooter-3.2.DIL Scooter-3.2.EX Scooter-3.2.JT Scooter-3.2.NIV Scooter-3.2.SF0 Scooter-3.2.snippet Scooter/3.3 Scooter-3.3dev Scooter/3.3.QA.pczukor Scooter/3.3_SF Scooter/3.3.vscooter Scooter-ARS-1.1 Scooter-ARS-1.1-ih Scooter_bh0-3.0.3 Scooter_trk3-3.0.3 scooter-venus-3.0.vns Scooter-W3-1.0 Scooter-W3.1.2 Scrubby SearchDaimon.com-dc searchpreview seekbot Seekbot Seekbot/1.0 SEOprofiler Shai'Hulud Shim-Crawler ShopWiki ShopWiki/1.0 SightupBot SiteBot SiteSnagger Slurp China SlySearch SmartDownload SmartDownload/1.2.76 (Win32; Apr 1 1999) SmartDownload/1.2.77 (Win32; Aug 17 1999) SmartDownload/1.2.77 (Win32; Feb 1 2000) SmartDownload/1.2.77 (Win32; Jun 19 2001) Snapbot Snappy Softlayer Server Sogou web spider sootle sosospider SpankBot spanner Speedy SpiderBot Sqworm Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux ssearcher100 Stanford Stanford Comp Sci suggybot SuperBot SuperBot/2.6 SuperBot/3.0 (Win32) SuperBot/3.1 (Win32) SuperHTTP SuperHTTP/1.0 Surfbot SurveyBot suzuran Szukacz Szukacz/1.4 tAkeOut Teleport TeleportPro Teleport Pro Teleport Pro/1.29 Teleport Pro/1.29.1590 Teleport Pro/1.29.1634 Teleport Pro/1.29.1718 Teleport Pro/1.29.1820 Teleport Pro/1.29.1847 Telesoft Templeton Teoma The Intraformant The.Intraformant TheNomad TightTwatBot Titan toCrawl toCrawl/UrlDispatcher True_Robot True_Robot/1.0 turingos TurnitinBot TurnitinBot/1.5 Tweetmeme TwengaBot Twiceler URL Control UrlDispatcher /URLFAN URL_Spider_Pro URLy Warning URLy.Warning VCI VCI WebViewer VCI WebViewer Win32 vobsub VoidEYE vscooter w3mir WatchDog/3.0 WebAuto WebAuto/3.40 (Win98; I) WebBandit WebBandit/3.50 WebCapture WebCapture 2.0 WebCatcher webcopier WebCopier WebCopier v.2.2 WebCopier v2.5 WebCopier v2.6 WebCopier v2.7a WebCopier v2.8 WebCopier v3.0 WebCopier v3.0.1 WebCopier v3.2 WebCopier v3.2a webcopy WebCopy webcrawl.net WebEmailExtrac WebEMailExtrac.* WebEnhancer WebFetch webfetch/2.1.0 WebFetcher WebGo IS Web Image Collector Web.Image.Collector WebLeacher WebmasterWorld Extractor WebmasterWorldForumBot webmirror WebMirror WebReaper Web Reaper WebReaper [info@webreaper.net] WebReaper v9.1 - www.otway.com/webreaper WebReaper v9.7 - www.webreaper.net WebReaper v9.8 - www.webreaper.net WebReaper vWebReaper v7.3 - www,otway.com/webreaper WebReaper [webreaper@otway.com] WebSauger WebSauger 1.20b WebSauger 1.20j WebSauger 1.20k website extractor Website eXtractor Website eXtractor (http:/www.asona.org) Website Quester Website.Quester Website Quester - www.asona.org Website Quester - www.esalesbiz.com/extra/ Webster Pro Webster.Pro WebStripper WebStripper/2.02 WebStripper/2.03 WebStripper/2.10 WebStripper/2.12 WebStripper/2.13 WebStripper/2.15 WebStripper/2.16 WebStripper/2.19 Web Sucker webvac WebVac WebVulnCrawl WebVulnScan WebWalk WebWasher WebWhacker WebZip WebZIP WebZIP/2.75 (http://www.spidersoft.com) WebZIP/2.75 (http:/www.spidersoft.com) WebZIP/3.65 (http://www.spidersoft.com) WebZIP/3.65 (http:/www.spidersoft.com) WebZIP/3.80 (http://www.spidersoft.com) WebZIP/3.80 (http:/www.spidersoft.com) WebZip/4.0 WebZIP/4.0 (http://www.spidersoft.com) WebZIP/4.0 (http:/www.spidersoft.com) WebZIP/4.1 (http://www.spidersoft.com) WebZIP/4.1 (http:/www.spidersoft.com) WebZIP/4.21 WebZIP/4.21 (http://www.spidersoft.com) WebZIP/4.21 (http:/www.spidersoft.com) WebZIP/5.0 WebZIP/5.0 (http://www.spidersoft.com) WebZIP/5.0 (http:/www.spidersoft.com) WebZIP/5.0 PR1 (http://www.spidersoft.com) WebZIP/5.0 PR1 (http:/www.spidersoft.com) wget wGet Wget Wget/1.10.2 Wget/1.5.2 Wget/1.5.3 Wget/1.6 Wget/1.7 Wget/1.8 Wget/1.8.1 Wget/1.8.1+cvs Wget/1.8.2 Wget/1.9-beta whitevector crawler Whitevector+Crawler Widow WikioFeedBot wikiwix-bot-3.0 Willow WinHTTrack Wise-Guys woozweb-monitoring woriobot WWW-Collector WWW-Collector-E WWWOFFLE Xaldon WebSpider Xaldon WebSpider 2.5.b3 Xenu Xenu Link Sleuth Xenu's Xenu's Link Sleuth 1.1c xGet Yahoo-MMCrawler YahooSeeker/CafeKelsa Yeti YodaoBot YRSPider Zao Zealbot Zeus Zeus 11389 Webster Pro V2.9 Win32 Zeus 11652 Webster Pro V2.9 Win32 Zeus 18018 Webster Pro V2.9 Win32 Zeus 26378 Webster Pro V2.9 Win32 Zeus 30747 Webster Pro V2.9 Win32 Zeus 32297 Webster Pro V2.9 Win32 Zeus 39206 Webster Pro V2.9 Win32 Zeus 41641 Webster Pro V2.9 Win32 Zeus 44238 Webster Pro V2.9 Win32 Zeus 51070 Webster Pro V2.9 Win32 Zeus 51674 Webster Pro V2.9 Win32 Zeus 51837 Webster Pro V2.9 Win32 Zeus 63567 Webster Pro V2.9 Win32 Zeus 6694 Webster Pro V2.9 Win32 Zeus 71129 Webster Pro V2.9 Win32 Zeus 82016 Webster Pro V2.9 Win32 Zeus 82900 Webster Pro V2.9 Win32 Zeus 84842 Webster Pro V2.9 Win32 Zeus 90872 Webster Pro V2.9 Win32 Zeus 94934 Webster Pro V2.9 Win32 Zeus 95245 Webster Pro V2.9 Win32 Zeus 95351 Webster Pro V2.9 Win32 Zeus 97371 Webster Pro V2.9 Win32 Zeus Link Scout ZyBorg Is not allowed to crawl
Robots.txt error
The reason of error location file content
Second lines of Disallow: path must start with /
Fifth lines of Disallow: path must start with /
avatar
hiwwe
Forumember

Posts : 31
Reputation : 1
Language : cn

Back to top Go down

Solved Re: Robots on my forum

Post by SLGray September 20th 2014, 5:40 pm

If Baidu gives mega tags, you can enter them in the administration panel.

AP > General > Forum Promotions > Search Engines
Press the create a mega tag button and fill in the information.

Also there was already a topic about this - https://help.forumotion.com/t135540-chinese-search-engine-baidu-not-included-i-created-forum .


Robots on my forum Slgray10

When your topic has been solved, ensure you mark the topic solved.
Never post your email in public.
SLGray
SLGray
Administrator
Administrator

Male Posts : 51463
Reputation : 3519
Language : English
Location : United States

https://forumsclub.com/gc/128-link-directory/

Back to top Go down

Solved Re: Robots on my forum

Post by hiwwe September 21st 2014, 12:49 am

SLGray wrote:If Baidu gives mega tags, you can enter them in the administration panel.

AP > General > Forum Promotions > Search Engines
Press the create a mega tag button and fill in the information.

Also there was already a topic about this - https://help.forumotion.com/t135540-chinese-search-engine-baidu-not-included-i-created-forum .

The problem with "AP > General > Forum Promotions > Search Engines" without any relationship, but the forum server "robots.txt" in the file "User-agent: Baiduspider" unexpectedly, this leads to any content forum server does not allow Baidu search engines crawl the station!

Your official should take "robots.txt" in the file "User-agent: Baiduspider" field is removed, so Baidu can included the contents of the forum I!
avatar
hiwwe
Forumember

Posts : 31
Reputation : 1
Language : cn

Back to top Go down

Solved Re: Robots on my forum

Post by hiwwe September 21st 2014, 1:40 am

SLGray wrote:If Baidu gives mega tags, you can enter them in the administration panel.

AP > General > Forum Promotions > Search Engines
Press the create a mega tag button and fill in the information.

Also there was already a topic about this - https://help.forumotion.com/t135540-chinese-search-engine-baidu-not-included-i-created-forum .

The question now is the forum under the root directory of the file "robots.txt" contains "User-agent: Baiduspider" field, which leads to Baidu search spider is completely shielded, so Baidu not included in any of the content of the forum! And this document I is not modify the permissions, you can modify the only official, search spiders shielding on Baidu please official release "robots.txt" file!
avatar
hiwwe
Forumember

Posts : 31
Reputation : 1
Language : cn

Back to top Go down

Solved Re: Robots on my forum

Post by hiwwe September 23rd 2014, 2:41 pm

The station site banned on Baidu
avatar
hiwwe
Forumember

Posts : 31
Reputation : 1
Language : cn

Back to top Go down

Solved Re: Robots on my forum

Post by Guest September 23rd 2014, 2:58 pm

Hey there,
Are you wanting to be able to access the Robots.txt file from the root directory to fix this? If this is what you are wanting to do then it is sadly not possible as we do not have access to the root directory at this time.
avatar
Guest
Guest


Back to top Go down

Solved Re: Robots on my forum

Post by Shadow September 23rd 2014, 3:28 pm

There are some robots on your forum. Do not worry it is normal. Robots indexe your forum on search engines. So there was nothing to worry about with that.


Next time, thanks for using English speaking and not Chinese.


Regards
Shadow
Shadow
Manager
Manager

Male Posts : 16217
Reputation : 1831
Language : French, English

Back to top Go down

Back to top

- Similar topics

 
Permissions in this forum:
You cannot reply to topics in this forum