The forum of the forums
Welcome to the Official Support Forum of Forumotion!

To take full advantage of everything offered by our forum, please log in if you are already a member, or join our community if you've not yet.



Create a free forum like this one.

How do I edit the Robots.txt file?

View previous topic View next topic Go down

How do I edit the Robots.txt file?

Post by Guest on July 9th 2010, 11:42 pm

This is my Robots.txt

Code:

1. User-agent: Mediapartners-Google
2. Disallow:
3.
4. User-agent: Mediapartners-Google*
5. Disallow:
6.
7. User-agent: eDintorni crawler
8. Disallow:
9.
10. User-agent: *
11. Disallow: /login.forum
12. Disallow: /modcp.forum
13. Disallow: /posting.forum
14. Disallow: /privmsg.forum
15. Disallow: /memberlist.forum
16. Disallow: /stats.htm
17. Disallow: /donate.forum
18. Disallow: /report_abuse.forum
19. Disallow: /contact.forum
20. Disallow: /go/
21.
22. User-agent: Alexibot
23. User-agent: asterias
24. User-agent: BackDoorBot/1.0
25. User-agent: Black Hole
26. User-agent: BlowFish/1.0
27. User-agent: BotALot
28. User-agent: BuiltBotTough
29. User-agent: Bullseye/1.0
30. User-agent: BunnySlippers
31. User-agent: Cegbfeieh
32. User-agent: CheeseBot
33. User-agent: CherryPicker
34. User-agent: CherryPickerElite/1.0
35. User-agent: CherryPickerSE/1.0
36. User-agent: CopyRightCheck
37. User-agent: ConveraCrawler
38. User-agent: cosmos
39. User-agent: Crescent
40. User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0
41. User-agent: DISCo Pump 3.1
42. User-agent: DittoSpyder
43. User-agent: EmailCollector
44. User-agent: EmailSiphon
45. User-agent: EmailWolf
46. User-agent: EroCrawler
47. User-agent: ExtractorPro
48. User-agent: Foobot
49. User-agent: Harvest/1.5
50. User-agent: hloader
51. User-agent: httplib
52. User-agent: humanlinks
53. User-agent: InfoNaviRobot
54. User-agent: JennyBot
55. User-agent: Jetbot
56. User-agent: Kenjin Spider
57. User-agent: LexiBot
58. User-agent: libWeb/clsHTTP
59. User-agent: LinkextractorPro
60. User-agent: LinkScan/8.1a Unix
61. User-agent: LinkWalker
62. User-agent: lwp-trivial
63. User-agent: lwp-trivial/1.34
64. User-agent: Mata Hari
65. User-agent: Microsoft URL Control - 5.01.4511
66. User-agent: Microsoft URL Control - 6.00.8169
67. User-agent: MIIxpc
68. User-agent: MIIxpc/4.2
69. User-agent: Mister PiX
70. User-agent: moget
71. User-agent: moget/2.1
72. User-agent: NetAnts
73. User-agent: NetAttache
74. User-agent: NetAttache Light 1.1
75. User-agent: NetMechanic
76. User-agent: NICErsPRO
77. User-agent: Offline Explorer
78. User-agent: Openfind
79. User-agent: Openfind data gathere
80. User-agent: ProPowerBot/2.14
81. User-agent: ProWebWalker
82. User-agent: psbot
83. User-agent: QueryN Metasearch
84. User-agent: RepoMonkey
85. User-agent: RepoMonkey Bait & Tackle/v1.01
86. User-agent: RMA
87. User-agent: SiteSnagger
88. User-agent: SpankBot
89. User-agent: spanner
90. User-agent: SuperBot
91. User-agent: SuperBot/2.6
92. User-agent: suzuran
93. User-agent: Szukacz/1.4
94. User-agent: Teleport
95. User-agent: Telesoft
96. User-agent: The Intraformant
97. User-agent: TheNomad
98. User-agent: TightTwatBot
99. User-agent: Titan
100. User-agent: toCrawl/UrlDispatcher
101. User-agent: True_Robot
102. User-agent: True_Robot/1.0
103. User-agent: turingos
104. User-agent: URLy Warning
105. User-agent: VCI
106. User-agent: VCI WebViewer VCI WebViewer Win32
107. User-agent: Web Image Collector
108. User-agent: WebAuto
109. User-agent: WebBandit
110. User-agent: WebBandit/3.50
111. User-agent: WebCopier
112. User-agent: webcopy
113. User-agent: WebEnhancer
114. User-agent: WebmasterWorldForumBot
115. User-agent: webmirror
116. User-agent: WebReaper
117. User-agent: WebSauger
118. User-agent: website extractor
119. User-agent: Website Quester
120. User-agent: Webster Pro
121. User-agent: WebStripper
122. User-agent: WebStripper/2.02
123. User-agent: WebZip
124. User-agent: WebZip/4.0
125. User-agent: Wget
126. User-agent: Wget/1.5.3
127. User-agent: Wget/1.6
128. User-agent: WinHTTrack
129. User-agent: WWW-Collector-E
130. User-agent: Xenu's
131. User-agent: Xenu's Link Sleuth 1.1c
132. User-agent: Zeus
133. User-agent: Zeus 32297 Webster Pro V2.9 Win32
134. User-Agent: MJ12bot
135. User-agent: HTTrack
136. User-agent: HTTrack 3.0
137. User-agent: TurnitinBot
138. User-agent: QuepasaCreep
139. Disallow: /

I want to know how I can modify this to disallow and allow certain robots to access my forum. I don't like the look of that bot that says "backdoor bot"... Theres seems to be no options for me to modify this =\

Guest
Guest


Back to top Go down

Re: How do I edit the Robots.txt file?

Post by Darren1 on July 10th 2010, 2:03 am

Hello,

I'm sorry but we cannot edit the robots.txt TT

By reading your robots.txt, it's stating that the "backdoor bot" is already DISallowed, so your safe;)

Darren1
Helper
Helper

Male Posts : 11853
Reputation : 563
Language : English

Back to top Go down

Re: How do I edit the Robots.txt file?

Post by Guest on July 10th 2010, 4:00 am

Oh... I looked at some other robots.txt on other sites and saw "Disallow: /" underneath every single robot that they didnt want to allow. I only see "Disallow: /" one time on my robot.txt all the way at the bottom, so I wasn't sure if it was blocking all those other bots.

Guest
Guest


Back to top Go down

View previous topic View next topic Back to top


 
Permissions in this forum:
You cannot reply to topics in this forum