Printer Friendly Version
Email this thread to a friend
|
For Sale Russia/USA Marriage/Dating/Meeting Site (In: I Want to Sell My Website)
I want to sell my site, mzkforums.com (In: I Want to Sell My Website)
I want to sell my site, mzkforums.com (In: I Want to Sell My Website)
For Sale - Child Gift Web Site (In: I Want to Sell My Website)
For Sale PR3 Site with Lots of Authority (In: I Want to Sell My Website)
Featured Web Site Template |
|
There are 0 guests and 1 members in the forums right now.
Reflects user activity within the last 5 minutes
|
|
| Member |
Message |
david68
Joined: May 16, 2005
# Posts: 144
|
Posted: 2007-Aug-07 14:51
I want better rankings and more traffic, but what bots actually help me and aren't just out there for data mining to sell information?
If they ignore robots.txt (everyone is disallowed until I allow them) I block them. I got a new bot, hit robots.txt, hasn't viewed a page so it's obeying it, but I googled about it and some people call it a data miner. It's called BlogPulseLive - hostname comes back as intelliseek dot com which redirects to nielsenbuzzmetrics dot com which openly admits it's a data miner company. So I assume it doesn't help me at all?
Is there a list of good bots out there? Some blog bots help get traffic.
|
 |
Hampstead
Joined: Feb 20, 2001
# Posts: 2015
|
Posted: 2007-Aug-07 17:09
There is no easy way of banning data mining or "bad" bots. They simply won't adhere to robots txt or tag. Indeed, they could simply identify themselves as browsers.
If you're really worried about it, you could try to find a list of IP adresses asociated with the various bots and ban them, but there is very little point.
By the way, all bots by definition are there for data mining purposes.
|
|
david68
Joined: May 16, 2005
# Posts: 144
|
Posted: 2007-Aug-07 17:20
By the way, all bots by definition are there for data mining purposes
Yes, but google/yahoo/msn at least help the cause in the process.
The question was: is there a list of GOOD bots, that I could specially "allow" in robot.txt. I guess I should have phrased it better.
|
|
Hampstead
Joined: Feb 20, 2001
# Posts: 2015
|
Posted: 2007-Aug-07 18:15
It is quite normal to allow all bots.
|
|
david68
Joined: May 16, 2005
# Posts: 144
|
Posted: 2007-Aug-07 18:32
Well, maybe I'm quite abnormal as I rather not give permission to companies who sell MY information without permission without me gaining anything in return. 
|
|
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10438
|
Posted: 2007-Aug-07 23:44
A Google search will find a number of sites that have compiled a list of what they consider to be "bad bots". I would use those as a start, making sure to review as many as possible to ensure that the list is still correct.
|
|
Hampstead
Joined: Feb 20, 2001
# Posts: 2015
|
Posted: 2007-Aug-08 07:05
I understand your problem with these companies, but the "bad bots" will take no notice of your robots.txt or your robots noindex tag and will simply crawl your site anyway.
|
|
david68
Joined: May 16, 2005
# Posts: 144
|
Posted: 2007-Aug-08 13:36
Hmph. I don't care about "bad bots" - I asked for a list of "GOOD" bots Bots which OBEY robots.txt that I should allow. BLOGPULSELIVE keeps asking for permission, I was wondering if I should allow them or if they won't really help me get rankings. I did google, but honestly the search engines are getting pretty crappy - bad results or no results.
I know most people allow everyone, but honestly why waste my bandwidth for something that won't benifit me.
|
|
dudibob
Joined: Oct 13, 2005
# Posts: 1462
|
Posted: 2007-Aug-08 14:08
banning bad bots will require a fair bit of server scripting of banning IPs to make sure they don't come in and new bad bots are generated every day so you will have to update your script a fair bit :s
Personally I'd say don't worry about them, just look out for MSNbot, Slurp (yahoo), Googlebot and Asks (I forget what it's called).
|
|
david68
Joined: May 16, 2005
# Posts: 144
|
Posted: 2007-Aug-08 14:53
OMG - forgive the rant/flame - but I never asked about BAD BOTS.
I asked: Is BlogPulseLive a GOOD bot?
I asked: What ones are considered GOOD (like Slurp, Google, etc) that I should specifically allow in robots.txt.
Sheesh.
Moderator - please LOCK this topic Better yet, move it to the xVault.
|
|
Hampstead
Joined: Feb 20, 2001
# Posts: 2015
|
Posted: 2007-Aug-08 15:59
Allow all bots by not using the robots.txt. The "good bots" will then have access to your site without you having to worry about it.
It's quite normal to do this.
|
|
david68
Joined: May 16, 2005
# Posts: 144
|
Posted: 2007-Aug-08 16:06
Thanks for your comments - even though I don't agree with them nor were they useful.
|
|
Hampstead
Joined: Feb 20, 2001
# Posts: 2015
|
Posted: 2007-Aug-08 16:47
|
|
You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
- You have not yet logged in, or registered properly as a member
- You are a member, but no longer have posting rights.
- This is a private forum, for which you do not have permissions.
If you are a recent member, it's possible that you simply have not yet confirmed your account. Please
check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions
contained within.
If you cannot find this message, click here to Re-Send it.
|
If you are still experiencing problem, please read the
Login Assistance
Article for some advice on what may be causing your login not to work properly.
|
Switch to Advanced Editor and ...
Create a New Topic
or Reply to this Thread
|
|
Inbox
