Printer Friendly Version
Email this thread to a friend
|
continue contract with yahoo register domain (In: Domains, Hosting, DNS and Server Issues)
Robots crawling (In: Members Lounge)
Yahoo Hates Me ? (In: Yahoo!)
Using Robots.txt on Your Web Site (In: General Search Engine Optimization)
Featured Web Site Template |
|
Reflects user activity within the last 5 minutes
|
|
| Member |
Message |
jsmith06
Joined: Nov 20, 2006
# Posts: 53
|
Posted: 2007-May-15 15:31
Anybody know how long it takes Slurp to start obeying the robots.txt file? We made an update a couple weeks ago and it's still crawling stuff we don't want it to.
|
 |
dudibob
Joined: Oct 13, 2005
# Posts: 1459
|
Posted: 2007-May-15 15:37
hmmm, should of certainly paid it the proper attention by then, could you post the code your using for your robots.txt file just to check it's alright?
|
|
jsmith06
Joined: Nov 20, 2006
# Posts: 53
|
Posted: 2007-May-15 15:55
This is a snippet of it:
# $Revision: 18073 $
# DE1665
User-agent: *
Disallow: /*Bedrooms
Disallow: /*Sleeps
This is designed to stop the crawl of URLs that contain the words Bedrooms and / or Sleeps in them.
Google began honoring the update within a day or two of posting it.
|
|
You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
- You have not yet logged in, or registered properly as a member
- You are a member, but no longer have posting rights.
- This is a private forum, for which you do not have permissions.
If you are a recent member, it's possible that you simply have not yet confirmed your account. Please
check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions
contained within.
If you cannot find this message, click here to Re-Send it.
|
If you are still experiencing problem, please read the
Login Assistance
Article for some advice on what may be causing your login not to work properly.
|
Switch to Advanced Editor and ...
Create a New Topic
or Reply to this Thread
|
|
Inbox
