Printer Friendly Version
Email this thread to a friend
|
Featured Web Site Template |
|
Reflects user activity within the last 5 minutes
|
|
| Member |
Message |
betty7
Joined: Sep 23, 2002
# Posts: 8
|
Posted: 2004-Jan-14 09:27
I have disallowed folders from google in my robot.txt file but have noticed that some pages from these folders are being listed. The listings in Google are the url of the page with no title, description/comment.
Does anyone know whats happening here? And how i can stop google from indexing these pages - r these pages actually being crawled by Googlebot?
(I have validated my robots.txt file and it said no errors were found!!)
Thanks for any comments
|
 |
bhartzer
Staff
Joined: Jun 08, 2000
# Posts: 7042
|
Posted: 2004-Jan-15 16:20
The only way to really stop Google from indexing these pages is to either remove them completely from the site or put them in a password-protected area.
That said, you might want to check the backlinks going to those pages. If you can find any links going to those pages, then I would remove them. It could be that you're linking to those pages is causing those pages to be spidered.
Also, take a look at your log files to see if the spiders are actually going to those pages.
Unfortunately, there's really not much I can tell you since I don't have the specific urls of the pages that you are referring to. If you would like me to take a look then send me a private message and I'd be glad to take a look.
|
|
unreviewed
Joined: Dec 07, 2000
# Posts: 6776
|
Posted: 2004-Jan-15 17:25
betty7, did you have the robots.txt file in place before Google had a chance to index the files?
If that is the case, they will eventually be dropped, or in the meantime you can use http://www.google.com/webmasters/3.html#removed
|
|
You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
- You have not yet logged in, or registered properly as a member
- You are a member, but no longer have posting rights.
- This is a private forum, for which you do not have permissions.
If you are a recent member, it's possible that you simply have not yet confirmed your account. Please
check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions
contained within.
If you cannot find this message, click here to Re-Send it.
|
If you are still experiencing problem, please read the
Login Assistance
Article for some advice on what may be causing your login not to work properly.
|
Switch to Advanced Editor and ...
Create a New Topic
or Reply to this Thread
|
|
Inbox
