Printer Friendly Version
Email this thread to a friend
|
Featured Web Site Template |
|
Reflects user activity within the last 5 minutes
|
|
| Member |
Message |
michaelalan
Joined: Feb 13, 2004
# Posts: 19
|
Posted: 2004-Mar-09 03:39
I have landing pages that I want the search engines to see and then I have the Miva Merchant Files that I don't want the search engines to see because they have duplicate content but no meta tags and such.
I have searched the forums here but I have not found any solid help. I would appreciate any advice you can offer!
Michael Alan
|
 |
crash
Staff
Joined: Dec 02, 2003
# Posts: 10626
|
Posted: 2004-Mar-09 06:36
You only use the robots.txt to EXCLUDE files/directories.
http://www.robotstxt.org/wc/exclusion.html#robotstxt
That shows you how to write it.
|
|
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10465
|
Posted: 2004-Mar-09 17:47
Or, put <meta name="robots" content="noindex,follow"> on all pages you do not want indexed.
|
|
patrickh
Joined: Oct 05, 2001
# Posts: 2187
|
Posted: 2004-Mar-09 18:01
Its best to use both robots.txt and the meta robots tag, as each standard is not 100% used.
|
|
crash
Staff
Joined: Dec 02, 2003
# Posts: 10626
|
Posted: 2004-Mar-09 20:10
and keep in mind that neither is a guarantee that they will honor your requests.
|
|
michaelalan
Joined: Feb 13, 2004
# Posts: 19
|
Posted: 2004-Mar-10 02:23
Thanks everyone. The "<meta name="robots" content="noindex,follow">" solved a different problem. You guys must have ESP.
Here is what I have for a robots.txt file. Is it just that easy?
User-agent: *
Disallow: /miva/
Disallow: /logs/
Disallow: /old/
Disallow: /ecomm/
Disallow: /database/
|
|
thejenn
Joined: Aug 08, 2001
# Posts: 9196
|
Posted: 2004-Mar-10 17:08
Yep. 
|
|
g1smd
Staff
Joined: Jul 28, 2002
# Posts: 10465
|
Posted: 2004-Mar-12 11:41
That should do the job.
|
|
You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
- You have not yet logged in, or registered properly as a member
- You are a member, but no longer have posting rights.
- This is a private forum, for which you do not have permissions.
If you are a recent member, it's possible that you simply have not yet confirmed your account. Please
check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions
contained within.
If you cannot find this message, click here to Re-Send it.
|
If you are still experiencing problem, please read the
Login Assistance
Article for some advice on what may be causing your login not to work properly.
|
Switch to Advanced Editor and ...
Create a New Topic
or Reply to this Thread
|
|
Inbox
