Printer Friendly Version Print this thread
Email this thread to a friend eMail this thread to a friend
Featured Web Site Template

Hundreds More at Free Site Templates.com!

Web Site Partners
Sponsored Links
Jet City Software
 
Whos Here ?
There are 0 guests and 1 members in the forums right now.
Reflects user activity within the last 5 minutes
Moderator(s): excell, SportsGuy, g1smd
Forum Index · Search Engine Forums · SEF Community and Networking · Members Lounge · Pet Spider - where can I get one?
Member Message

SportsGuy
Staff
Joined: Aug 30, 2002
# Posts: 3600

View the profile for SportsGuy Send SportsGuy a private message

Posted: 2008-Apr-28 18:00
Edit Message Delete Message Reply to this message

So, say a person's looking for code to build a crawler.

Nothing fancy, just a basic crawler that you can:

1 - feed a select list of links into to tell it to start crawling from (think the main page of different sites)
2 - tell the spider to watch for select keywords (give it a list to look for)
3 - index only those with the keyword/phrase mentioned
4 - spec whether the mention happens in the metas, the body, the domain, the URL, etc.

Assume the desire is for a crawler that will live on a server and can be given it's own IP if absolutely needed - MySQL database friendly and, thus, likely php-based, or at least Linux box friendly.

...and it's care-taker is not a programmer, so this pet spider will need to come with instructions. wink



Quadrille
Joined: Nov 15, 2000
# Posts: 1064

View the profile for Quadrille Send Quadrille a private message

Posted: 2008-Apr-28 19:27
Edit Message Delete Message Reply to this message

I really don't know ... but there must be an open source answer?



formerskeptic
Joined: Oct 05, 2001
# Posts: 301

View the profile for formerskeptic Send formerskeptic a private message

Posted: 2008-Apr-29 06:26
Edit Message Delete Message Reply to this message

Pretty tall order you got there smile Try Risearch.
The script has most if not all the features you're looking for wink


Forum Index · Search Engine Forums · SEF Community and Networking · Members Lounge · Pet Spider - where can I get one?
You are not permitted to post messages in this forum or topic, because of one or more of the following reasons:
  1. You have not yet logged in, or registered properly as a member
  2. You are a member, but no longer have posting rights.
  3. This is a private forum, for which you do not have permissions.

If you are a recent member, it's possible that you simply have not yet confirmed your account. Please check your email for a message entitled 'JimWorld Forums: Confirm Your Account' and follow the instructions contained within.

If you cannot find this message, click here to Re-Send it.

If you are still experiencing problem, please read the Login Assistance Article for some advice on what may be causing your login not to work properly.

Switch to Advanced Editor and ... Create a New Topic or Reply to this Thread

New posts Forum is locked
© 1995  ·  iWeb, Inc  ·  DBA JimWorld Productions