Re: Web crawling

Top Page
Attachments:
Message as email
+ (text/plain)
Delete this message
Reply to this message
Author: Craig White
Date:  
To: plug-discuss
Subject: Re: Web crawling
On Mon, 2005-02-28 at 07:51 -0700, Nathan England wrote:
> I am looking for a way to create a list of every file on a web site.
> Similar to the way wget will check your bookmarks, but I want something that
> will scan a website and create a list of everything found.
>
> Anyone know of anything like this?

----
wget has a '--spider' option

man wget

Craig

---------------------------------------------------
PLUG-discuss mailing list -
To subscribe, unsubscribe, or to change you mail settings:
http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss