Re: Web crawling

Startseite
Anhänge:
Nachricht
+ (text/plain)
Nachricht löschen
Nachricht beantworten
Autor: Craig White
Datum:  
To: plug-discuss
Betreff: Re: Web crawling
On Mon, 2005-02-28 at 07:51 -0700, Nathan England wrote:
> I am looking for a way to create a list of every file on a web site.
> Similar to the way wget will check your bookmarks, but I want something that
> will scan a website and create a list of everything found.
>
> Anyone know of anything like this?

----
wget has a '--spider' option

man wget

Craig

---------------------------------------------------
PLUG-discuss mailing list -
To subscribe, unsubscribe, or to change you mail settings:
http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss