automation of collecting web site size

Top Page
Attachments:
Message as email
+ (text/plain)
Delete this message
Reply to this message
Author: EricJohnsonej@netasm.com
Date:  
Subject: automation of collecting web site size
On Fri, 13 Oct 2000 wrote:
: \_: There must be a way to get this info, without either sucking the whole
: \_: site down, or having access to the webserver?
: \_:
: \_: Anyone have any ideas, suggestions, etc.?
: \_
: \_ lynx -dump -head $URL | grep Content-Length | cut -d: -f 2
:
: Hum, this could lead to amusing issues if it hits a cgi script (or
: other) that doesn't play nice with issuing Content-Length.

You bring up a good point. Keep in mind this was a suggestion to
help get Wes started, not a complete solution.

: Furthermore just getting the head doesn't tell you what else might be
: linked to it down the tree.

Huh? Are you saying sizeof(Content-Length) != sizeof(website). That's
kind of restating the obvious, isn't it? I kind of presumed Wes would
understand the fragment had to be looped, called recursively, whatever;
and also called only if certain conditions he designed were met, like
your comment regaring CGI.

    --Eric