An actual Linux problem/solution (slicehost)

I have a slicehost running Gentoo. Because I'm cheap, I got the smallest
slice available (256M RAM, 10G disk, 100G bandwidth, small amount of CPU).
This usually worked fine since I'm not running a whole lot of stuff or getting
high traffic.

However, if the slice got a large amount of traffic in a short period of time,
the load on it would go through the roof. Then it'd run out of RAM, then
usually run out of swap. This usually led to an unresponsive slice that
couldn't be recovered without a reboot. This was *really* inconvenient,
especially if it happened while I was sleeping or busy.

So after screwing around for a while and tweaking some parameters, I came up
with a small Perl script, below in pseudocode:

daemonize();
$count=0;
while(1){
  $load=check_load();
  $date=`date`;
  if($load > 10.0 and $count == 0){ # seems about right
    writelog("$date load $load WHOA! force-killing everything.\n");
    force_kill("httpd","mysqld");
    sleep(30);
    restart("httpd","mysqld");
    $count=2; # seems OK
    }
  elsif($load > 3.0 and $count == 0){ # seems about right
    writelog("$date load $load restarting everything\n");
    restart("httpd","mysqld");
    $count=2; # seems about right
    }
  sleep(60); # maybe 45?  This works though.
  if($count > 0){ $count--; }
  }
# end

....with this in place, things have been working for over a week with no
manual reboots required. Restarts of one stripe or another get triggered
about once or twice a day, at random times, and there's not any correlation
that I could see to what's in the apache logs or the slow query logs.

Later today, if I don't get beaten down with 5 tons of other junk, I'll post
the real code. Maybe some other slicehost user will find it useful.
Comments/criticism/stuff RELATED TO THIS HERE MESSAGE are welcome. cat
"Offtopic stuff" >> offtopic_thread , please. Thank you, and keep reaching
for the stairs.

--
Matt G / Dances With Crows
The Crow202 Blog: http://crow202.org/wordpress/
There is no Darkness in Eternity/But only Light too dim for us to see


---------------------------------------------------
PLUG-discuss mailing list - PLUG-discuss@lists.plug.phoenix.az.us
To subscribe, unsubscribe, or to change your mail settings:
http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss