An actual Linux problem/solution (slicehost)

[Dan Dubovik]

If you are running into issues with too many httpd processes spinning up,
would it make more sense to limit the number of child processes it can spin
up?  You could run into similar constraints if you hit MaxClients (assuming
apache here?), however, you could monitor for that to trigger your restart
instead.    Is the cause of the load from the web server spinning up a bunch
of processes all of a sudden? Or from the code it is attempting to parse /
deliver?  Increasing the number of servers that are started up could also
help in preventing a spike of load if it is the former.  If the latter is
the issue, perhaps some other optimizations could be made.  I've had some
pretty decent success with memcache in helping reduce mysql traffic and
server load.

While the script you have here could keep the server from having to be
rebooted, it seems that having that 30 second sleep in the middle of a bunch
of traffic, would go against the (supposed) goal of driving traffic in the
first place.

-- Dan.

On Wed, May 19, 2010 at 12:12 PM, Matt Graham <danceswithcrows at usa.net>wrote:

> I have a slicehost running Gentoo.  Because I'm cheap, I got the smallest
> slice available (256M RAM, 10G disk, 100G bandwidth, small amount of CPU).
> This usually worked fine since I'm not running a whole lot of stuff or
> getting
> high traffic.
>
> However, if the slice got a large amount of traffic in a short period of
> time,
> the load on it would go through the roof.  Then it'd run out of RAM, then
> usually run out of swap.  This usually led to an unresponsive slice that
> couldn't be recovered without a reboot.  This was *really* inconvenient,
> especially if it happened while I was sleeping or busy.
>
> So after screwing around for a while and tweaking some parameters, I came
> up
> with a small Perl script, below in pseudocode:
>
> daemonize();
> $count=0;
> while(1){
>  $load=check_load();
>  $date=`date`;
>  if($load > 10.0 and $count == 0){ # seems about right
>    writelog("$date load $load WHOA! force-killing everything.\n");
>    force_kill("httpd","mysqld");
>    sleep(30);
>    restart("httpd","mysqld");
>    $count=2; # seems OK
>    }
>  elsif($load > 3.0 and $count == 0){ # seems about right
>    writelog("$date load $load restarting everything\n");
>    restart("httpd","mysqld");
>    $count=2; # seems about right
>    }
>  sleep(60); # maybe 45?  This works though.
>  if($count > 0){ $count--; }
>  }
> # end
>
> ....with this in place, things have been working for over a week with no
> manual reboots required.  Restarts of one stripe or another get triggered
> about once or twice a day, at random times, and there's not any correlation
> that I could see to what's in the apache logs or the slow query logs.
>
> Later today, if I don't get beaten down with 5 tons of other junk, I'll
> post
> the real code.  Maybe some other slicehost user will find it useful.
> Comments/criticism/stuff RELATED TO THIS HERE MESSAGE are welcome.  cat
> "Offtopic stuff" >> offtopic_thread , please.  Thank you, and keep reaching
> for the stairs.
>
> --
> Matt G / Dances With Crows
> The Crow202 Blog:  http://crow202.org/wordpress/
> There is no Darkness in Eternity/But only Light too dim for us to see
>
>
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
> To subscribe, unsubscribe, or to change your mail settings:
> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.PLUG.phoenix.az.us/pipermail/plug-discuss/attachments/20100519/14d73c9c/attachment.htm>