Strange Server Behavior

Tue Jun 26 21:02:00 MST 2012

What about pings from the server? Also my paranoia about this would have me
checking the arp tables to see if the ip address is getting
mis-somethinged. Also see if uou can make a task that will wrife to a file
once every qp sex and see if the server is falling asleep or if it is
network related.  It may be nic or switch for example.
On Jun 26, 2012 7:42 PM, "Mark Phillips" <mark at phillipsmarketing.biz> wrote:

> My ping test is not what I expected but still shows a problem.
>
> I setup the test to ping (64 bytes, ttl =64) the problem server every 10
> seconds from my laptop. Both are plugged in, on the same subnet. The boxes
> are about 5 feet apart. Here are the results:
>
> 3434 packets transmitted, 3307 received, 3% packet loss, time 34336321ms
> rtt min/avg/max/mdev = 0.100/0.231/4.332/0.275 ms
>
> I found 38 instances where the time stamps for the pings "hiccuped" and
> there was a delay. Each hiccup lasted (min/avg/max) 10/44/70 seconds. The
> time between hiccups was (min/avg/max) 1:20/14:29/29:00 minutes.
>
> I grep'd the log files for messages around these 38 incidents, but did not
> find any messages in any of the logs. However this is not a good test, so I
> just trolled the logs and didn't find anything significant.
>
> Do these numbers strike a chord with anyone?
>
> I will check the caps on the MB later this week.
>
> I looked in syslog, and could not find any correlation with the 38
> hiccups. However, what do these two cron jobs do, since the run quite
> frequently:
> CMD (   cd / && run-parts --report /etc/cron.hourly)
> CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find
> /var/lib/php5/ -type f -cmin +$(/usr/lib/php5/maxlifetime) -delete)
>
> A search through the log files for "error" does not return anything
> interesting.
>
> Since apache and few other apps were running on the server, I will run the
> ping test again tonight after I kill everything on the server.
>
> Thanks!
>
> Mark
>
> On Tue, Jun 26, 2012 at 9:35 AM, Carruth, Rusty <
> Rusty.Carruth at smartstoragesys.com> wrote:
>
>> Curious how your test turned out.****
>>
>> ** **
>>
>> You may also want to run an iostat to a file and see if that correlates
>> to the slow responses.****
>>
>> ** **
>>
>> However, that ‘bulging capacitor’ thing others have mentioned sounds like
>> a pretty convincing coincidence, as it were….****
>>
>> ** **
>>
>> (I will say that USUALLY I’d agree with JD – Iobound or low RAM (and thus
>> iobound on swap space) are the only things I’ve seen that cause
>> unresponsiveness (never seen an overheat slow it down, usually it just dies
>> suddenly.  I probably get fast overheating and not slow increases in heat
>> levels J)****
>>
>> ** **
>>
>> OH!  WAIT!  I just remembered another event – and it WON’T show up in
>> normal performance logs.  If ‘you’ send a command to a disk drive, and it
>> goes busy for a long time, your system can become totally locked until the
>> timeout happens and the kernel gives up.  (If that happens, there SHOULD be
>> a timeout recorded in the syslog or /var/log/messages.  Check there for
>> timeouts on disk drives or hard resets or such).  (I know this because of
>> where I work J)  (Disk drives are supposed to acknowledge the command
>> almost immediately.  It is almost always a bad thing when the drive takes
>> the command but does not finish the initial command handshake sequence…
>> You might want to look at the S.M.A.R.T. attributes for your drives as well
>> to see if any of them are showing ‘pre-fail’ conditions)****
>>
>> ** **
>>
>> Rusty****
>>
>> ** **
>>
>> *From:* plug-discuss-bounces at lists.plug.phoenix.az.us [mailto:
>> plug-discuss-bounces at lists.plug.phoenix.az.us] *On Behalf Of *Mark
>> Phillips
>> *Sent:* Monday, June 25, 2012 10:13 PM
>> *To:* Main PLUG discussion list
>> *Subject:* Re: Strange Server Behavior****
>>
>> ** **
>>
>> Right now, the server is not doing anything but sitting there....
>>
>> Tasks:  98 total,   1 running,  97 sleeping,   0 stopped,   0 zombie
>> Cpu(s):  0.0%us,  0.3%sy,  0.0%ni, 99.7%id,  0.0%wa,  0.0%hi,  0.0%si,
>> 0.0%st
>> Mem:   1033780k total,   217560k used,   816220k free,     6220k buffers
>> Swap:  2019320k total,        0k used,  2019320k free,    94056k cached
>>
>> Plenty of swap, not very busy. It may be over heating, but not sure why.
>>
>> I am going to run a test tonight - ping every 10 seconds and time stamp
>> the output into a file. Perhaps I will see gaps or unusually long response
>> times and I can correlate that with the log files.
>>
>> Mark****
>>
>> On Mon, Jun 25, 2012 at 10:09 PM, JD Austin < <jd at twingeckos.com>> wrote:
>> ****
>>
>> I've had servers that act like that.. usually they're over heating,
>> completely I/O bound, or swapping due to low available memory. ****
>>
>> ** **
>>
>> On Mon, Jun 25, 2012 at 10:00 PM, Mark Phillips <
>> mark at phillipsmarketing.biz> wrote:****
>>
>> Nope - everything just stops - ping waits for a response, web services
>> just wait for the server, file transfers stop and wait.......as if time
>> just stopped for the server, then starts again without any errors being
>> evident.
>>
>> Mark****
>>
>> ** **
>>
>> On Mon, Jun 25, 2012 at 9:57 PM, Stephen < > wrote:****
>>
>> Can you do access any other services hosted by the server during this
>> time? Or even an extended ping?****
>>
>> On Jun 25, 2012 9:53 PM, "Mark Phillips" < > wrote:****
>>
>> I have a headless server running Linux version 2.6.32-5-686 (Debian
>> 2.6.32-45) (dannf at debian.org) (gcc version 4.3.5 (Debian 4.3.5-4) and no
>> X or window manager, and I have noticed in the past couple of days that
>> when I ssh in the server it occasionally stops responding for a minute or
>> two, then comes back as if nothing had happened. It is a random event -
>> maybe once an hour. I cannot find anything in the logs - no error messages.
>> There is nothing wrong with the machine where I initiated the ssh session,
>> and it is not connected to ssh. The server completely stops responding,
>> then comes back as if nothing had happened.
>>
>> How would I go about diagnosing this problem?
>>
>> Thanks,
>>
>>
>> ****
>>
>> ** **
>>
>> ---------------------------------------------------
>> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
>> To subscribe, unsubscribe, or to change your mail settings:
>> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>>
>
>
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
> To subscribe, unsubscribe, or to change your mail settings:
> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.PLUG.phoenix.az.us/pipermail/plug-discuss/attachments/20120626/7b7b0776/attachment.html>