Am 28. Jun, 2011 schwätzte Bryan O'Neal so:

> I too would like some answers on how to track down the source of Io
> wait but I can ask some other questions. Did you check the raid

First off, don't believe sendmail when it claims you have an empty mail
queue. Apparently it's lazy and if the mail queue gets too large sendmail
stops trying to count and just says the queue is empty :(. The machine
only gets a few emails a minute, so an empty queue made sense.

James' suggestion of looking for processes in a blocked state was what
finally got me. I had done that, but was apparently been too bleary-eyed
to notice the capital D that accompanied each sendmail process.

Mike's suggestion of iotop looks good, but it turns out I can't use it on
that machine right now anyway.

I was starting to use oprofile when I finally figured out the problem.

Lisa's suggestion of updating ( or better yet avoiding ) proprietary
firmware is also good. A few more things in our datacenters to fix before
I can add firmware updates to the rotation, but that's definitely now
something on my radar.

> controllers health? BBU in good shape? Still have all your cache? did

3ware tool claimed the hardware is in good shape.

> you end up in write through? Did you tweak things before and lose your
> tweaks becuse they were not in the appropriate confs? by tweaks I mean
> things like you fs levelers or disabling atime etc.

Not that I know of and if we did they're gone as that machine had been on
the air long since the guy who set it up left the company...

I'm documenting and/or moving to puppet all such things as I find them.

ciao,

der.hans
-- 
#  http://www.LuftHans.com/        http://www.LuftHans.com/Classes/
#  Dissent is patriotic.