cause of IO wait

[James Mcphee]

If it's consistently consuming a lot of CPU, doing a "ps auxwww" and
checking for blocked state should do it.

On Mon, Jun 27, 2011 at 5:42 PM, Mike Ballon <mike.ballon at gmail.com> wrote:

> you've mentioned iostat and vmstat so lets skip those.
>
> I would start sar and saving running process at the same interval, normal
> 5mins.
>
> I would also take a look at lsof, tracing the pids
>
> Then there is iotop if you have it.
>
> -Mike
>
> On Monday, June 27, 2011, der.hans <PLUGd at lufthans.com> wrote:
> > moin moin,
> >
> > I've got a machine experiencing a lot of IO wait.
> >
> > We had power at a datacenter go down last week. Since then IO wait has
> > been over 35%. At first we thought it was due to 3ware RAID verify taking
> > place due to the crash. That took a few days, then the weekly verify
> > started. We stopped that and IO wait stayed high. 8 disks in a RAID 10.
> >
> > Load avg is also very high, presumably due to the IO wait.
> >
> > smartctl short tests didn't turn up any issues.
> >
> > We're not swapping at all.
> >
> > Disk read and write are fairly low.
> >
> > Network traffic is down as is the total number of process and the number
> > of running processes. No evidence of network errors on the box or at the
> > switch.
> >
> > Not much going on in the logs. We've stopped several reporting processes
> > in order to reduce disk access.
> >
> > On the positive side, entropy has been staying high :).
> >
> > IO wait is not explicitly disk? It could be network, serial, USB, etc.?
> >
> > How do I determine what resource is causing the IO wait? Is there a way
> to
> > track to a specific process?
> >
> > vmstat, iostat, top and lots of other tools have been great at showing
> > that there's overall IO wait ( I've been able to show that almost all
> > processors have high wait, one was only at 5% ), but I haven't yet
> > determined what and how.
> >
> > The server is running CentOS in case that matters.
> >
> > ciao,
> >
> > der.hans
> > --
> > #  http://www.LuftHans.com/        http://www.LuftHans.com/Classes/
> > #  Hope has two beautiful daughters: Anger and Courage. Anger at the way
> > #  things are, and Courage to struggle to create things as they should
> be.
> > #  -- St. Augustine
> > ---------------------------------------------------
> > PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
> > To subscribe, unsubscribe, or to change your mail settings:
> > http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
> >
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.plug.phoenix.az.us
> To subscribe, unsubscribe, or to change your mail settings:
> http://lists.PLUG.phoenix.az.us/mailman/listinfo/plug-discuss
>



-- 
James McPhee
jmcphe at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.PLUG.phoenix.az.us/pipermail/plug-discuss/attachments/20110627/7e39afb1/attachment.html>