SCSI & storage

[Kevin Buettner]

On Mar 22, 10:46am, Trent Shipley wrote:

> I have repeatedly read that there should be no more than 1 IDE device on a
> channel or 3 SCSI devices if you want sustained throughput.  However, all
> these sources quote a SUN oriented analysis of system performance tuning
> published in the early 1990's.
> 
> Questions:
> 
> 1) Is the 3 SCSI drives per "channel":
>    a) still current.
>    b) applicable to a given SCSI chain
>    c) applicable to a given SCSI controller no matter how many chains it
> says it supports.

The current state of the art SCSI controller can (supposedly) handle
160 MB/sec.  Assuming that the maximum sustained transfer rate to an
individual drive is 30 MB/sec, that would mean that a single chain can
handle about 5 drives.

BUT...

The standard PCI bus has a maximum theoretical bandwidth of 132
MB/sec.  This means that you can only handle 4 drives at maximum
throughput FOR ALL INTERFACES.  (I.e, adding an additional SCSI
controller won't help.)

Apparently, there are enhanced versions of the PCI bus that are 64
bits wide and that run at 66 MHz.  This would give you a theoretical
maximum bandwidth of 528 MB/sec.  I would imagine that this would help
a lot in situations where you expect to be hammering all of your drives
constantly.

> 2) We mirror all our drives, and use RAID 10 on all larger installations.
> If we stick with SCSI-3 should we:
>    a) Only put 2 drives on a chain
>    b) Put four drives on a chain.
>    c) Put three drives on a chain but use a partitioning scheme to balance
> mirroring.  (I sense the technicians who have to build the thing preparing
> to lynch me . . .)
> 
> 3) How many drives can go on an optical channel before they start to
> interfere with each other and bottleneck?
> 
> 4)  Fiber solutions are slower per drive.  Without regard to cost, can I get
> more bandwidth out of an array of RAID 10 if they use a fiber optic protocol
> instead of SCSI-3?
> 
> 5) Assuming RAID 10, which costs less per Mbit-sec, SCSI-3 or fiber channel?
>    a)  Assume 7 devices.  RAID 10 uses six drives and the seventh is a
> spare.
>    b)  On a WinTel box, when will you be forced to use a fiber-optical
> solution because you run out of slots for SCSI cards?
>    c)  Is there a point when the SCSI and Fiber Channel cost lines cross?

I don't know anything about Fiber Channel, but it seems to me that a
lot of your above questions are moot given the bandwidth restrictions
of the PCI bus.  (Unless you're using the enhanced PCI bus.)

Here's my opinion.  These days SCSI is overrated.  Both the drives
and the controllers are *expensive*.  It is true that a state of the
art SCSI drive will still outperform a state of the art EIDE drive,
but not by much.  Plus, you can't find large capacity SCSI drives.
So here's what I would do...

Put four inexpensive Ultra ATA 100 drives in a box along with an extra
IDE controller, e.g. a Promise Ultra 100.  (I'm assuming that your
motherboard already gives you two channels.)  Use the savings to buy a
boatload of memory.  An operating system like Linux will cache your
files so that you won't hit disk all that often.  Get a hardware RAID
solution if you think you need it.  OTOH, I've been using software
RAID on Linux and have had no problems with it.

Anyway, the theory here is that any performance that you lose by going
with the (dramatically) cheaper controller and drives will be more than
made up for by the extra memory.

As an example of the performance benefits that more memory can give you,
check this out:

saguaro:kernel$ time find linux-2.4.2 -type f -print | xargs wc > /dev/null

real    1m0.813s
user    0m3.040s
sys     0m2.140s
saguaro:kernel$ time find linux-2.4.2 -type f -print | xargs wc > /dev/null

real    0m9.596s
user    0m2.730s
sys     0m0.830s

The above commands do a word count on every file in the 2.4.2 linux
kernel sources and then discard the output.  The first run (on a
machine with 1GB of memory) took a little over a minute and I could
hear the disk getting hit a lot in the process.  The second run hit
the disk minimally and took under 10 seconds.  (This same machine also
had around 241 processes at the time too.  But most of them weren't
doing anything.)

Here's the same test run on a newly rebooted machine running Wolverine
(Red Hat's beta for the upcoming release).  The machine in question
has a 1.1GHz Athlon and 768MB of memory.

[kev@mesquite /mesquite1]$ uname -a
Linux mesquite 2.4.1-0.1.9 #1 Wed Feb 14 22:15:15 EST 2001 i686 unknown
[kev@mesquite /mesquite1]$ time find linux-2.4.2 -type f -print | xargs wc > /dev/null

real    0m19.849s
user    0m1.430s
sys     0m0.870s
[kev@mesquite /mesquite1]$ time find linux-2.4.2 -type f -print | xargs wc > /dev/null

real    0m1.651s
user    0m1.210s
sys     0m0.440s

And here's the same test on the same hardware, but running FreeBSD 4.2:

mesquite:/mesquite1$ time find linux-2.4.2 -type f -print | xargs wc > /dev/null
 
real    0m21.455s
user    0m1.587s
sys     0m1.296s
mesquite:/mesquite1$ time find linux-2.4.2 -type f -print | xargs wc > /dev/null 

real    0m7.239s
user    0m1.522s
sys     0m0.744s

It looks like Linux does more aggressive caching.

I should point out that in other tests that matter more to me, FreeBSD
sometimes comes out ahead.  (I anyone's interested, I could also run
these tests on the same box using NetBSD 1.5, Solaris 8, and Unixware 7.
It might be interesting to see how the commercial OSes do...)


Kevin