How I built a 2.8TB RAID storage array

I need to stay away from this thread for a while, I am starting to feel some
inspiration. It has been some time since I have run Linux, and well, to be
honest I have always had an urge to build a functional linux box for myself.
And raid fascinates me, so, well, I need to stop reading this stuff. I
can't afford a new toy now.

--Dan

"Yeechang Lee" wrote in message
...
Great project by the way.

Thank you. It's still amazes me to see that little '2.6T' label appear
in the 'df -h' output.

"Yeechang Lee" wrote in message
...

CONTROLLER CARDS
Initial: Two Highpoint RocketRAID 454 cards.
Actual: Two 3Ware 7506-4LP cards.
Why: I needed PATA cards to go with my PATA drives, and also wanted to
put the two PCI-X slots on my motherboard to use. I found exactly two
PATA PCI-X controller cards: The 3Ware, and the Acard AEC-6897. Given
that the Acard's Linux driver compatibility looked really, really
iffy, I went with the 3Ware. I briefly considered the 7506-8 model,
which would've saved me about $120, but figured I'd be better off
distributing the bandwidth over two PCI-X slots rather than one.

No, one PCI-X card would be just as good.

You don't mention the ethernet card, which could also be PCI-X.

SOFTWARE
Initial: Linux software RAID 5 and XFS or JFS.
Actual: Linux software RAID 5 and JFS.
Why: Initially I planned on software RAID knowing that the Highpoint
(and the equivalent Promise and Adaptec cards) didn't do true hardware
RAID. Even after switching over to 3Ware (which *does* do true
hardware RAID), everything I saw and read convinced me that software
RAID was still the way to go for performance, long-term compatibility,
and even 400GB extra space (given I'd be building one large RAID 5
array instead of two smaller ones).

Is there a comparison of Linux RAID 5 to top-end RAID cards? I suspect 3Ware is
better.

Eric Gisin wrote:
"Yeechang Lee" wrote in message
...

CONTROLLER CARDS
Initial: Two Highpoint RocketRAID 454 cards.
Actual: Two 3Ware 7506-4LP cards.
Why: I needed PATA cards to go with my PATA drives, and also wanted to
put the two PCI-X slots on my motherboard to use. I found exactly two
PATA PCI-X controller cards: The 3Ware, and the Acard AEC-6897. Given
that the Acard's Linux driver compatibility looked really, really
iffy, I went with the 3Ware. I briefly considered the 7506-8 model,
which would've saved me about $120, but figured I'd be better off
distributing the bandwidth over two PCI-X slots rather than one.


No, one PCI-X card would be just as good.

Not necessarily. PCI (and PCI-X) bandwidth is per bus, not per slot.
So if those two cards are in two slots on one PCI-X bus, that's not
distributing the bandwidth at all. The motherboard may offer multiple
PCI-X busses, in which case the OP may want to ensure the cards are in
slots that correspond to different busses. The built-in NIC on most
motherboards (along with most other built-in devices) are also on one
(or more) of the PCI busses, so consider bandwidth used by those as well
when distributing the load.

"Eric Gisin" wrote in message
"Yeechang Lee" wrote in message ...

CONTROLLER CARDS
Initial: Two Highpoint RocketRAID 454 cards.
Actual: Two 3Ware 7506-4LP cards.
Why: I needed PATA cards to go with my PATA drives, and also wanted to
put the two PCI-X slots on my motherboard to use. I found exactly two
PATA PCI-X controller cards: The 3Ware, and the Acard AEC-6897. Given
that the Acard's Linux driver compatibility looked really, really
iffy, I went with the 3Ware. I briefly considered the 7506-8 model,
which would've saved me about $120, but figured I'd be better off
distributing the bandwidth over two PCI-X slots rather than one.

No, one PCI-X card would be just as good.

Probably, yes.
Depends on what PCI-X (version, clock) and whether the slots are
seperate PCI buses or not.

If seperate buses the highest clock is atainable and they both have the
full PCI-X bandwidth, say 1GB/s (133MHz) or 533 MB/s (66MHz)
If on same bus, the clock is lower to start with and they have to share
that bus PCI-X bandwidth, say a still plenty 400MB/s each (100MHz)
but may become iffy in case of 66MHz clock (266MB/s) or even 50MHz.


You don't mention the ethernet card, which could also be PCI-X.

What if?


SOFTWARE
Initial: Linux software RAID 5 and XFS or JFS.
Actual: Linux software RAID 5 and JFS.
Why: Initially I planned on software RAID knowing that the Highpoint
(and the equivalent Promise and Adaptec cards) didn't do true hardware
RAID. Even after switching over to 3Ware (which *does* do true
hardware RAID), everything I saw and read convinced me that software
RAID was still the way to go for performance, long-term compatibility,
and even 400GB extra space (given I'd be building one large RAID 5
array instead of two smaller ones).

Is there a comparison of Linux RAID 5 to top-end RAID cards?
I suspect 3Ware is better.

John-Paul Stewart wrote:
No, one PCI-X card would be just as good.

Not necessarily. PCI (and PCI-X) bandwidth is per bus, not per slot.

The Supermicro X5DAL-G motherboard does indeed offer a dedicated bus
to each PCI-X slot, thus my desire to spread out the load with two
cards. Otherwise I'd have gone with the 7506-8 eight-channel card
instead and saved about $120.

The built-in Gigabit Ethernet jack does indeed share one of the PCI-X
slots' buses, but I only have a 100Mbit router right now. I wonder
whether I should expect it to significantly contribute to overall
bandwidth usage on that bus, either now or if/when I upgrade to
Gigabit?

--
Read my Deep Thoughts @ URL:http://www.ylee.org/blog/ PERTH ---- *
Cpu(s): 5.6% us, 5.4% sy, 0.2% ni, 73.9% id, 10.4% wa, 4.6% hi, 0.0% si
Mem: 515800k total, 511808k used, 3992k free, 1148k buffers
Swap: 2101032k total, 240k used, 2100792k free, 345344k cached

"Yeechang Lee" wrote in message
...
The built-in Gigabit Ethernet jack does indeed share one of the PCI-X
slots' buses, but I only have a 100Mbit router right now. I wonder
whether I should expect it to significantly contribute to overall
bandwidth usage on that bus, either now or if/when I upgrade to
Gigabit?


When you DO go gigabit, be sure to at least do some basic throughput
benchmarks (even if its just with a stopwatch, but I suspect you will come
up with a good method) and then compare afterwards. That is really good
data to get firsthand from somebody with such an extreme array and well
documented hardware and software setup. Really good stuff! I wonder what
kind of data rates that array is capable of within the machine too.
Somewhere there is a guy claiming to get 90+MB per second over gigabit
ethernet using raid arrays on both ends.

Gigabit switches are getting so cheap its incredible.

--Dan

Eric Gisin wrote:
Is there a comparison of Linux RAID 5 to top-end RAID cards? I
suspect 3Ware is better.

No, the consensus is that Linux software RAID 5 has the edge on even
3Ware (the consensus hardware RAID leader). See, among others,
URL:http://www.chemistry.wustl.edu/~gelb/castle_raid.html (which
does note that software striping two 3Ware hardware RAID 5 solutions
"might be competitive" with software) and
URL:http://staff.chess.cornell.edu/~schuller/raid.html (which states
that no, all-software still has the edge in such a scenario).

--
Read my Deep Thoughts @ URL:http://www.ylee.org/blog/ PERTH ---- *
Cpu(s): 5.6% us, 5.6% sy, 0.3% ni, 72.2% id, 11.9% wa, 4.5% hi, 0.0% si
Mem: 515800k total, 512004k used, 3796k free, 37608k buffers
Swap: 2101032k total, 240k used, 2100792k free, 293748k cached

In article ,
Yeechang Lee wrote:
Eric Gisin wrote:
Is there a comparison of Linux RAID 5 to top-end RAID cards? I
suspect 3Ware is better.

No, the consensus is that Linux software RAID 5 has the edge on even
3Ware (the consensus hardware RAID leader). See, among others,

If all you care about is "rod length check" long-sequential-read or
long-sequential-write performance, that's probably true. If, of
course, you restrict yourself to a single stream...

....of course, in the real world, people actually do short writes and
multi-stream large access every once in a while. Software RAID is
particularly bad at the former because it can't safely gather writes
without NVRAM. Of course, both software implementations *and* typical
cheap PCI RAID card (e.g. 3ware 7/8xxx) implementations are pretty
awful at the latter, too, and for no good reason that I could ever see.

--
Thor Lancelot Simon

"The inconsistency is startling, though admittedly, if consistency is to be
abandoned or transcended, there is no problem." - Noam Chomsky

No, one PCI-X card would be just as good.

Not necessarily. PCI (and PCI-X) bandwidth is per bus, not per slot.

The Supermicro X5DAL-G motherboard does indeed offer a dedicated bus
to each PCI-X slot, thus my desire to spread out the load with two
cards. Otherwise I'd have gone with the 7506-8 eight-channel card
instead and saved about $120.

The built-in Gigabit Ethernet jack does indeed share one of the PCI-X
slots' buses, but I only have a 100Mbit router right now. I wonder
whether I should expect it to significantly contribute to overall
bandwidth usage on that bus, either now or if/when I upgrade to
Gigabit?

The numbers that you posted from Bonnie++ , if I followed them correctly,
showed max throughputs in the 20 MB/second range. That seems awfully slow
for this sort of setup.

As a comparison, I have two machines with software RAID 5 arrays, one a
2x866 P3 system with 5x120-gig drives, the other an A64 system with 8x300
gig drives, and both of them can read and write to/from their RAID 5 array
at 45+ MB/s, even with the controller cards plugged into a single 32/33 PCI
bus.

To answer your question, GigE at full speed is a bit more than 100
MB/sec. The PCI-X busses on that motherboard are both capable of at least
100 MHz operation, which at 64 bits would give you a max *realistic*
throughput of about 500 MB/second, so any performance detriment from using
the gigE would likely be completely insignificant.

I've got another machine with a 3Ware 7000-series card with a bunch of
120-gig drives on it (I haven't looked at the machine in quite a while), and
I was pretty disappointed with the performance from that controller. It
works for the intended usage (point-in-time snapshots), but responsiveness
of the machine under disk I/O is pathetic - even with dual Xeons.

steve

Steve Wolfe wrote:
The numbers that you posted from Bonnie++ , if I followed them correctly,
showed max throughputs in the 20 MB/second range. That seems
awfully slow for this sort of setup.

Agreed. However, those benchmarks were done with no tuning whatsoever
(and, as noted, the three distributed computing projects going full
blast); since then I've done some minor tweaking, notably the noatime
mount option, which has helped. I'd post newer benchmarks but the
array's right now rebuilding itself due to a kernel panic I caused by
trying to use smartctl to talk to the bare drives without invoking the
special 3ware switch.

To answer your question, GigE at full speed is a bit more than
100 MB/sec. The PCI-X busses on that motherboard are both capable
of at least 100 MHz operation, which at 64 bits would give you a max
*realistic* throughput of about 500 MB/second, so any performance
detriment from using the gigE would likely be completely
insignificant.

That was my sense as well; I suspect network saturation-by-disk will
only cease to be an issue when we all hit the 10GigE world.

(Actually, the 7506 cards are 66MHz PCI-X, so they don't take full
advantage of the theoretical bandwidth available on the slots,
anyway.)

I've got another machine with a 3Ware 7000-series card with a bunch of
120-gig drives on it (I haven't looked at the machine in quite a
while), and I was pretty disappointed with the performance from that
controller.

Appreciate the report. Fortunately, as a home user performance (or
given that I'm only recording TV episodes, even data integrity
actually; thus no backup plans for the array, even if backing up 2.8TB
was practical in any way budgetwise) isn't my prime
consideration. Were I after that, I'd probably have gone with the
9000-series controllers and SATA drives, but my wallet's busted enough
with what I already have!

--
Read my Deep Thoughts @ URL:http://www.ylee.org/blog/ PERTH ---- *
Cpu(s): 4.7% us, 3.2% sy, 0.3% ni, 75.7% id, 14.0% wa, 2.0% hi, 0.0% si
Mem: 515800k total, 510704k used, 5096k free, 18540k buffers
Swap: 2101032k total, 240k used, 2100792k free, 305484k cached