building on your own a large data storage ...

Hi,
~
I need to store a really large number of texts and I (could) have a
number of ATA100 S.M.A.R.T.-compliant hard drives, which I would like
to use to somehow build a large and safe (RAID-5?) data store
~
Now I am definitely more of a software person (at least
occupationally) and this is what I have in mind:
~
* I will have to use standard (and commercially available (meaning
cheap ;-))) x86-based hardware and open source software
~
* AFAIK you could maximally use 4 hard drives in such boxes
~
* heat dissipation could become a problem with so many hard drives
~
* I need a reliable and stable power supply
~
Should I got for ATA or SATA drives and why?
~
You could use firewire and/or USB cards to plug in that many
harddrives. Wouldn't it be faster/better using extra ATA PCI cards?
What else would it entail? How many such cards could Linux take?
~
People in the know use software based RAID. Could you give me links
to these kinds of discussions?
~
What would be my weak/hotspot points in my kind of design?
~
Any suggestions of the type of boxes/racks I should use?
~
Is this more or less feasible? What am I missing here? Any other
suggestions? or intelligent posts in which people have discussed these
issues before? I found two in which some people have said a few right
and some other questionable things:
~
comp.sys.ibm.pc.hardware.storage: "2 TB storage solution"
comp.arch.storage: "Homebuilt server (NAS/SAN) vs the prefab ones?
Peformance"
~
Do you know of any such "do-it-yourself" projects out there?
~
thanks
lbrtchx

wrote in message
ups.com...
Hi,
~
I need to store a really large number of texts and I (could) have a
number of ATA100 S.M.A.R.T.-compliant hard drives, which I would like
to use to somehow build a large and safe (RAID-5?) data store
~
Now I am definitely more of a software person (at least
occupationally) and this is what I have in mind:
~
* I will have to use standard (and commercially available (meaning
cheap ;-))) x86-based hardware and open source software
~
* AFAIK you could maximally use 4 hard drives in such boxes
~
* heat dissipation could become a problem with so many hard drives
~
* I need a reliable and stable power supply
~
Should I got for ATA or SATA drives and why?
~
You could use firewire and/or USB cards to plug in that many
harddrives. Wouldn't it be faster/better using extra ATA PCI cards?
What else would it entail? How many such cards could Linux take?
~
People in the know use software based RAID. Could you give me links
to these kinds of discussions?
~
What would be my weak/hotspot points in my kind of design?
~
Any suggestions of the type of boxes/racks I should use?
~
Is this more or less feasible? What am I missing here? Any other
suggestions? or intelligent posts in which people have discussed these
issues before? I found two in which some people have said a few right
and some other questionable things:
~
comp.sys.ibm.pc.hardware.storage: "2 TB storage solution"
comp.arch.storage: "Homebuilt server (NAS/SAN) vs the prefab ones?
Peformance"
~
Do you know of any such "do-it-yourself" projects out there?
~

Have a look at OpenFiler. It's a linux based iSCSI, NFS and SMB appliance.
Takes about 10 minutes to install and configure. I'm using it as iSCSI
shared storage (pseudo-SAN) between a couple of ESX Servers and it works
fine.

--
Kwyj.

["Followup-To:" header set to comp.sys.ibm.pc.hardware.storage.]
On 2007-07-04, wrote:

I need to store a really large number of texts and I (could) have a
number of ATA100 S.M.A.R.T.-compliant hard drives, which I would like
to use to somehow build a large and safe (RAID-5?) data store
~
* I will have to use standard (and commercially available (meaning
cheap ;-))) x86-based hardware and open source software
~
* AFAIK you could maximally use 4 hard drives in such boxes

On a motherboard with 2 IDE ports, you cannot make a 4-disk
RAID-5 array because doing I/O on two devices on the same IDE
port gives poor performance.

You could make two RAID-1 arrays each having one disk on the
primary IDE port and one on the secondary IDE port. Performance
will still suck when you do I/O on both arrays at the same time
but when one array is idle, the other will work OK.

This is of course not as good as RAID-5 from a disk space/euro
POV.

Should I got for ATA or SATA drives and why?

SATA is better because 1) it doesn't have the master/slave
issues of IDE, i.e. if you have 4 SATA ports on your
motherboard, you *can* do a 4-disk RAID-5 array and 2)
motherboards with 8 SATA ports are easy to find.

* heat dissipation could become a problem with so many hard drives

I would not want to do it without adequate ventilation.

* I need a reliable and stable power supply

Fortron FSP-400-60GLN works for me. We have had issues with
Antec.

People in the know use software based RAID. Could you give me links
to these kinds of discussions?

The archives of the linux-raid mailing list (the administration
tool is called mdadm).

What would be my weak/hotspot points in my kind of design?

For me, the time was spent on
- understanding mdadm,
- understanding the trade-offs (partioning an array of disks vs.
making an array of partitions, using LVM or not, optimum
granularity) and
- hardware (how to fit 8 or more disks in a PC case with decent
ventilation).

Any suggestions of the type of boxes/racks I should use?

3ware make 3-disks-in-2-5.25"-spaces trays. They are expensive
and the fans they use die after about a year. When a fan goes
bad, the tray helpfully warns you about it by beeping loudly and
constantly. The fans are not the easiest to find (60 mm or some
such). Be prepared to hear a lot of beeping.

I made my own trays. It was a lot of work and they look ugly but
it was cheap and they do the job. The ventilation is superior to
commercial trays (120 mm fan than moves a lot of air quietly and
reliably).

One very important thing about RAID that too many people
overlook : don't make an N-disk array from N disks of the same
make and model bought the same day. Our sysadmin at work did and
both drives on a RAID-1 array failed within days of each
other...

--
André Majorel URL:http://www.teaser.fr/~amajorel/
(Counterfeit: )
"Duty, honor, country" -- Douglas MacArthur
"Travail, famille, patrie" -- Philippe Pétain