If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. |
|
|
|
Thread Tools | Display Modes |
#1
|
|||
|
|||
Very High Rate Continous Transfer
I am looking into what it will take to support continuous (not burst)
160 to 200 MBytes/sec transfer to disk. What types of drives and how would they be configured as an array (or multiple arrays)? What type of processor and bus architecture would be appropriate? Data will be from a capture memory that is shared between the processor and the capture electronics. So memory bandwidth will be at least 320 to 400 MBytes/sec. Thanks in advance, Jim |
#2
|
|||
|
|||
|
#3
|
|||
|
|||
In article ,
Bill Todd wrote: wrote: I am looking into what it will take to support continuous (not burst) 160 to 200 MBytes/sec transfer to disk. What types of drives and how would they be configured as an array (or multiple arrays)? What type of processor and bus architecture would be appropriate? Data will be from a capture memory that is shared between the processor and the capture electronics. So memory bandwidth will be at least 320 to 400 MBytes/sec. Modern x86 processor/memory configurations should be able to handle that kind of memory bandwidth without even starting to break a sweat - if the capture electronics can use standard DMA mechanisms to provide the data. Agree. Although you have to be careful with memory - CPU bandwidth. If you need to do multiple passes over the data (for example, copy if between buffers, or run CRCs or TCP checksums over them) the memory bandwidth could become an issue. But in that case you'd need a double-speed, double-width (64/66) PCI bus at a minimum to handle the bi-directional bandwidth unless you used a chipset that bypassed the PCI for disk activity (I'm not even sure they're available: if they only support IDE, you'd have to split the load across two IDE ports - and where are you going to find an IDE RAID box?) or worked around the problem by using the AGP port for the input or output stream, in which case you'd still need double-width or double-speed PCI but probably not both. You can get motherboards with multiple PCI channels. Somewhere in my lab, I have a few 2-U rackmounts with 3 open PCI busses (on 3 slots); if I remember right the Ethernet and SCSI/RAID chips on the motherboard are on a separate bus. Don't know whether such motherboards are sold in the white-box market; you may have to buy a server-class x86 box from one of the big vendors to get a system like this. Also, AFAIK the AGP bus is a superset of the PCI bus, just on a different connector. If you are custom-building your electronics, you might want to use the AGP connector. In other words, PCI-X or PCI-express might be a better option (and today probably much more available than double-speed, double-width PCI). One other issue: For the outgoing link, I would split the bandwidth over two separate fibre channel ports. With 2Gbit FC, each port can theoretically handle 200 MB/sec, so each would be loaded 50%, which will make the whole thing run much more smoothly. You could for either stripe the data yourself (if you control the data writigng software, or for example use a LVM on the host to stripe the data over the two FC ports. Whether to also stripe it over two disk arrays depends on what disk array you buy. As 2-port 2Gbit FC cards are easily available, this does not require extra PCI slots. For an inexpensive solution (no redundancy, do-it-yourself), get a PCI SCSI controller with two U320 ports. Connect a few (maybe a half dozen) 10K RPM SCSI disks to each port. This might be quite easy: Buy a rackmount JBOD (a.k.a. disk tray) with two SCSI ports; make sure you get a model with a splittable SCSI backplane (two half-backplanes, each with about a half dozen slots, instead of one long backplane with two SCSI connevtors and with a dozen SCSI slots). Then use custom software or an LVM to stripe the data across the disks. Let's look at the numbers: 10K RPM SCSI drives can write data at about 50 MB/sec each; the reason I picked a half dozen drives per SCSI port is to match disk bandwidth with SCSI bandwidth. This hardware configuration can theoretically handle about 600 MB/sec, so it should have no problem runnind day-in day-out at 1/3 of that. Problem is: No redundancy, and you have to roll your own striping. RAID-3 sounds like what you want for disk storage (assuming that you want some redundancy in it - though I'm not sure where you'd find a non-redundant implementation comparable in performance to a good RAID-3 box even if redundancy weren't necessary). I saw a single 1 Gbit/s fibre channel link deliver almost 90 MB/s of streaming write bandwidth to an 8 + 1 RAID-3 array back in 1998, so 2 Gbit/s fiber channel with at most 6 + 1 of today's disks - which should offer 30 - 40 MB/s/disk streaming bandwidth at an absolute minimum - should do the job with a single array (if not, perhaps the application can easily distribute the streaming data across two arrays or use software RAID-0 to do so). To some extent I agree. RAID-3 has traditionally been used for large streaming IO (examples: multimedia, supercomputing). On the other hand, the bandwidth required here is so small that any modern mid-range disk array could do it, in any RAID-level. So if disk cost is an issue and you don't need redundancy (meaning you are willing to tolerate loss of data, and downtime), you might want to try RAID-0. If disk costs are irrelevant, and you want the best possible redundancy, you could try RAID-1 (in effect implemented as RAID-10). Or maybe RAID-5 might work better than RAID-3: few people use RAID-3 today, so it is quite possible that RAID-5 implementations have been carefully tested and tuned, and are quite fast. Another warning: If you think RAID will give you redandancy, and at the same time you are relying on maxing out the performance of the disk array, you are cheating yourself. What I mean is this: If you buy a disk array that can barely handle 200 MB/sec (meaning your configuration is cost optimized), then it will not handle that bandwidth in degraded mode (with a dead disk). RAID redundancy is in some sense about preserving your data; while running with a dead disk, the speed will be quite low. If your data is lost for good if you can't write it to disk, you'll have to significantly overdesign your RAID arrays to make sure they can handle the traffic even in degraded mode. This should be less of an issue for RAID-1 (which is easier to write to in degraded mode) than for the parity-based RAIDs. Happy experimenting! -- The address in the header is invalid for obvious reasons. Please reconstruct the address from the information below (look for _). Ralph Becker-Szendy |
#4
|
|||
|
|||
|
#5
|
|||
|
|||
In article ,
Bill Todd wrote: wrote: ... Another warning: If you think RAID will give you redandancy, and at the same time you are relying on maxing out the performance of the disk array, you are cheating yourself. What I mean is this: If you buy a disk array that can barely handle 200 MB/sec (meaning your configuration is cost optimized), then it will not handle that bandwidth in degraded mode (with a dead disk). RAID redundancy is in some sense about preserving your data; while running with a dead disk, the speed will be quite low. If your data is lost for good if you can't write it to disk, you'll have to significantly overdesign your RAID arrays to make sure they can handle the traffic even in degraded mode. While I generally agree with the other points you made, my distinct impression is that good RAID-3 implementations (unlike, say, RAID-4, -5, or -6) suffer no noticeable performance degradation (for either reads or writes) while running with a dead disk. That was one of the reasons I suggested it. I agree, and correction gladly accepted. With one minor fly in the ointment: While the disk is dead, things should run just fine. As soon as you put a spare disk in, the array might try to rebuild onto the new disk; that rebuild will compete with the foreground workload. With good systems management, this could be circumvented, for example: once a LUN has a dead disk, slowly drain the data from it, then get the disk array to accept a spare disk without rebuilding, for example by destroying the LUN, and recreating it using the spare (doing all this on a live system without having to shut the software stack down might be tricky). Compared to all the other questions the original poster should think about (SCSI or FC? Commercial disk arrays or JBODs? PCI/AGP and memory bandwidth? Redundancy or not? Stripe by hand, by LVM, or not at all? and many others) the selection of RAID level is actually a minor point. -- The address in the header is invalid for obvious reasons. Please reconstruct the address from the information below (look for _). Ralph Becker-Szendy |
#6
|
|||
|
|||
Also, AFAIK the AGP bus is a superset of the PCI bus, just on a
different connector. If you are custom-building your electronics, you might want to use the AGP connector. AGP is obsolete, being replaced by PCI-X in all newer mobos, even for home gaming platforms. AGP is rather hardly 3D-oriented, while PCI-X is universal. -- Maxim Shatskih, Windows DDK MVP StorageCraft Corporation http://www.storagecraft.com |
#7
|
|||
|
|||
Maxim S. Shatskih wrote:
AGP is obsolete, being replaced by PCI-X in all newer mobos, even for home gaming platforms. PCI-express, that is. PCI-X is 100/133 MHz PCI in servers, PCI express is a serial version. Thomas |
#8
|
|||
|
|||
I appreciate the advice. I am absorbing it a fast as I can.
I'll be using this as a real-time recording device, and data can be transferred to more reliable long term memory after the recording procedure is completed, which may be several hours later. I am inclined to think that RAID-0 for performance, or RAID-5 for some redundancy, is the way to go. I am also considering 2.5" vs 3.5" drives, and a review at Tom's Hardware suggests an investment in 2.5" will reduce my noise and power by a lot. Also, I expect the equivalent array of 2.5" drives to be cooler, lighter and inherently more rugged, which I need for this application. I will need at least 600 MB of hard drive storage. I may be willing to reduce the hard drive continuous bandwidth by half (at the expense of losing a main feature). All other requirements remain the same. Now I will need to identify RAID controllers and a processor board. I would prefer a 3U form factor. I'll need about 10 GB of fast memory, and will DMA data to it from an input device. I expect this to be pricey, so no need to beat me about that. I am going about this blindly at the moment, but any suggestions will be greatly appreciated. You are all very helpful, and I appreciate it very much. Jim |
#9
|
|||
|
|||
wrote in message oups.com... I am looking into what it will take to support continuous (not burst) 160 to 200 MBytes/sec transfer to disk. What types of drives and how would they be configured as an array (or multiple arrays)? What type of processor and bus architecture would be appropriate? Data will be from a capture memory that is shared between the processor and the capture electronics. So memory bandwidth will be at least 320 to 400 MBytes/sec. Might be worth looking for articles and papers from CERN - they've been doing this sort of stuff for years and used to publishe papers on the computing architectures. |
#10
|
|||
|
|||
You need to just go get a CX700... It will handle the I/O you are
talking about even with a bad disk. Just make sure you have a global hot spare for every shelf. As long as the data is not random you will be fine with Raid 5 on that arrary. |
|
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Running a M7NCD Motherboard with 400 FSB | Krutibas Biswal | Homebuilt PC's | 17 | October 4th 04 11:36 AM |
Increasing disk performance with many small files (NTFS/ Windowsroaming profiles) | Benno... | Storage & Hardrives | 18 | July 23rd 04 12:41 PM |
Whats the fasted sustanined transfer rate of a drive ? | We Live For The One We Die For The One | Storage (alternative) | 1 | April 6th 04 09:50 PM |
Slow transfer rate | WzL | Storage (alternative) | 0 | February 25th 04 04:53 PM |
RAID 5 vs. single SCSI drive? | Carlos Moreno | Storage (alternative) | 35 | December 19th 03 06:20 PM |