If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below. |
|
|
Thread Tools | Display Modes |
#1
|
|||
|
|||
High Disk Read Performance
I need to architect 2TB of disk storage with absolute maximum read
capabilities. I have an application that sequentially reads and analyzes large flat data base files and the faster I can read them the faster I can analyze. CPU and memory are not a factor - currently the disk "reads" are my bottle neck. What can I do to push the envelope? i.e. Do I use lots of smaller disks or fewer large disks? What RAID config? Lots of cache or no cache? Multiple SCSI/FC controllers? Etc….. I am open to any ideas (assuming the cost is reasonable) My growth will be from 2TB to 4TB over the next three years. I would like to achieve disk "reads" around 200MB/sec - 300MB/sec (Mega Bytes not bits!) |
#2
|
|||
|
|||
|
#3
|
|||
|
|||
|
#4
|
|||
|
|||
DB wrote:
I need to architect 2TB of disk storage with absolute maximum read capabilities. I have an application that sequentially reads and analyzes large flat data base files and the faster I can read them the faster I can analyze. The solution could depend on your app. What is the kind of data? biological? geographic? pictures? unstructured texts, like web pages? You need sequential scan; are you doing data mining, info. retrieval? CPU and memory are not a factor - currently the disk "reads" are my bottle neck. What can I do to push the envelope? i.e. Do I use lots of smaller disks or fewer large disks? use half the capacity of each disk anyway. Each will deliver up to 50MB/s in sequential read. Filling the disk will cause the throughput to fall quickly when reading the end. What RAID config? Lots of cache or no cache? you don't need much cache for seq. read. Multiple SCSI/FC controllers? Etc….. I am open to any ideas (assuming the cost is reasonable) My growth will be from 2TB to 4TB over the next three years. I would like to achieve disk "reads" around 200MB/sec - 300MB/sec (Mega Bytes not bits!) what about data compression? |
#5
|
|||
|
|||
Stephane Guyetant wrote:
DB wrote: [ ... ] I would like to achieve disk "reads" around 200MB/sec - 300MB/sec (Mega Bytes not bits!) You'll also need more than one Fiber Channel card and some way to distribute the I/O across the channels (e.g. powerpath, dmp, hdlm, ...) What OS? Do you need a file system, or could your source file be a raw partition? Dave |
#6
|
|||
|
|||
"DB" wrote in message om... I need to architect 2TB of disk storage with absolute maximum read capabilities. I have an application that sequentially reads and analyzes large flat data base files and the faster I can read them the faster I can analyze. CPU and memory are not a factor - currently the disk "reads" are my bottle neck. What can I do to push the envelope? i.e. Do I use lots of smaller disks or fewer large disks? What RAID config? Lots of cache or no cache? Multiple SCSI/FC controllers? Etc... I am open to any ideas (assuming the cost is reasonable) My growth will be from 2TB to 4TB over the next three years. I would like to achieve disk "reads" around 200MB/sec - 300MB/sec (Mega Bytes not bits!) You seem to be starting at the wrong end of the problem. The prior issue is the processing and analysis application to deal with a data flow of that level? I'm guessing that a quad CPU or larger box may be required. Can the analysis app apply usefully a number of CPUs to your single task? You'll need a system bus[es] PCI[-X] fast enough to support the desired data rate. After that there are a number of HD array technologies that are capable of delivering high sequential data rates. For the flat file(sequential read) case [S]ATA HDs are by far the most cost effective choice. A German university has recently setup a multi-TB array based on ATA HDs purely as a target of backups of other systems and that's primarily a sequential I/O situation. Then comes the question of redundancy(relaibility) required for your disk array. Since you describe it as multi-TB, then likely some version of RAID 5 seems suitable and probably several such arrays. Another factor in the storage system design is backup and the real time it takes to complete a backup or restore. Backup may be the primary determinate of your storage system design. Is this a standalone single user analysis situation(one pass at a time) or might there be several unrelated analyses happening concurrently? |
#7
|
|||
|
|||
In article , DB wrote:
I need to architect 2TB of disk storage with absolute maximum read capabilities. I have an application that sequentially reads and analyzes large flat data base files and the faster I can read them the faster I can analyze. CPU and memory are not a factor - currently the disk "reads" are my bottle neck. What can I do to push the envelope? A lower cost option is lots of IDE disks attached to multiple 3ware controllers. I've got two boxes set up using dual Xeons on Supermicro MBs, 2 3ware 7500-8 boards (on separate PCI busses), and 16 180GB WD 7200 RPM drives. The 3wares are setup for hardware RAID5 w/ a hot spare, and then I do a RAID0 stripe across the two arrays in software (Linux is the OS). Here are some benchmarks from bonnie++: [jlb@buckbeak tmp]$ bonnie++ -s 8192 Version 1.02c ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP buckbeak 8G 20485 76 55244 17 27481 10 27383 97 365660 81 446.0 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 2168 19 +++++ +++ 2710 23 1196 11 +++++ +++ 3178 32 buckbeak,8G,20485,76,55244,17,27481,10,27383,97,36 5660,81,446.0,1,16,2168,19,+++ ++,+++,2710,23,1196,11,+++++,+++,3178,32 And tiobench: Unit information ================ File size = megabytes Blk Size = bytes Rate = megabytes per second CPU% = percentage of CPU used during the test Latency = milliseconds Lat% = percent of requests that took longer than X seconds CPU Eff = Rate divided by CPU% - throughput per cpu load Sequential Reads File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency 2s 10s Eff ---------------------------- ------ ----- --- ------ ------ --------- --------- -- -------- -------- ----- 2.4.20-19.7.XFS1.3.0smp 4096 4096 1 276.27 81.68% 0.013 99. 82 0.00000 0.00000 338 2.4.20-19.7.XFS1.3.0smp 4096 4096 2 275.17 102.6% 0.027 106. 56 0.00000 0.00000 268 2.4.20-19.7.XFS1.3.0smp 4096 4096 4 225.11 108.5% 0.067 256. 05 0.00000 0.00000 207 2.4.20-19.7.XFS1.3.0smp 4096 4096 8 221.65 111.8% 0.132 217. 67 0.00000 0.00000 198 Random Reads File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency 2s 10s Eff ---------------------------- ------ ----- --- ------ ------ --------- --------- -- -------- -------- ----- 2.4.20-19.7.XFS1.3.0smp 4096 4096 1 1.13 2.676% 3.454 54. 82 0.00000 0.00000 42 2.4.20-19.7.XFS1.3.0smp 4096 4096 2 1.93 16.33% 4.002 53. 27 0.00000 0.00000 12 2.4.20-19.7.XFS1.3.0smp 4096 4096 4 2.94 20.87% 4.971 78. 67 0.00000 0.00000 14 2.4.20-19.7.XFS1.3.0smp 4096 4096 8 4.61 26.52% 5.989 103. 03 0.00000 0.00000 17 Sequential Writes File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency 2s 10s Eff ---------------------------- ------ ----- --- ------ ------ --------- --------- -- -------- -------- ----- 2.4.20-19.7.XFS1.3.0smp 4096 4096 1 58.92 23.21% 0.054 2981. 43 0.00038 0.00000 254 2.4.20-19.7.XFS1.3.0smp 4096 4096 2 44.03 22.14% 0.150 4731. 74 0.00114 0.00000 199 2.4.20-19.7.XFS1.3.0smp 4096 4096 4 33.53 26.27% 0.382 6946. 62 0.00458 0.00000 128 2.4.20-19.7.XFS1.3.0smp 4096 4096 8 29.41 24.99% 0.889 13357. 19 0.01450 0.00010 118 Random Writes File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency 2s 10s Eff ---------------------------- ------ ----- --- ------ ------ --------- --------- -- -------- -------- ----- 2.4.20-19.7.XFS1.3.0smp 4096 4096 1 1.66 0.531% 0.011 0. 11 0.00000 0.00000 313 2.4.20-19.7.XFS1.3.0smp 4096 4096 2 1.67 1.071% 0.016 0. 13 0.00000 0.00000 156 2.4.20-19.7.XFS1.3.0smp 4096 4096 4 1.63 1.873% 0.029 4. 73 0.00000 0.00000 87 2.4.20-19.7.XFS1.3.0smp 4096 4096 8 1.50 2.117% 0.025 1. 25 0.00000 0.00000 71 -- Joshua Baker-LePain Department of Biomedical Engineering Duke University |
#8
|
|||
|
|||
In article , Joshua Baker-LePain wrote:
In article , DB wrote: I need to architect 2TB of disk storage with absolute maximum read capabilities. I have an application that sequentially reads and analyzes large flat data base files and the faster I can read them the faster I can analyze. CPU and memory are not a factor - currently the disk "reads" are my bottle neck. What can I do to push the envelope? A lower cost option is lots of IDE disks attached to multiple 3ware controllers. I've got two boxes set up using dual Xeons on Supermicro MBs, 2 3ware 7500-8 boards (on separate PCI busses), and 16 180GB WD 7200 RPM drives. The 3wares are setup for hardware RAID5 w/ a hot spare, and then I do a RAID0 stripe across the two arrays in software (Linux is the OS). Here are some benchmarks from bonnie++: Following up to myself (sigh): XFS is the filesystem. -- Joshua Baker-LePain Department of Biomedical Engineering Duke University |
#9
|
|||
|
|||
Thank you all for your comments - they have been very useful.
I am going to be doing the following 1. Purchase a Quad CPU box with three PCI-X buses. I will install an Ultra 320 SCSI PCI-X controller with 256MB Cache in each bus 2. Each controller will connect to a 14 disk SCSI drive shelve with 14 x 73GB 15K Ultra 320 drives configured as RAID5 3. Using some software I will create a single large partition that spans all three RAIDs. I will initially try this using an Windows Extended volume set for three dynamic disks. I will also evaluate Veritas volume manager to see if there is a big difference. (..and the database app may be able to span the partitions itself) 4. I will set the cache to 100% reads on each controller. Is there anything else I can do? Any suggestions on the stripe sizes in the RAIDS at both hardware and OS level? Again - thanks for all your feedback. Daz |
Thread Tools | |
Display Modes | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
A8N-SLI Deluxe: Disk Read Error | PJ | Asus Motherboards | 0 | March 1st 05 08:54 PM |
P4S333 hard drive capacity | Katy | Asus Motherboards | 6 | September 19th 04 04:26 AM |
Some P4C800-E Deluxe Q's and upgrade advice requested please | Paul | Asus Motherboards | 6 | April 13th 04 08:46 PM |
Nero 6 Burn Rom - Help file, cd-rw disks, cd audio/data combination disk | Bun Mui | Cdr | 1 | January 14th 04 10:42 AM |
Nero 6 Burn Rom- Help file, cd-rw disks, cd audio/data combination disk | Bun Mui | Cdr | 1 | January 12th 04 12:46 AM |