A computer components & hardware forum. HardwareBanter

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

Go Back   Home » HardwareBanter forum » General Hardware & Peripherals » Storage & Hardrives
Site Map Home Register Authors List Search Today's Posts Mark Forums Read Web Partners

IOPS benchmarks - looking for opinions



 
 
Thread Tools Display Modes
  #1  
Old March 28th 08, 01:17 AM posted to comp.arch.storage
fx [François-Xavier Peretmere]
external usenet poster
 
Posts: 4
Default IOPS benchmarks - looking for opinions

Hi everyone,

I recently read about the heated contest between two Storage vendors
regarding some IOPS tests one did with the other's gear. The latter's claim
is that the IOPS benchmark is biased and irrelevant, and therefore they
don't run it with their own products. Not being neither a HPC or a high IO
expert, I can't really decide whether or how relevant the claims are -
whilst I found some arguments compelling.

Being curious, I'll be interested in reading what some of the knowledgeable
minds haunting this group would have to say regarding the IOPS benchmark
validity and how relevant they are to the "real world" - whether or not they
are a good proxy to assess how a given piece of hardware is able to sustain
an heavy IO load.

I don't want to discuss the validation process used in by one of the vendor,
I have my own opinion on this matter, and I'd rather prefer to avoid
replicating the heated exchanges I've read in the respective blogs of those
two vendors :-)

Thanks in advance.

Fx

--
"Ah, the beauty of OSS. Hundreds of volunteers worldwide volunteering
their time inventing and implementing new, exciting ways for software
to suck." -- Toni L. in alt.sysadmin.recovery
  #2  
Old March 28th 08, 02:14 AM posted to comp.arch.storage
Lon
external usenet poster
 
Posts: 99
Default IOPS benchmarks - looking for opinions

fx [François-Xavier Peretmere] wrote:
Hi everyone,

I recently read about the heated contest between two Storage vendors
regarding some IOPS tests one did with the other's gear. The latter's
claim is that the IOPS benchmark is biased and irrelevant, and therefore
they don't run it with their own products. Not being neither a HPC or a
high IO expert, I can't really decide whether or how relevant the claims
are - whilst I found some arguments compelling.

Being curious, I'll be interested in reading what some of the
knowledgeable minds haunting this group would have to say regarding the
IOPS benchmark validity and how relevant they are to the "real world" -
whether or not they are a good proxy to assess how a given piece of
hardware is able to sustain an heavy IO load.

I don't want to discuss the validation process used in by one of the
vendor, I have my own opinion on this matter, and I'd rather prefer to
avoid replicating the heated exchanges I've read in the respective blogs
of those two vendors :-)


IOPs are much like having a speedometer that goes to 200 miles per hour,
but an engine that may only stay put together at speeds up to 65 miles
per hour.

Put a set of file systems or luns on half of the storage pool on a set
of arrays. Then benchmark. Then put two file systems or luns on the
same set of arrays such that you fill the arrays. Crank up benchmarks
on both. Some arrays will do very well, some won't. What have you
demonstrated as far as your actual application performance? Very little
if nothing at all.

There are better tools, most are application specific and some of those
can be coaxed into producing results predictive of performance on that
application.
  #3  
Old March 28th 08, 03:20 PM posted to comp.arch.storage
Lon
external usenet poster
 
Posts: 99
Default IOPS benchmarks - looking for opinions

moojit wrote:
""fx [François-Xavier Peretmere]"" wrote in message
...
Hi everyone,

I recently read about the heated contest between two Storage vendors
regarding some IOPS tests one did with the other's gear. The latter's
claim is that the IOPS benchmark is biased and irrelevant, and therefore
they don't run it with their own products. Not being neither a HPC or a
high IO expert, I can't really decide whether or how relevant the claims
are - whilst I found some arguments compelling.

Being curious, I'll be interested in reading what some of the
knowledgeable minds haunting this group would have to say regarding the
IOPS benchmark validity and how relevant they are to the "real world" -
whether or not they are a good proxy to assess how a given piece of
hardware is able to sustain an heavy IO load.


IOPS are a valid test. All IOPS calcuations (atleast the ones advertised by
a vendor) are performed using 512B transfer sizes which is
the typical sector size used by most block SCSI devices. It gives you an
indication of how well the vendors HBA driver and firmware perform.
IOPS are also dependent on the setup because this value is not dependent on
just the HBA driver and firmware, but is dependent on the system itself.


Although 512 is the scsi size, it isnt the size used for IOPS
calculations. Nor frankly is it even relevant for arrays that use a
much larger transfer size and still have the same IOP. Some arrays may
have a lower IOP count for 512 byte transfers, or not.

Nor is 512 byte transfers overly useful for predicting application
performance with apps that regularly use multi-megabyte transfers. e.g.
databases.

  #4  
Old March 28th 08, 06:56 PM posted to comp.arch.storage
Cydrome Leader
external usenet poster
 
Posts: 113
Default IOPS benchmarks - looking for opinions

Lon wrote:
moojit wrote:
""fx [Fran?ois-Xavier Peretmere]"" wrote in message
...
Hi everyone,

I recently read about the heated contest between two Storage vendors
regarding some IOPS tests one did with the other's gear. The latter's
claim is that the IOPS benchmark is biased and irrelevant, and therefore
they don't run it with their own products. Not being neither a HPC or a
high IO expert, I can't really decide whether or how relevant the claims
are - whilst I found some arguments compelling.

Being curious, I'll be interested in reading what some of the
knowledgeable minds haunting this group would have to say regarding the
IOPS benchmark validity and how relevant they are to the "real world" -
whether or not they are a good proxy to assess how a given piece of
hardware is able to sustain an heavy IO load.


IOPS are a valid test. All IOPS calcuations (atleast the ones advertised by
a vendor) are performed using 512B transfer sizes which is
the typical sector size used by most block SCSI devices. It gives you an
indication of how well the vendors HBA driver and firmware perform.
IOPS are also dependent on the setup because this value is not dependent on
just the HBA driver and firmware, but is dependent on the system itself.


Although 512 is the scsi size, it isnt the size used for IOPS
calculations. Nor frankly is it even relevant for arrays that use a
much larger transfer size and still have the same IOP. Some arrays may
have a lower IOP count for 512 byte transfers, or not.


small tranfers are pretty taxing and a good way to see how a system
performs. Reading and writing large files and getting decent speeds isn't
hard.

Nor is 512 byte transfers overly useful for predicting application
performance with apps that regularly use multi-megabyte transfers. e.g.
databases.


Not everything is a database with tranfer to and from the storage unit
being nearly ideal in size.

weak performance for small tranfers can mean horrible backup and restore
times as well.

even untarring a bunch of small files is tedious on a crappy array.

  #5  
Old March 28th 08, 06:58 PM posted to comp.arch.storage
nik Simpson
external usenet poster
 
Posts: 20
Default IOPS benchmarks - looking for opinions

fx [François-Xavier Peretmere] wrote:
Hi everyone,

I recently read about the heated contest between two Storage vendors
regarding some IOPS tests one did with the other's gear. The latter's
claim is that the IOPS benchmark is biased and irrelevant, and therefore
they don't run it with their own products. Not being neither a HPC or a
high IO expert, I can't really decide whether or how relevant the claims
are - whilst I found some arguments compelling.

Being curious, I'll be interested in reading what some of the
knowledgeable minds haunting this group would have to say regarding the
IOPS benchmark validity and how relevant they are to the "real world" -
whether or not they are a good proxy to assess how a given piece of
hardware is able to sustain an heavy IO load.

I don't want to discuss the validation process used in by one of the
vendor, I have my own opinion on this matter, and I'd rather prefer to
avoid replicating the heated exchanges I've read in the respective blogs
of those two vendors :-)


For the record, I assume you are talking about the Netapp/EMC SPC-1
results that Netapp "kindly" published on EMC's behalf. This is much
more complex than simply running 512KB IoMeter type loads.

--
Nik Simpson
  #6  
Old March 29th 08, 12:48 AM posted to comp.arch.storage
Faeandar
external usenet poster
 
Posts: 191
Default IOPS benchmarks - looking for opinions

On Fri, 28 Mar 2008 12:58:16 -0500, nik Simpson
wrote:

fx [François-Xavier Peretmere] wrote:
Hi everyone,

I recently read about the heated contest between two Storage vendors
regarding some IOPS tests one did with the other's gear. The latter's
claim is that the IOPS benchmark is biased and irrelevant, and therefore
they don't run it with their own products. Not being neither a HPC or a
high IO expert, I can't really decide whether or how relevant the claims
are - whilst I found some arguments compelling.

Being curious, I'll be interested in reading what some of the
knowledgeable minds haunting this group would have to say regarding the
IOPS benchmark validity and how relevant they are to the "real world" -
whether or not they are a good proxy to assess how a given piece of
hardware is able to sustain an heavy IO load.

I don't want to discuss the validation process used in by one of the
vendor, I have my own opinion on this matter, and I'd rather prefer to
avoid replicating the heated exchanges I've read in the respective blogs
of those two vendors :-)


For the record, I assume you are talking about the Netapp/EMC SPC-1
results that Netapp "kindly" published on EMC's behalf. This is much
more complex than simply running 512KB IoMeter type loads.


I thought Dave Hitz explained the situation very well and had good
points. But I hate EMC so my bias is unfettered.

But, to the point. Synthetic tests are just that, synthetic. They
can give you great insight into worst case or specific case IO
scenario's. Some of those may or may not apply to your workload.

Real World is so diverse that there's no Real World validation other
than each users own application.

Oracle data warehousing v. OLTP v. EDA grid apps v. web server v.
.......

You get the picture. The truth of it all is that you HAVE to know
what your target is, or at least what your current IO is. You can't
just say "build me a house" and expect to be happy with the outcome.
There are details involved.

~F
 




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
IOPS from RAID units davidhoffer Storage & Hardrives 11 March 24th 08 08:38 PM
IOPS calculation for SAN Design [email protected] Storage & Hardrives 4 October 21st 07 02:41 PM
IOPS from RAID units [email protected] Storage & Hardrives 8 July 21st 07 04:20 PM
CPU Benchmarks confused General 1 September 30th 06 05:58 PM
nfs ops to iops formula? Faeandar Storage & Hardrives 7 February 23rd 06 05:18 AM


All times are GMT +1. The time now is 12:30 PM.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 HardwareBanter.
The comments are property of their posters.