Disk system for storing archival data

I am interested in our thoughts and comments on a raid system that
would be used for storing archival data.

Requirements:
- maximum capacity
- minimum dollars (but not cheap, as in poor quality)
- minimal performance need (the data mostly just sits there)
- must provide "phone home" capability to the vendor for hardware
problems

Any thoughts/comments/recommendations are appreciated!

Thanks

Richard Rhodes

"richard rhodes" wrote in message
om...
I am interested in our thoughts and comments on a raid system that
would be used for storing archival data.

Requirements:
- maximum capacity
- minimum dollars (but not cheap, as in poor quality)
- minimal performance need (the data mostly just sits there)
- must provide "phone home" capability to the vendor for hardware
problems

Any thoughts/comments/recommendations are appreciated!

Thanks

Richard Rhodes

The obvious answer is: A tape library. Best price/performance for archival
purposes, especially when calculating the power requirements of spinning and
maintaining multi-TB of RAID disk versus having equal capacity tape
cartridges sit quietly in their slots.

Rob

"Rob Turk" wrote in message
ll.nl...
"richard rhodes" wrote in message
om...
I am interested in our thoughts and comments on a raid system that
would be used for storing archival data.

Requirements:
- maximum capacity
- minimum dollars (but not cheap, as in poor quality)
- minimal performance need (the data mostly just sits there)
- must provide "phone home" capability to the vendor for hardware
problems

Any thoughts/comments/recommendations are appreciated!

Thanks

Richard Rhodes

The obvious answer is: A tape library.

That might be the answer to some question, but certainly not to the one that
was asked.

Best price/performance for archival
purposes, especially when calculating the power requirements of spinning
and
maintaining multi-TB of RAID disk versus having equal capacity tape
cartridges sit quietly in their slots.

Only if it's necessary to keep the disks spinning all the time to provide
sub-second access to archival data - and if that's the case, tape wouldn't
satisfy the requirements at all. If few-second access were acceptable and
accesses were relatively rare, disks could be spun up on demand.

It would be reasonable to question the relative lifetimes of archived data
on disk and tape, however: whether a disk in intermittent use would have an
expected service life in excess of the normal 5 years could be significant
(if not, periodic migrations to new disks would be required, though in a
RAID-1 array this could be accomplished by the normal mechanisms used to
replace failed disks - or even with RAID-5, though with higher overhead).

This does sound possibly like an ideal application for inexpensive ATA
disks, if they're not running 24/7. That might allow sufficient redundancy
to be incorporated (3-way mirroring, or something fancier like RAID-6
double-parity) that disks could just be run until they failed - though that
would still make me a bit nervous myself, since errors can come thick and
fast near end of life.

- bill

"Bill Todd" wrote in message ...
"Rob Turk" wrote in message
ll.nl...
"richard rhodes" wrote in message
om...
I am interested in our thoughts and comments on a raid system that
would be used for storing archival data.

Requirements:
- maximum capacity
- minimum dollars (but not cheap, as in poor quality)
- minimal performance need (the data mostly just sits there)
- must provide "phone home" capability to the vendor for hardware
problems

Any thoughts/comments/recommendations are appreciated!

Thanks

Richard Rhodes

The obvious answer is: A tape library.

That might be the answer to some question, but certainly not to the one that
was asked.

Best price/performance for archival
purposes, especially when calculating the power requirements of spinning
and
maintaining multi-TB of RAID disk versus having equal capacity tape
cartridges sit quietly in their slots.

Only if it's necessary to keep the disks spinning all the time to provide
sub-second access to archival data - and if that's the case, tape wouldn't
satisfy the requirements at all. If few-second access were acceptable and
accesses were relatively rare, disks could be spun up on demand.

It would be reasonable to question the relative lifetimes of archived data
on disk and tape, however: whether a disk in intermittent use would have an
expected service life in excess of the normal 5 years could be significant
(if not, periodic migrations to new disks would be required, though in a
RAID-1 array this could be accomplished by the normal mechanisms used to
replace failed disks - or even with RAID-5, though with higher overhead).

This does sound possibly like an ideal application for inexpensive ATA
disks, if they're not running 24/7. That might allow sufficient redundancy
to be incorporated (3-way mirroring, or something fancier like RAID-6
double-parity) that disks could just be run until they failed - though that
would still make me a bit nervous myself, since errors can come thick and
fast near end of life.

- bill

Richard,

We've seen companies build/buy RAIDs/SANs to store their archive data;
I hope it is not a path you want to take. Here are the problems that
occur when you store archive data on-line:

1) You still won't have an archive methodology. Your just band-aiding
the problem for now.
2) Backing up large volumes of data is not easy or inexpensive. I
have one customer that refuses to archive and spends $20,000 to
$22,000 a year in time and money just maintaining his 2.5 TB of data.
Half the time, he forgets to swap tapes.
3) If you add a 5 TB system today, how long before you add another 5
TB system? And another? And another? Where does it end?
4) When you have a hard drive failure, RAIDed or not, you still have
to ensure your data is not corrupt nor needs reloading from tape.
5) When the building burns down, who will be tasked with carrying the
hard drive under their arms as they run out of the building?
6) On-line data is never retired and is still a moving target. You
will have to spend time reindexing it if you want to know what you
have.

Read Kevin Roden's "Seven Deadly Sins of Data mismanagement" at
http://www.cio.com/sponsors/041503im/?page=2. Although he doesn't
address your specific issue, I think you'll get something out of his
message.

Please consider archiving your data to DVD. It does require more
thought and preparation. DVDs are cheap, have a ridiculous life span,
and will be readable in drives for decades to come. Tape and RAID
systems will become antiquated long before DVD.

Our target customer has never been energy companies so your probably
not a good fit for our services, but Kevin Roden group may have a
solution for you.

Rob Stokes
ASSURED SOLUTIONS
Emergency Help Line: 214-747-9911
Information/Report Line: 214-747-4411
http://www.idigmedia.com/id1457
http://www.remotearchiving.com/ra1457

"Helping our clients increase productivity through measured
performance."

In article ,
Rob wrote:
Please consider archiving your data to DVD. It does require more
thought and preparation. DVDs are cheap, have a ridiculous life span,
and will be readable in drives for decades to come. Tape and RAID
systems will become antiquated long before DVD.

Are you referring to DVD-R or DVD+R? The last time I checked, the life
span of CD-R was still a bit of a question mark. DVD-R and DVD+R use
similar chemistry, so I don't share your confidence. There is the
additional problem of finding a high quality vendor for the blank media.
The big companies that manufacture the media all produce various grades,
and they don't give any guidance as to which disks are reliable and
which are junk. And they all seem to produce some junk, to meet the
demand of cheap home users.
--
David Arnstein Please do not look at laser with remaining eye

(Rob) wrote in message . com...
"Bill Todd" wrote in message
Richard,

We've seen companies build/buy RAIDs/SANs to store their archive data;
I hope it is not a path you want to take. Here are the problems that
occur when you store archive data on-line:

1) You still won't have an archive methodology. Your just band-aiding
the problem for now.
2) Backing up large volumes of data is not easy or inexpensive. I
have one customer that refuses to archive and spends $20,000 to
$22,000 a year in time and money just maintaining his 2.5 TB of data.
Half the time, he forgets to swap tapes.
3) If you add a 5 TB system today, how long before you add another 5
TB system? And another? And another? Where does it end?
4) When you have a hard drive failure, RAIDed or not, you still have
to ensure your data is not corrupt nor needs reloading from tape.
5) When the building burns down, who will be tasked with carrying the
hard drive under their arms as they run out of the building?
6) On-line data is never retired and is still a moving target. You
will have to spend time reindexing it if you want to know what you
have.


Rob Stokes
ASSURED SOLUTIONS
Emergency Help Line: 214-747-9911
Information/Report Line: 214-747-4411
http://www.idigmedia.com/id1457
http://www.remotearchiving.com/ra1457

"Helping our clients increase productivity through measured
performance."

Thanks for the comments everyone.

A little more explanation of what we might be trying to do . . .

We have 4 sap instances that reside on emc symmetrix storage. The
growth of these systems will exceed the capacity of these storage
systems unless we implement sap archiving. If we do this, we believe
we can keep within our current frames.

We currently perform sap archiving for one of these 4 sap instances.
I don't know the software package that interfaces to sap, but the
backend product that actually handles the data is ibm's TSM backup
product. Currently, TSM is archiving the data to DLT tape. Within
TSM we handle the issues of duplicate (backup) copies of all data via
a copy storage pool.

One option is to just keep archiving to tape. We would definetely
replace the DLT tape system since it's old, and possibly with another
tape technology.

We've discussed optical jukeboxes (anything that TSM supports), but
our experience in the past with these systems hasn't been very good.

Another options we discussed is to just dump the data into a very
large TSM disk pool and just leave it there - not migrating it to
tape/optical. We would backup the data to a tape copy pool no a
offsite TSM instance (which we would do for any solution -
tape/optical/disk). Over several years the data would grow to many
TB, but the class of disk storage need does not have to be EMC
Symmetrix class. Just about any kind of disk subsystem would provide
more than enough performance and allow many concurrent retrevals.

I know Netapp has a disk system designed for archival data, and I
believe STK has something also. EMC has their Centerra, but it
requires special driving software since it's not a "disk subsystem"
per say.

The driving goals are to keep the sap databases growth down.

I was just wondering what others thought and/or have done.

Thanks

Rick

In article , (richard rhodes) writes:


Another options we discussed is to just dump the data into a very
large TSM disk pool and just leave it there - not migrating it to
tape/optical. We would backup the data to a tape copy pool no a
offsite TSM instance (which we would do for any solution -
tape/optical/disk). Over several years the data would grow to many
TB, but the class of disk storage need does not have to be EMC
Symmetrix class. Just about any kind of disk subsystem would provide
more than enough performance and allow many concurrent retrevals.

I know Netapp has a disk system designed for archival data, and I
believe STK has something also. EMC has their Centerra, but it
requires special driving software since it's not a "disk subsystem"
per say.

The driving goals are to keep the sap databases growth down.

I was just wondering what others thought and/or have done.


What are your timeframes? IBM's SATA solution (SATA drives in
FastT) is a good one. Cost comes down next year with larger drives
(RAID5 with 10-14 members, fewer drives needed, fewer shelves).

What you are proposing (large TSM disk pool and of course off-site
tape copies for DR) is exactly what we are looking at. How
much are you budgeting ;-)?

Rob

In article ,
(Rob) writes:
| "Bill Todd" wrote in message ...
| "Rob Turk" wrote in message
| ll.nl...
| "richard rhodes" wrote in message
| om...
| I am interested in our thoughts and comments on a raid system that
| would be used for storing archival data.
|
| Requirements:
| - maximum capacity
| - minimum dollars (but not cheap, as in poor quality)
| - minimal performance need (the data mostly just sits there)
| - must provide "phone home" capability to the vendor for hardware
| problems
....
| Only if it's necessary to keep the disks spinning all the time to provide
| sub-second access to archival data - and if that's the case, tape wouldn't
| satisfy the requirements at all. If few-second access were acceptable and
| accesses were relatively rare, disks could be spun up on demand.
|
| It would be reasonable to question the relative lifetimes of archived data
| on disk and tape, however: whether a disk in intermittent use would have an
| expected service life in excess of the normal 5 years could be significant
| (if not, periodic migrations to new disks would be required, though in a
| RAID-1 array this could be accomplished by the normal mechanisms used to
| replace failed disks - or even with RAID-5, though with higher overhead).
....

| Please consider archiving your data to DVD. It does require more
| thought and preparation. DVDs are cheap, have a ridiculous life span,
| and will be readable in drives for decades to come. Tape and RAID
| systems will become antiquated long before DVD.

The job of archiving data is NOT selecting a long-lived media! Who
cares if your DVDs are in excellent shape if they contain a file format
that is obsolete and unsupportable in two years? This is a far more
significant risk that media degradation. What would you do if PDF
format (for example) gave way to something newer in the marketplace?
Sure there would be backward compatibility for a while, but sooner or
later you'd have to do a migration to a new format and it would be far
sooner than the media lifetime. So it's wasteful to have 100 year media
that you have to replace every 5-10 years.

Data archiving IS a management responsibility to insure continuing access
to data in the face of obsolescences from format, software, operating system,
media, and to take action whenever needed. This need for vigilance does
not end until the data reaches the end of its lifecycle.

So select a media that will last one technological generation and expect
to replace it periodically. There is an advantage to choosing rewritable
media: software/format dependencies are usually on a shorter lifecycle
than media so rewritability lets you update the file format without
replacing the media.

-Keith

In article ,
Keith Michaels wrote:
The job of archiving data is NOT selecting a long-lived media! Who
cares if your DVDs are in excellent shape if they contain a file format
that is obsolete and unsupportable in two years? This is a far more
significant risk that media degradation.

No, it's not. The problem of obsolescent file formats gets a lot of
press, but it's not a serious problem. This is because file formats
won't go obsolete overnight. You're not going to get an e-mail announcing
that in 5 minutes, you won't be able to read any of your DVD-ROMs. It
will be more like, slowly, over a period of years, the drives and media
will become more scarce. Microsoft will announce that operating system
support is ending for the last of the drives, well in advance. Linux
will support the drives forever, most probably.

When this happens, you will have monthes, perhaps years, to plan and
execute your big copy job: DVD-ROM to whatever media you adopt for your
future needs. Yes it will be an expense, but that's life. You will
have plenty of time to make your copies gradually, over a period of
monthes if you prefer.

Executive summary: if you find yourself holding valuable data whose
only home is an unreadable medium or file format, you're stupid.

On the other hand, media degradation is a big problem, currently. We
still don't know how long CD-R media will last, it's just too new.
DVD-R and DVD+R use similar chemistry. Also, there are no quality
standards in the DVD-R/DVD+R industry. Each manufacturer makes a broad
range of media, from truly awful to great. Telling them apart by reading
the product labels doesn't work reliably. Happy shopping.
--
David Arnstein Please do not look at laser with remaining eye

(richard rhodes) wrote in message . com...

We have 4 sap instances that reside on emc symmetrix storage. The
growth of these systems will exceed the capacity of these storage
systems unless we implement sap archiving. If we do this, we believe
we can keep within our current frames.

We currently perform sap archiving for one of these 4 sap instances.
I don't know the software package that interfaces to sap, but the
backend product that actually handles the data is ibm's TSM backup
product. Currently, TSM is archiving the data to DLT tape. Within
TSM we handle the issues of duplicate (backup) copies of all data via
a copy storage pool.


We are a large international company that also uses SAP. We do a lot
with SAP archiving. We uses IXOS (www.ixos.com) to archive our SAP
data to an HP optical jukebox. We put the data on WORM (write once
read many) platters due to legal requirements. So depending on the
type of data you are archiving out of SAP you might also have a number
of legal requirements to consider...

hth,
john