Using a hard disk drive as an archival media

William Brown wrote:
On Tue, 10 May 2011 11:33:43 -0400, JW wrote:


After reading about the (not so) longevity of using CD-R and DVD-R as
archival media I decided to get two 2TB Hitachi drives from Newegg keeping
them mirrored in case one dies. As I was moving the data from the optical
media to the fixed disks, about 10 of 350 or so optical discs I had gave
me fits (CRC errors, bad blocks, etc.) I was able to recover 8 of them and
2 were a complete loss. (Note that I always used verify after burning to
make sure the discs were good.)

Then I came across this:
http://www.larryjordan.biz/articles/...k_warning.html

So, do I still need to worry, or is that bull****? Perhaps I need to look
for a utility that "refreshes" the data by reading and writing the data
back to disk?

I have an old Engineer friend who used to work for Quantum a number of
years ago and his response was "Depends on the media but yes, it can lose
data if power off for a very long time and if subjected to magnetic
interference." Of course being more than 10 years ago, things have
possibly changed...

Thanks.




No just use Gold 100 year archival CD/DVD the DVD's are only rated at
80 years, there some 4-5 firms that make them or Cased DVD RAM 100 year
life span.

CD/DVD in whatever version are unsuitable for archiving, unless
you do extensive evaluation on a specific burner+media combination
and have a binding assurance from the media vendor that they
will not change the media characteristics. Even with that,
100 years is completely fictional under real conditions.

Arno


--
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email:
GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

On 11 May 2011 12:00:20 GMT Arno wrote in Message id:
:

William Brown wrote:
On Tue, 10 May 2011 11:33:43 -0400, JW wrote:


After reading about the (not so) longevity of using CD-R and DVD-R as
archival media I decided to get two 2TB Hitachi drives from Newegg keeping
them mirrored in case one dies. As I was moving the data from the optical
media to the fixed disks, about 10 of 350 or so optical discs I had gave
me fits (CRC errors, bad blocks, etc.) I was able to recover 8 of them and
2 were a complete loss. (Note that I always used verify after burning to
make sure the discs were good.)

Then I came across this:
http://www.larryjordan.biz/articles/...k_warning.html

So, do I still need to worry, or is that bull****? Perhaps I need to look
for a utility that "refreshes" the data by reading and writing the data
back to disk?

I have an old Engineer friend who used to work for Quantum a number of
years ago and his response was "Depends on the media but yes, it can lose
data if power off for a very long time and if subjected to magnetic
interference." Of course being more than 10 years ago, things have
possibly changed...

Thanks.




No just use Gold 100 year archival CD/DVD the DVD's are only rated at
80 years, there some 4-5 firms that make them or Cased DVD RAM 100 year
life span.

CD/DVD in whatever version are unsuitable for archiving, unless
you do extensive evaluation on a specific burner+media combination
and have a binding assurance from the media vendor that they
will not change the media characteristics. Even with that,
100 years is completely fictional under real conditions.

Additionally, having the data on DVD and CD-R media was becoming a real
hassle anyway when I needed to access it. I think I will go with the
advice of using 3 drives, and keeping one of them offsite. I'll also
schedule long SMART tests. Would Hitachi's DFT be the correct tool
http://www.hitachigst.com/support/downloads/#DFT

Or is there a third party tool that is better or recommended?

Thanks to all for your suggestions.

On May 11, 2:34*pm, JW wrote:
On 11 May 2011 12:00:20 GMT Arno wrote in Message id:
:





William Brown wrote:
On Tue, 10 May 2011 11:33:43 -0400, JW wrote:

After reading about the (not so) longevity of using CD-R and DVD-R as
archival media I decided to get two 2TB Hitachi drives from Newegg keeping
them mirrored in case one dies. As I was moving the data from the optical
media to the fixed disks, about 10 of 350 or so optical discs I had gave
me fits (CRC errors, bad blocks, etc.) I was able to recover 8 of them and
2 were a complete loss. (Note that I always used verify after burning to
make sure the discs were good.)

Then I came across this:
http://www.larryjordan.biz/articles/...k_warning.html

So, do I still need to worry, or is that bull****? Perhaps I need to look
for a utility that "refreshes" the data by reading and writing the data
back to disk?

I have an old Engineer friend who used to work for Quantum a number of
years ago and his response was "Depends on the media but yes, it can lose
data if power off for a very long time and if subjected to magnetic
interference." Of course being more than 10 years ago, things have
possibly changed...

Thanks.

No just use Gold 100 year archival CD/DVD *the DVD's are only rated at
80 years, there some 4-5 firms that make them or Cased *DVD RAM 100 year
life span.

CD/DVD in whatever version are unsuitable for archiving, unless
you do extensive evaluation on a specific burner+media combination
and have a binding assurance from the media vendor that they
will not change the media characteristics. Even with that,
100 years is completely fictional under real conditions.

Additionally, having the data on DVD and CD-R media was becoming a real
hassle anyway when I needed to access it. I think I will go with the
advice of using 3 drives, and keeping one of them offsite. I'll also
schedule long SMART tests. Would Hitachi's DFT be the correct toolhttp://www.hitachigst.com/support/downloads/#DFT

Or is there a third party tool that is better or recommended?

Thanks to all for your suggestions.

As well as drives - consider online storage as PART of your backup
scheme.

On 5/10/2011 8:23 PM, Arno wrote:
Ed wrote:
On 5/10/2011 2:26 PM, Arno wrote:

Depending on your local customs, you might be able to
rent a safe deposit box cheaply.

I have one of my USB backup drives in a safe deposit box. I take it out
once a month and update the backups.

....

What do you pay for the box? Here in Swizerland it
is something like 100EUR/USD per year for a standard-sized
box that can take two external 3.5" drives.

It's $40 a year for a box wide enough for DVD's.
--
Ed Light

Better World News TV Channel:
http://realnews.com

Iraq Veterans Against the War and Related:
http://ivaw.org
http://couragetoresist.org
http://antiwar.com

Send spam to the FTC at

Thanks, robots.

Ed Light wrote:
On 5/10/2011 8:23 PM, Arno wrote:
Ed wrote:
On 5/10/2011 2:26 PM, Arno wrote:

Depending on your local customs, you might be able to
rent a safe deposit box cheaply.

I have one of my USB backup drives in a safe deposit box. I take it out
once a month and update the backups.

...

What do you pay for the box? Here in Swizerland it
is something like 100EUR/USD per year for a standard-sized
box that can take two external 3.5" drives.

It's $40 a year for a box wide enough for DVD's.

Thanks for the info. You do not get ones that small here,
but this price is entirely reasonable.

Arno
--
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email:
GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

Amusing and somewhat timely discussion, working through regenerating
backups.

DAT
Old 2003 DDS2 tape took a few goes to get data off in DDS4 drive,
despite correctly stored & retension first. I was considering using
DAT40 or DAT72 as a secondary backup device, but I fundamentally
distrust data restored from it. It is very easy to get a creeping
corruption which gets subsequently re backed over the years and only
detected years later. Not exactly good if it is tax & account
archives.

HD
Ironically I have never lost any data stored on hard drive, but I
introduce new drives into the pool regularly and use multiple brands.

CD/DVD
Early burners & poor dye were a recipe for disaster, running some A-
One Gold tonight on a burner I got 40% failure - just picked them up
on the way home. It shows the need to match media (Verbatim Indonesia
a better bet) with drive (which do not seem to be built with longevity
in mind).

MO
Magneto optical 3.5" has proven totally reliable, bar a few miss-
formatted discs in the early 2000s easily spotted as they were
unusable. Media from Sony, Maxell, Fujifilm & Verbatim - probably only
2 manufacturers in there. Small capacity at 640MB limits use to core
critical data, but longevity seems ok thus far. Spare brand new drives
checked and unlike dreaded DDS2 I can guarantee thus far to read a
disk in any drive without problem.

Paper
Ironically this still seems to be quite robust! Unfortunately the cost
of storing several hundred thousand pages is actually pretty
ridiculous in "self store" units these days. A lot of companies are
shredding for paper-less office.

Frankly it comes down to data set size & criticality, usually there is
a smaller critical data set which can not be regenerated and for that
higher redundancy across multiple media is financially practical
rather than applying the same cost structure to the entire data set.
Multiple copies in multiple locations really does matter. I am not too
impressed by "clouds" re recent security breaches, and apart from LTO
1/2" tape I am not impressed with tape - it makes me uneasy. HD makers
have a history of shovelling out the initial-dip products as they
change geographic location than binning them (every time Seagate
changed plant location the first products off the line were doorstops
but still ended up in retail).

Dropping both backups down the stairs is not a joke, been there - one
a HD & one a DVD-RAM cartridge. The HD was scrap, the DVD-RAM ok but
from re-using the same DVD-RAM disk in video recorders I am wholly
unimpressed by their "overwrite count" despite being type-4 cartridge.

MO & HD seems to win out, MO for the critical stuff and never sell
that spare drive. One reason why LTO is twice the price it appears
unfortunately.

wrote:
Amusing and somewhat timely discussion, working through regenerating
backups.

DAT
Old 2003 DDS2 tape took a few goes to get data off in DDS4 drive,
despite correctly stored & retension first. I was considering using
DAT40 or DAT72 as a secondary backup device, but I fundamentally
distrust data restored from it. It is very easy to get a creeping
corruption which gets subsequently re backed over the years and only
detected years later. Not exactly good if it is tax & account
archives.

No surprise there. I did some reads with individual
timing (shows whether the drive needs rewind and retry)
a long time ago and decided that DAT was basically consumer
grade.

HD
Ironically I have never lost any data stored on hard drive, but I
introduce new drives into the pool regularly and use multiple brands.

I have, but not without warning. Unless you cound the
ine time were I made a 4-way Y-cable from a a pair where
red and yellow were reversed in one and I did not notice.
That killed 2 drives spectacularly. FOrtunately I had backup.

CD/DVD
Early burners & poor dye were a recipe for disaster, running some A-
One Gold tonight on a burner I got 40% failure - just picked them up
on the way home. It shows the need to match media (Verbatim Indonesia
a better bet) with drive (which do not seem to be built with longevity
in mind).

MO
Magneto optical 3.5" has proven totally reliable, bar a few miss-
formatted discs in the early 2000s easily spotted as they were
unusable. Media from Sony, Maxell, Fujifilm & Verbatim - probably only
2 manufacturers in there. Small capacity at 640MB limits use to core
critical data, but longevity seems ok thus far. Spare brand new drives
checked and unlike dreaded DDS2 I can guarantee thus far to read a
disk in any drive without problem.

I second that. Unfortunately my MOD drive was SCSI and I could
not get the controller to work anymore without jumpong through
hoops. I did pull all data off my MODs some years ago without
any problems, and some of them written 7-8 years before.
Definitely an archival media that deserves the term. Also
definitely not commercially viable, it seems people do not
care about longterm archiving or just do not get it. The number
of people still recommending consumer-trash writable DVDs
as archival medium in this group tells it all.


Paper
Ironically this still seems to be quite robust! Unfortunately the cost
of storing several hundred thousand pages is actually pretty
ridiculous in "self store" units these days. A lot of companies are
shredding for paper-less office.

I have one book that has a "data lifetime statement" of
several hundred years with regard to the paper in it.

Frankly it comes down to data set size & criticality, usually there is
a smaller critical data set which can not be regenerated and for that
higher redundancy across multiple media is financially practical
rather than applying the same cost structure to the entire data set.
Multiple copies in multiple locations really does matter. I am not too
impressed by "clouds" re recent security breaches, and apart from LTO
1/2" tape I am not impressed with tape - it makes me uneasy. HD makers
have a history of shovelling out the initial-dip products as they
change geographic location than binning them (every time Seagate
changed plant location the first products off the line were doorstops
but still ended up in retail).

Dropping both backups down the stairs is not a joke, been there - one
a HD & one a DVD-RAM cartridge. The HD was scrap, the DVD-RAM ok but
from re-using the same DVD-RAM disk in video recorders I am wholly
unimpressed by their "overwrite count" despite being type-4 cartridge.

DVD-RAM is really, really bad. I did evaluate a set of different
cartritges a while ago and was very unimpressed. Many did exceed
the ISO error limits and only repeated full overwrites grought
dem down a bit. I don't need this high level of media maintenance
at all.

MO & HD seems to win out, MO for the critical stuff and never sell
that spare drive.

All the major data recovery outfits do recover MOs.

One reason why LTO is twice the price it appears
unfortunately.

It is very unfortunate that MO did not get enough sales for the
vendors to continue development. I now use spinning drives
(one on my side with 3-way RAID1, 2 virtual servers) and
regular automated tests of the data for critical data
and two external HDs with hadrware tetst every few months
for less critical stuff. So far so good. But except going back
to MO, I do not see any write & store solution. As to the
"cloud", the word "pathetic" comes to mind.

BTW, I completely agree that archival data falls into "must have"
and "nive to have" classes that justify different levels of effort.

Arno

--
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email:
GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

On May 25, 1:24*am, Arno wrote:
I second that. Unfortunately my MOD drive was SCSI and I could
not get the controller to work anymore without jumpong through
hoops. I did pull all data off my MODs some years ago without
any problems, and some of them written 7-8 years before.

Likewise.
I sold my SCSI MO drive & Adaptec EZ-SCSI PCMCIA because its short
cable was cumbersome with a laptop and a growing data set meant
generating 2-3 copies became expensive. I moved to HD & DVD-RAM,
mainly due to cheap 4.7/9.4GB phase change media & cheap drives.

Thankfully the HD backups remained intact: DVD-RAM demonstrated it
could lose data and when subjected to repeated overwrites could lose
video far sooner than specifications suggested.

I returned to MO very quickly.


Definitely an archival media that deserves the term. Also
definitely not commercially viable, it seems people do not
care about longterm archiving or just do not get it. The number
of people still recommending consumer-trash writable DVDs
as archival medium in this group tells it all.

I do not understand it either.
CDR DVD are ok for content which you can recreate - such as films.
My experience is they are more a vehicle for those selling than those
betting their own business survivability on them.

DVD-RAM is really, really bad. I did evaluate a set of different
cartritges a while ago and was very unimpressed. Many did exceed
the ISO error limits and only repeated full overwrites grought
dem down a bit. I don't need this high level of media maintenance
at all.

As I found, used like a "fixed disk" in a DVD-RAM video recorder their
overwrite count was dire before "recorder lockup".

MO & HD seems to win out, MO for the critical stuff and never sell
that spare drive.

All the major data recovery outfits do recover MOs.

MO was great for tax & medical records.
Tape requires good technology & good storage technique.

It is very unfortunate that MO did not get enough sales for the
vendors to continue development. I now use spinning drives
(one on my side with 3-way RAID1, 2 virtual servers) and
regular automated tests of the data for critical data
and two external HDs with hadrware tetst every few months
for less critical stuff. So far so good. But except going back
to MO, I do not see any write & store solution. As to the
"cloud", the word "pathetic" comes to mind.

Cloud is marketing creating a revenue stream out of a box of bits :-)

BTW, I completely agree that archival data falls into "must have"
and "nive to have" classes that justify different levels of effort.

The problem with all media is there comes a point when it moves from
cash-cow to cash-dog and quality control can decline, which may be the
media you just stored the business archives on. Hard drives are
particularly vulnerable to this, at least when miniscribe shipped
bricks you could chisel some 1s & 0s on them. The other good thing
about paper is (as Enron & Arthur Anderson found) that it takes a long
time to shred a pile of paper, but very little time to shred some
files (or DVDs).

wrote:
On May 25, 1:24?am, Arno wrote:
[...]
The problem with all media is there comes a point when it moves from
cash-cow to cash-dog and quality control can decline, which may be the
media you just stored the business archives on. Hard drives are
particularly vulnerable to this, at least when miniscribe shipped
bricks you could chisel some 1s & 0s on them.

Before my time, I think when I got my first HDD, Miniscribe was
already out of the business.

The other good thing
about paper is (as Enron & Arthur Anderson found) that it takes a long
time to shred a pile of paper, but very little time to shred some
files (or DVDs).

Indeed. Whether that is an advantage or disadvantage depends on
your point of view though ;-)

Arno

--
Arno Wagner, Dr. sc. techn., Dipl. Inform., CISSP -- Email:
GnuPG: ID: 1E25338F FP: 0C30 5782 9D93 F785 E79C 0296 797F 6B50 1E25 338F
----
Cuddly UI's are the manifestation of wishful thinking. -- Dylan Evans

I would argue against the very idea of archiving. Commodity PC hardware
is very cheap, and can be forced into serving as a reliable backup
system. But it cannot be forced into serving as a reliable archive
system.

Here is my minimal arrangement for data security. The principal behind
this scheme is protection against an undetected failure. If you insist
on using an archive scheme with commodity PC hardware (and software!)
then you can never obtain this protection. This scheme requires a
minimum of three USB disk drives that are used as backup media.

I start by KEEPING all of my data in my computer. Old photos, email from
ten years ago, all of it. No exceptions! No archived media! I am not
going to spend ten thousand dollars on professional grade tape libraries
and I am certainly not going to trust commodity (cheap) peecee hardware
to archive any data that I care about.

I keep two drives near the computer, and I use them to perform backups
on alternate days. In this way, I get a measure of protection against
failure of backup software, which is a non-trivial concern. I also get
protection against hardware failure in the backup media (disk drives).

The third drive I keep offsite. This gives me protection against fire,
theft, incoming artillery, and so forth.

About once a month, I rotate the positions of the three disk drives.

The key point is that I regularly write to all of my backup media. This
serves as a test for hardware failure. In other words, I am constantly
defending myself against the undetected failure.

If I want greater security, I rotate the drives more frequently. If I
want longer backup history, I add more USB drives to the rotation.
--
David Arnstein (00)
{{ }}
^^