How would you store 100TB data?

Hi, I'm a student and out of my curiousness I want to know how data
centers usually store their data. Some mail provider liker Gmail and
Hotmail must have huge volume of data, even properly compressed. I just
can't imagine they could store it in hard drives. Is there any special
devices?

Thanks in advance.

writes:
Hi, I'm a student and out of my curiousness I want to know how data
centers usually store their data. Some mail provider liker Gmail and
Hotmail must have huge volume of data, even properly compressed. I just
can't imagine they could store it in hard drives. Is there any special
devices?

Of course they use hard drives. Installed in umpteen thousand
computers. What else would they use?!

On 25 Feb 2006 15:28:24 -0800, wrote:

Hi, I'm a student and out of my curiousness I want to know how data
centers usually store their data. Some mail provider liker Gmail and
Hotmail must have huge volume of data, even properly compressed. I just
can't imagine they could store it in hard drives. Is there any special
devices?



If the data is needed quickly then disk is it. Just check out the some
vendors websites - HP (whose products I will use a examples), IBM, EMC,
Fujitsu, Hitachi, 3Par, ... the list goes on.

One example would be the XP1200. Currently, it supports upto 290TB
usable storage per array (5 cabinets in width).

http://h18006.www1.hp.com/products/s...000/index.html

For Linux HPC there is the Scalable File Share (HP SFS) which can scale
upt to 1024TB.

http://h20311.www2.hp.com/HPC/cache/...0-0-0-121.html


If the need is for nearline archival storage, then products such as the
StorageWorks 6000 Virtual Library System can the used. It currently can
store upto 70TB per system.

http://h18006.www1.hp.com/products/s...vls/index.html

Thanks. You really broadened my vision.

In article .com,
wrote:
Hi, I'm a student and out of my curiousness I want to know how data
centers usually store their data. Some mail provider liker Gmail and
Hotmail must have huge volume of data, even properly compressed. I just
can't imagine they could store it in hard drives. Is there any special
devices?

In a nutshell, they are stored on disks. Typically, one takes several
or many disks, and puts them together in a "disk array" - a box
ranging in size from a toaster (3U rackmount) to several
refrigerators, containing anywhere between a half-dozen and several
thousand disks. They range in cost from $1K to $10M. Disk arrays are
typically connected to the computer by a variety of network
technologies, ranging from SCSI or SATA for low-end to fibre channel
and ESCON/FICON for high end; Ethernet (iSCSI and other IP-based
protocols) is making rapid inroads.

Another option is to build a box that contains many disks, and then
serves file system storage protocols such as NFS and CIFS. This is
known as NAS or network-attached-storage. Such NAS boxes again range
in size from toaster to multiple refrigerators, with similar cost
ranges.

The largest single storage system I know of is the disk farm used by
the Livermore supercomputer; I think it contains about 10000 disks,
and stores several petabytes. I'm sure you can google for the
details. There are many places that have more storage than this
system, but typically not in a single storage system.

Since disks are so unreliable (they are typically the least reliable
computer thing in a data center, excluding the air conditioning, which
tends to be even less reliable, and often leaks all over the place),
disk arrays and NAS boxes typically use multiple disks in a redundant
way. The easiest thing is to store every byte on two separate disks,
which in effect amounts to always using pairs of disks to build
mirrored virtual disks (known is the trade as RAID-1). More efficient
ways to achieve redundancy use erasure codes, beginning with simple
parity (typically known as RAID-5). For even higher reliability, one
can use more than copies of the data. High-end disk arrays often have
the capability to automatically copy the data over wide-area network
links to remote sites (other buldings, other cities), thereby
achieving reliability in the case of site desasters.

To store 100 TB today, you only need 200 disks (with 500GB SATA
disks); with redundancy you better make that 250 to 400 disks. With
the most current technology, this will require about one to two racks
worth of real estate (about the size of a really large refrigerator);
if you go for the lowest cost, it should be somewhere between a
quarter and half miliion. A really nice 100TB system (with
redundancy, remote mirroring, great management software, from a
supplier with a good reputation and excellent customer support) can
easily cost you 10x that much. Usually, you get what you pay for,
unless you manage to find a slightly discounted lunch.

In todays data centers, 100TB systems are common; many corporations
store considerable more data than this. Mail providers and
web-related businesses (microsoft, google, yahoo) are actually not
particularly large users of storage; a lot of storage is also used in
banks, insurance, manufacturing, research, medical, as archives, and
in government agencies.

--
The address in the header is invalid for obvious reasons. Please
reconstruct the address from the information below (look for _).
Ralph Becker-Szendy

it seems the hard disk is quite expensive. but why they didn't use
optic based disc? it's cheaper and also can hold large data. i heard HP
have an optic drive with 10+ optic pickup head, which can faster
position and pick the data.

"Ed Wilts" writes:
5 years ago we had 10TB on the floor. Now it's 150. We've got an
internal pool going for when we hit a petabyte :-)

Out of curiosity what kind of data is it?

In article 1140933623.229616@smirk,
wrote:

Since disks are so unreliable (they are typically the least reliable

Oh really?

computer thing in a data center, excluding the air conditioning, which

What about electronics, motherboards, CPUs, memory?

In todays data centers, 100TB systems are common;

I wonder if there are ANY data centers that store 100 TB let alone...

many corporations
store considerable more data than this. Mail providers and

Paul Rubin wrote:
"Ed Wilts" writes:
5 years ago we had 10TB on the floor. Now it's 150. We've got an
internal pool going for when we hit a petabyte :-)

Out of curiosity what kind of data is it?

Scanned images and database files probably make up the majority. Then
there's the usual miscellaneous home area files, Exchange, etc.. 5
different operating systems access our SAN today.

/Ed

In article ,
HVB wrote:

On Tue, 28 Feb 2006 00:14:40 -0500, flux wrote:

I wonder if there are ANY data centers that store 100 TB let alone...

?!?!

I know one data centre that has 1PB (yes, you read that right) of
useable storage, with over half of it actually consumed.

I'm currently designing a new data centre which will require over
2.5PB of useable storage capacity. The amount of actual raw storage
required is very much higher than this.

These sound like special exceptions.

So, yes, plenty of data centres actually store more than 100TB of
data.

You seem to be saying that 2 cases you just quoted count for plenty.