At what point is ILM needed?

Hi,
I am trying to figure out at what point I should be looking into some
sort of ILM technology. In reading vendor case studies such as
(http://www.netapp.com/library/cs/mustang.pdf), I dont see why I cant
just keep adding more disks to my primary storage right now as demand
increases... when should I be looking into ILM (at what storage
capacity)? How many GigaBytes or TeraBytes before I should consider an
ILM strategy?

Jon

On 7 Aug 2006 13:38:41 -0700, wrote:

Hi,
I am trying to figure out at what point I should be looking into some
sort of ILM technology. In reading vendor case studies such as
(http://www.netapp.com/library/cs/mustang.pdf), I dont see why I cant
just keep adding more disks to my primary storage right now as demand
increases... when should I be looking into ILM (at what storage
capacity)? How many GigaBytes or TeraBytes before I should consider an
ILM strategy?

Jon

Generally speaking ILM is used for legal purposes. Retention for
things like Intellectual Property and SOX compliance are usually the
driving factor. In some cases pure volume is the driver but those are
less frequent these days for just the reason you stated.

If I can get 500GB SATA drives, move data to those and stop backups,
why invest in ILM insfrastructure? You wouldn't.

But for pure volume I don't think you can beat tape as the backend
medium for cost. Footprint, capacity, heat, power, etc., all are won
by tape. Longevity and ease of migration are won by disk.

There is no magic dial that says "invest in ILM here" so you've got to
make judegement calls.

If you want to provide more information there may be people here who
can help.

~F

Faeandar wrote:
On 7 Aug 2006 13:38:41 -0700, wrote:

Hi,
I am trying to figure out at what point I should be looking into some
sort of ILM technology. In reading vendor case studies such as
(http://www.netapp.com/library/cs/mustang.pdf), I dont see why I cant
just keep adding more disks to my primary storage right now as demand
increases... when should I be looking into ILM (at what storage
capacity)? How many GigaBytes or TeraBytes before I should consider an
ILM strategy?

Jon

Generally speaking ILM is used for legal purposes. Retention for
things like Intellectual Property and SOX compliance are usually the
driving factor. In some cases pure volume is the driver but those are
less frequent these days for just the reason you stated.

If I can get 500GB SATA drives, move data to those and stop backups,
why invest in ILM insfrastructure? You wouldn't.

But for pure volume I don't think you can beat tape as the backend
medium for cost. Footprint, capacity, heat, power, etc., all are won
by tape. Longevity and ease of migration are won by disk.

There is no magic dial that says "invest in ILM here" so you've got to
make judegement calls.

If you want to provide more information there may be people here who
can help.

~F

That helps somewhat... is there some specific growth rate at which ILM
for pure volume makes sense? Should I be worried about 4 TB of data in
my primary storage? I keep hearing about so many case studies etc
where poeple have applied ILM to help primary storage volume and my own
management says I should be doing the same... So I cant understand the
disconnect...

Jon.

On 7 Aug 2006 15:13:18 -0700, wrote:


Faeandar wrote:
On 7 Aug 2006 13:38:41 -0700, wrote:

Hi,
I am trying to figure out at what point I should be looking into some
sort of ILM technology. In reading vendor case studies such as
(http://www.netapp.com/library/cs/mustang.pdf), I dont see why I cant
just keep adding more disks to my primary storage right now as demand
increases... when should I be looking into ILM (at what storage
capacity)? How many GigaBytes or TeraBytes before I should consider an
ILM strategy?

Jon

Generally speaking ILM is used for legal purposes. Retention for
things like Intellectual Property and SOX compliance are usually the
driving factor. In some cases pure volume is the driver but those are
less frequent these days for just the reason you stated.

If I can get 500GB SATA drives, move data to those and stop backups,
why invest in ILM insfrastructure? You wouldn't.

But for pure volume I don't think you can beat tape as the backend
medium for cost. Footprint, capacity, heat, power, etc., all are won
by tape. Longevity and ease of migration are won by disk.

There is no magic dial that says "invest in ILM here" so you've got to
make judegement calls.

If you want to provide more information there may be people here who
can help.

~F

That helps somewhat... is there some specific growth rate at which ILM
for pure volume makes sense? Should I be worried about 4 TB of data in
my primary storage? I keep hearing about so many case studies etc
where poeple have applied ILM to help primary storage volume and my own
management says I should be doing the same... So I cant understand the
disconnect...

Jon.

Well, the disconnect is usually management that has heard about this
new thing that will "save them a ton of money". 4TB of data is huge
to some, and a pitance to others. The pure volume is hard to quantify
for ILM.

What storage type is your 4TB on? How long did it take you to get
there? What is your anticipated growth over the next 6-12 months?
What does your backup infrastructure consist of (drive classes, number
of tapes, backup servers, etc)? What type of data is it (structured,
unstructured, IP, e-mail, home directories)?

All these things will be used in a formula that will only work for
your environment.

~F

Faeandar wrote:
On 7 Aug 2006 15:13:18 -0700, wrote:


Faeandar wrote:
On 7 Aug 2006 13:38:41 -0700, wrote:

Hi,
I am trying to figure out at what point I should be looking into some
sort of ILM technology. In reading vendor case studies such as
(http://www.netapp.com/library/cs/mustang.pdf), I dont see why I cant
just keep adding more disks to my primary storage right now as demand
increases... when should I be looking into ILM (at what storage
capacity)? How many GigaBytes or TeraBytes before I should consider an
ILM strategy?

Jon

Generally speaking ILM is used for legal purposes. Retention for
things like Intellectual Property and SOX compliance are usually the
driving factor. In some cases pure volume is the driver but those are
less frequent these days for just the reason you stated.

If I can get 500GB SATA drives, move data to those and stop backups,
why invest in ILM insfrastructure? You wouldn't.

But for pure volume I don't think you can beat tape as the backend
medium for cost. Footprint, capacity, heat, power, etc., all are won
by tape. Longevity and ease of migration are won by disk.

There is no magic dial that says "invest in ILM here" so you've got to
make judegement calls.

If you want to provide more information there may be people here who
can help.

~F

That helps somewhat... is there some specific growth rate at which ILM
for pure volume makes sense? Should I be worried about 4 TB of data in
my primary storage? I keep hearing about so many case studies etc
where poeple have applied ILM to help primary storage volume and my own
management says I should be doing the same... So I cant understand the
disconnect...

Jon.

Well, the disconnect is usually management that has heard about this
new thing that will "save them a ton of money". 4TB of data is huge
to some, and a pitance to others. The pure volume is hard to quantify
for ILM.

What storage type is your 4TB on? How long did it take you to get
there? What is your anticipated growth over the next 6-12 months?
What does your backup infrastructure consist of (drive classes, number
of tapes, backup servers, etc)? What type of data is it (structured,
unstructured, IP, e-mail, home directories)?

All these things will be used in a formula that will only work for
your environment.

~F

Our 4TB of data are on two FAS960Cs... we are expected to grow 50% this
year (year end we will have 6TB) and next year we expect to add another
4TB. We just started growing rapidly due to the nature of construction
projects being taken on, and they mostly consist of engineering and
design documents... all unstructured... actually, depending on how you
look at it, 70% unstructured and 30% email.

We have backup (tape) hanging off of the netapp directly and use
veritas to move stuff into backup (bi weekly I beleive, but I dont
manage that).

I spend about 15K per TB of data and our current solution cost about
$450K.

PS: One area where I get dinged all the time is the backup restore
times... management says I should do it faster and tell me I should be
looking into ILM for that. Also, they cringe whenever I go and put a
PO in for another TB of data.

On 7 Aug 2006 15:51:40 -0700, wrote:



Our 4TB of data are on two FAS960Cs... we are expected to grow 50% this
year (year end we will have 6TB) and next year we expect to add another
4TB. We just started growing rapidly due to the nature of construction
projects being taken on, and they mostly consist of engineering and
design documents... all unstructured... actually, depending on how you
look at it, 70% unstructured and 30% email.

We have backup (tape) hanging off of the netapp directly and use
veritas to move stuff into backup (bi weekly I beleive, but I dont
manage that).

I spend about 15K per TB of data and our current solution cost about
$450K.

PS: One area where I get dinged all the time is the backup restore
times... management says I should do it faster and tell me I should be
looking into ILM for that. Also, they cringe whenever I go and put a
PO in for another TB of data.

Well, since the 960's will expand to 64TB you've got plenty of grow
room.

For backup performance, if you don;t manage it why do you get dinged?
And how is it being done today? Volume level backups? Qtree level?
Any replication involved (snapmirror or snapvault)?

ILM is not a fix for any problem, period. It is a conglomerate of
products/programs/scripts that automate a business process. The
business process is the fix. Never lose sight of that and your battle
is assured.

Does your business have a process for long term retention? Data
destruction? E-mail compliance? Whatever it happens to be, ILM can
wrap some automation around it but the process and requirements have
to exist first. Your management needs an education and you are going
to have to do it. But first you need to educate yourself, and this
was a good start.

Take a look at some of the ILM procucts out there. See what they do.
You'll find ILM is just HSM rebranded with a few more bells and
whistles attached. Information Lifecycle Management, the name alone
should fire off some bells and sirens for you to deal with your
management.

The flip side is ILM is a great resume builder. However, I'd bet that
if you did due diligence, set your requirements, selected a vendor,
then got a quote, management would be happy to sign PO's for TB's of
storage for the next 2-3 years.

Don;t forget all the hidden costs of ILM; it's not just the software.
Tape, disk, switch ports, FC ports, whatever. Make sure you add it
all in.

~F

Faeandar wrote:
On 7 Aug 2006 15:51:40 -0700, wrote:



Our 4TB of data are on two FAS960Cs... we are expected to grow 50% this
year (year end we will have 6TB) and next year we expect to add another
4TB. We just started growing rapidly due to the nature of construction
projects being taken on, and they mostly consist of engineering and
design documents... all unstructured... actually, depending on how you
look at it, 70% unstructured and 30% email.

We have backup (tape) hanging off of the netapp directly and use
veritas to move stuff into backup (bi weekly I beleive, but I dont
manage that).

I spend about 15K per TB of data and our current solution cost about
$450K.

PS: One area where I get dinged all the time is the backup restore
times... management says I should do it faster and tell me I should be
looking into ILM for that. Also, they cringe whenever I go and put a
PO in for another TB of data.

Well, since the 960's will expand to 64TB you've got plenty of grow
room.

For backup performance, if you don;t manage it why do you get dinged?
And how is it being done today? Volume level backups? Qtree level?
Any replication involved (snapmirror or snapvault)?

ILM is not a fix for any problem, period. It is a conglomerate of
products/programs/scripts that automate a business process. The
business process is the fix. Never lose sight of that and your battle
is assured.

Does your business have a process for long term retention? Data
destruction? E-mail compliance? Whatever it happens to be, ILM can
wrap some automation around it but the process and requirements have
to exist first. Your management needs an education and you are going
to have to do it. But first you need to educate yourself, and this
was a good start.

Take a look at some of the ILM procucts out there. See what they do.
You'll find ILM is just HSM rebranded with a few more bells and
whistles attached. Information Lifecycle Management, the name alone
should fire off some bells and sirens for you to deal with your
management.

The flip side is ILM is a great resume builder. However, I'd bet that
if you did due diligence, set your requirements, selected a vendor,
then got a quote, management would be happy to sign PO's for TB's of
storage for the next 2-3 years.

Don;t forget all the hidden costs of ILM; it's not just the software.
Tape, disk, switch ports, FC ports, whatever. Make sure you add it
all in.

~F

Thanks... well I get dinged because its my data thats increasing the
backup restore window times... and the backup guys say I have too much
data to be restored...

In any event, I understand your arguments.

One other question... IN our company, overall we must have over 100TB
of data (I just manage 4TB of that)... I cant help but imagine that
someone should be consolidating all of this... I will raise this to
management, but before I do so, do folks here think this is a big
problem just in our company or are others out there like this... ?

How many others have a similar situation?

Thanks
Jon.

wrote:
Faeandar wrote:
On 7 Aug 2006 15:51:40 -0700, wrote:


Our 4TB of data are on two FAS960Cs... we are expected to grow 50% this
year (year end we will have 6TB) and next year we expect to add another
4TB. We just started growing rapidly due to the nature of construction
projects being taken on, and they mostly consist of engineering and
design documents... all unstructured... actually, depending on how you
look at it, 70% unstructured and 30% email.

We have backup (tape) hanging off of the netapp directly and use
veritas to move stuff into backup (bi weekly I beleive, but I dont
manage that).

I spend about 15K per TB of data and our current solution cost about
$450K.

PS: One area where I get dinged all the time is the backup restore
times... management says I should do it faster and tell me I should be
looking into ILM for that. Also, they cringe whenever I go and put a
PO in for another TB of data.
Well, since the 960's will expand to 64TB you've got plenty of grow
room.

For backup performance, if you don;t manage it why do you get dinged?
And how is it being done today? Volume level backups? Qtree level?
Any replication involved (snapmirror or snapvault)?

ILM is not a fix for any problem, period. It is a conglomerate of
products/programs/scripts that automate a business process. The
business process is the fix. Never lose sight of that and your battle
is assured.

Does your business have a process for long term retention? Data
destruction? E-mail compliance? Whatever it happens to be, ILM can
wrap some automation around it but the process and requirements have
to exist first. Your management needs an education and you are going
to have to do it. But first you need to educate yourself, and this
was a good start.

Take a look at some of the ILM procucts out there. See what they do.
You'll find ILM is just HSM rebranded with a few more bells and
whistles attached. Information Lifecycle Management, the name alone
should fire off some bells and sirens for you to deal with your
management.

The flip side is ILM is a great resume builder. However, I'd bet that
if you did due diligence, set your requirements, selected a vendor,
then got a quote, management would be happy to sign PO's for TB's of
storage for the next 2-3 years.

Don;t forget all the hidden costs of ILM; it's not just the software.
Tape, disk, switch ports, FC ports, whatever. Make sure you add it
all in.

~F

Thanks... well I get dinged because its my data thats increasing the
backup restore window times... and the backup guys say I have too much
data to be restored...

In any event, I understand your arguments.

One other question... IN our company, overall we must have over 100TB
of data (I just manage 4TB of that)... I cant help but imagine that
someone should be consolidating all of this... I will raise this to
management, but before I do so, do folks here think this is a big
problem just in our company or are others out there like this... ?

How many others have a similar situation?

Thanks
Jon.

Not sure I understand. Your business unit (you) creates only 4% of the
company's data and YOU are under the gun to create an ILM strategy? I
don't think you'll find many others in that situation.

Does the remaining 96% have an ILM strategy?

Also I can't help noticing a disconnect. In addition to ILM, your
management is asking for faster backups but you think consolidation
(that would probably lengthen backup times) is needed. What gives?

If I may make a suggestion try to learn more about ILM (BTW is NOT a
technology). The EMC site is a good place where to start, not because
they are necessarily the best, but they have just about every peace to
implement an ILM strategy, and a rich Web site.

mapicella wrote:
wrote:
Faeandar wrote:
On 7 Aug 2006 15:51:40 -0700, wrote:


Our 4TB of data are on two FAS960Cs... we are expected to grow 50% this
year (year end we will have 6TB) and next year we expect to add another
4TB. We just started growing rapidly due to the nature of construction
projects being taken on, and they mostly consist of engineering and
design documents... all unstructured... actually, depending on how you
look at it, 70% unstructured and 30% email.

We have backup (tape) hanging off of the netapp directly and use
veritas to move stuff into backup (bi weekly I beleive, but I dont
manage that).

I spend about 15K per TB of data and our current solution cost about
$450K.

PS: One area where I get dinged all the time is the backup restore
times... management says I should do it faster and tell me I should be
looking into ILM for that. Also, they cringe whenever I go and put a
PO in for another TB of data.
Well, since the 960's will expand to 64TB you've got plenty of grow
room.

For backup performance, if you don;t manage it why do you get dinged?
And how is it being done today? Volume level backups? Qtree level?
Any replication involved (snapmirror or snapvault)?

ILM is not a fix for any problem, period. It is a conglomerate of
products/programs/scripts that automate a business process. The
business process is the fix. Never lose sight of that and your battle
is assured.

Does your business have a process for long term retention? Data
destruction? E-mail compliance? Whatever it happens to be, ILM can
wrap some automation around it but the process and requirements have
to exist first. Your management needs an education and you are going
to have to do it. But first you need to educate yourself, and this
was a good start.

Take a look at some of the ILM procucts out there. See what they do.
You'll find ILM is just HSM rebranded with a few more bells and
whistles attached. Information Lifecycle Management, the name alone
should fire off some bells and sirens for you to deal with your
management.

The flip side is ILM is a great resume builder. However, I'd bet that
if you did due diligence, set your requirements, selected a vendor,
then got a quote, management would be happy to sign PO's for TB's of
storage for the next 2-3 years.

Don;t forget all the hidden costs of ILM; it's not just the software.
Tape, disk, switch ports, FC ports, whatever. Make sure you add it
all in.

~F

Thanks... well I get dinged because its my data thats increasing the
backup restore window times... and the backup guys say I have too much
data to be restored...

In any event, I understand your arguments.

One other question... IN our company, overall we must have over 100TB
of data (I just manage 4TB of that)... I cant help but imagine that
someone should be consolidating all of this... I will raise this to
management, but before I do so, do folks here think this is a big
problem just in our company or are others out there like this... ?

How many others have a similar situation?

Thanks
Jon.

Not sure I understand. Your business unit (you) creates only 4% of the
company's data and YOU are under the gun to create an ILM strategy? I
don't think you'll find many others in that situation.

Does the remaining 96% have an ILM strategy?

Also I can't help noticing a disconnect. In addition to ILM, your
management is asking for faster backups but you think consolidation
(that would probably lengthen backup times) is needed. What gives?

If I may make a suggestion try to learn more about ILM (BTW is NOT a
technology). The EMC site is a good place where to start, not because
they are necessarily the best, but they have just about every peace to
implement an ILM strategy, and a rich Web site.

We have grown quite chaotic and our growth has been ad-hoc with many
regional offices (typically near the major customers)... we acquired a
lot of small consulting shops - so thats why we have so many
distributed sites... Our dept is however the main IT shop in the
company... no the others do not have one yet as well...

we are learning as we go - but we are paying a lot of money on our
data... and I dont think the remote distrbuted data is helping... I am
sure I should first bring their data in centrally and I think I need to
suggest that first...

Regarding faster backups - this is where I am led to beleive ILM
couldve helped (yes as you say, it is not a product but best
practises)... I guess to refine my original question, should I propose
something like this: 1. consolidate and 2. buy a product that lets us
move data from the netapp filers to some cheaper storage for older data

Jon

wrote:
mapicella wrote:
wrote:
Faeandar wrote:
On 7 Aug 2006 15:51:40 -0700, wrote:


Our 4TB of data are on two FAS960Cs... we are expected to grow 50% this
year (year end we will have 6TB) and next year we expect to add another
4TB. We just started growing rapidly due to the nature of construction
projects being taken on, and they mostly consist of engineering and
design documents... all unstructured... actually, depending on how you
look at it, 70% unstructured and 30% email.

We have backup (tape) hanging off of the netapp directly and use
veritas to move stuff into backup (bi weekly I beleive, but I dont
manage that).

I spend about 15K per TB of data and our current solution cost about
$450K.

PS: One area where I get dinged all the time is the backup restore
times... management says I should do it faster and tell me I should be
looking into ILM for that. Also, they cringe whenever I go and put a
PO in for another TB of data.
Well, since the 960's will expand to 64TB you've got plenty of grow
room.

For backup performance, if you don;t manage it why do you get dinged?
And how is it being done today? Volume level backups? Qtree level?
Any replication involved (snapmirror or snapvault)?

ILM is not a fix for any problem, period. It is a conglomerate of
products/programs/scripts that automate a business process. The
business process is the fix. Never lose sight of that and your battle
is assured.

Does your business have a process for long term retention? Data
destruction? E-mail compliance? Whatever it happens to be, ILM can
wrap some automation around it but the process and requirements have
to exist first. Your management needs an education and you are going
to have to do it. But first you need to educate yourself, and this
was a good start.

Take a look at some of the ILM procucts out there. See what they do.
You'll find ILM is just HSM rebranded with a few more bells and
whistles attached. Information Lifecycle Management, the name alone
should fire off some bells and sirens for you to deal with your
management.

The flip side is ILM is a great resume builder. However, I'd bet that
if you did due diligence, set your requirements, selected a vendor,
then got a quote, management would be happy to sign PO's for TB's of
storage for the next 2-3 years.

Don;t forget all the hidden costs of ILM; it's not just the software.
Tape, disk, switch ports, FC ports, whatever. Make sure you add it
all in.

~F

Thanks... well I get dinged because its my data thats increasing the
backup restore window times... and the backup guys say I have too much
data to be restored...

In any event, I understand your arguments.

One other question... IN our company, overall we must have over 100TB
of data (I just manage 4TB of that)... I cant help but imagine that
someone should be consolidating all of this... I will raise this to
management, but before I do so, do folks here think this is a big
problem just in our company or are others out there like this... ?

How many others have a similar situation?

Thanks
Jon.

Not sure I understand. Your business unit (you) creates only 4% of the
company's data and YOU are under the gun to create an ILM strategy? I
don't think you'll find many others in that situation.

Does the remaining 96% have an ILM strategy?

Also I can't help noticing a disconnect. In addition to ILM, your
management is asking for faster backups but you think consolidation
(that would probably lengthen backup times) is needed. What gives?

If I may make a suggestion try to learn more about ILM (BTW is NOT a
technology). The EMC site is a good place where to start, not because
they are necessarily the best, but they have just about every peace to
implement an ILM strategy, and a rich Web site.

We have grown quite chaotic and our growth has been ad-hoc with many
regional offices (typically near the major customers)... we acquired a
lot of small consulting shops - so thats why we have so many
distributed sites... Our dept is however the main IT shop in the
company... no the others do not have one yet as well...

we are learning as we go - but we are paying a lot of money on our
data... and I dont think the remote distrbuted data is helping... I am
sure I should first bring their data in centrally and I think I need to
suggest that first...

Regarding faster backups - this is where I am led to beleive ILM
couldve helped (yes as you say, it is not a product but best
practises)... I guess to refine my original question, should I propose
something like this: 1. consolidate and 2. buy a product that lets us
move data from the netapp filers to some cheaper storage for older data

Jon