Copieing 60.000 items from Windows Live Mail extreme performance degradation

Hello,

My Windows Live Mail file disk image has become full (4 gigabytes).

I am now in the process of copieing these files over to a new file disk image of 8 gigabytes.

(I am also playing the Starwars Soundtrack at the same and buffering is at 30.000 milliseconds, so don't think that has much to do with it).

60.000 items like eml and nws files from windows live mail are being copied from

drive P: to drive W:

(I set the new filedisk image/file system to gpt/ntfs and 512 bytes per sector.)

What I am noticing during this copy process is the following:

As the number of items remaining goes down from 60.000 to 50.000 the speed is constantly dropping.

It started at 5 megabyte/secs or even 15 megabytes/sec... and now it's dropping
to 4 mb/sec 3.9, 3.8, 3.7, 3.6, 3.5.... and so forth... every 100 files or so, the copieing speed drops with 100 kilobytes/sec which is quite severe !

It's now down to 1.90 megabytes/sec and continues to drop !

I am now starting to understand why windows live mail was so slow starting all these weeks/months.

It's apperently a very serious issue with NTFS itself... storing all these files into one or two folders makes NTFS very slow...

Probably because it has to find the file's entries somewhere.

Now I am not sure about my hypothesis above.

I have some further information:

The 2 terrabyte C: drive was defragged very recently, up to 5 passes.

So in all likelyhood to new 8 gigabytes should be pretty consolidated, big large chucnks... very little file fragmentation.

So the question is why is the speed so low and also why is the speed dropping ?

There are two "processes" at work:

"reading"
"writing"

At first I was also considering the possibility that it's perhaps a disk/platter issue... perhaps the cylinder/track it's on is slowing down... but I don't think it would be this extreme... so I'd think I can rule that out.

It has something to do with the file system itself.

So there are two possibilities:

1. Either the reading process is slowing down because the newer files are more fragmented.

This is a valid hypothesis, which could be de-bunked by first defragging the 4 gb "drive" and then repeating the copy process to a new 8 gb drive to see if the same performance degradation occurs.

If it does occur then I will consider it proven that NTFS slows down in a big way... the more files there are in the folder.

For now MY NEW HOPE lol... is that it is a file fragmentation issue on the reading drive... and not a file system issue on the writing drive/NTFS in general.

Otherwise this would be a somewhat sad/untested design of Windows Live Mail..

I highly recommend future e-mailing/newsgrouping software is tested with synthetic generation of millions of files to test what performance will be after years of usage and adjusting storage structures accordingly to handle such large ammount of files performance-wise efficiently if possible.

Maybe if I am lucky it was reading drive fragmentation and the new drive will performance better.

Further notes of importance: the defrag tool didn't really show that much file fragmentation on this P/reading drive if I remember correctly sub 20%.... maybe even a few percent.

I may report back later with my findings.

Bye for now,
and me the force be with you,
always ! =D
Skybuck Flying, with a not so flying copieing process, will still take 25 minutes... total time probably 1 hour worth it though.

Hello,

My Windows Live Mail file disk image has become full (4 gigabytes).

I am now in the process of copieing these files over to a new file disk image of 8 gigabytes.

(I am also playing the Starwars Soundtrack at the same and buffering is at 30.000 milliseconds, so don't think that has much to do with it).

60.000 items like eml and nws files from windows live mail are being copied from

drive P: to drive W:

(I set the new filedisk image/file system to gpt/ntfs and 512 bytes per sector.)

What I am noticing during this copy process is the following:

As the number of items remaining goes down from 60.000 to 50.000 the speed is constantly dropping.

It started at 5 megabyte/secs or even 15 megabytes/sec... and now it's dropping
to 4 mb/sec 3.9, 3.8, 3.7, 3.6, 3.5.... and so forth... every 100 files or so, the copieing speed drops with 100 kilobytes/sec which is quite severe !

It's now down to 1.90 megabytes/sec and continues to drop !

I am now starting to understand why windows live mail was so slow starting all these weeks/months.

It's apperently a very serious issue with NTFS itself... storing all these files into one or two folders makes NTFS very slow...

Probably because it has to find the file's entries somewhere.

Now I am not sure about my hypothesis above.

I have some further information:

The 2 terrabyte C: drive was defragged very recently, up to 5 passes.

So in all likelyhood to new 8 gigabytes should be pretty consolidated, big large chucnks... very little file fragmentation.

So the question is why is the speed so low and also why is the speed dropping ?

There are two "processes" at work:

"reading"
"writing"

At first I was also considering the possibility that it's perhaps a disk/platter issue... perhaps the cylinder/track it's on is slowing down... but I don't think it would be this extreme... so I'd think I can rule that out.

It has something to do with the file system itself.

So there are two possibilities:

1. Either the reading process is slowing down because the newer files are more fragmented.

This is a valid hypothesis, which could be de-bunked by first defragging the 4 gb "drive" and then repeating the copy process to a new 8 gb drive to see if the same performance degradation occurs.

If it does occur then I will consider it proven that NTFS slows down in a big way... the more files there are in the folder.

For now MY NEW HOPE lol... is that it is a file fragmentation issue on the reading drive... and not a file system issue on the writing drive/NTFS in general.

Otherwise this would be a somewhat sad/untested design of Windows Live Mail..

I highly recommend future e-mailing/newsgrouping software is tested with synthetic generation of millions of files to test what performance will be after years of usage and adjusting storage structures accordingly to handle such large ammount of files performance-wise efficiently if possible.

Maybe if I am lucky it was reading drive fragmentation and the new drive will performance better.

Further notes of importance: the defrag tool didn't really show that much file fragmentation on this P/reading drive if I remember correctly sub 20%.... maybe even a few percent.

I may report back later with my findings.

Bye for now,
and me the force be with you,
always ! =D
Skybuck Flying, with a not so flying copieing process, will still take 25 minutes... total time probably 1 hour worth it though.

More testing/defragging/copieing will be done to shed some more light on this in coming days, stay tuned.