Why is this folder so slow?

I have a folder on one of my SSD drives that takes 8 to 10 hours to back
up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.

Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this
inefficient for small files like this?

Yousuf Khan

Yousuf Khan wrote:

I have a folder on one of my SSD drives that takes 8 to 10 hours to back
up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.

Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this
inefficient for small files like this?

Using WHAT backup software? Doing a file-based or image-based backup?

Is it a direct access to the folder, or are you using a redirection,
like a junction (reparse point)? Does that folder itself have any
redirections which could run the backup program into a loop if it
doesn't specifically ignore those?

Yousuf Khan wrote:
I have a folder on one of my SSD drives that takes 8 to 10 hours to back
up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.

Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this
inefficient for small files like this?

Yousuf Khan

Have you tried to "defragment" the drive ?

Normally, the "optimize" dialog will not offer defragmentation
as an option in Windows 10. It's supposed to offer "TRIM" as
the option for an SSD.

However, there is a "Copy On Write" or COW issue with SSDs.
Under the right circumstances, there will be a slowdown.

Now, consider what you're doing. Your backup software uses VSS
to make a shadow copy. It's possible some "COW activity" is happening
during the backup.

The Optimize dialog knows about this, and the Optimize dialog
has some sort of metric it uses to decide what to do. While
most of the time, it will only offer TRIM, I bet in your
case, it's "going to have a COW" and defragment your drive.
This should not be as thorough as a regular defragment,
and the design of what's done, should have something to do
with whatever the root cause of "having a COW" is.

I've not seen this slow behavior here, so have no
first hand experiences to offer on it. Note that over the
years Windows 10 has existed, the behavior of the Optimize
panel has been "as crazy as Cocoa Puffs". The software
frequently could not properly tell an HDD from an SSD,
and it would be damn hard to see any "subtle" behaviors,
when this software has had so many bugs in the past. I've
had a machine full of HDDs offer nothing but TRIM and
the Optimize panel declared all my drives as SSD drives.
Which is total bull**** and most annoying when you
actually want the defrag to work. As far as I can remember,
Optimize is working in 1909 OK now. It's been a hell of a
bumpy ride though, over the years.

See if you're offered a defrag option.

Do Properties on the drive letter, and in the Tools tab
you'll find the Optimize. Then retest your backup rate
after the partition has been cleaned up.

Paul

VanguardLH wrote in :

Yousuf Khan wrote:

I have a folder on one of my SSD drives that takes 8 to 10 hours to
back up. It is only about 1.4 GB, but it is allocated 2.4 GB of space
altogether, and there are 580,000 files here. Indicates that per file
it's using up a little bit over half of a cluster on average. File
system is NTFS.

Meanwhile, this same drive can backup the remainder of the drive in
under 2 hours, and the remainder of the drive is 390 GB! Is NTFS this
inefficient for small files like this?

Using WHAT backup software? Doing a file-based or image-based backup?

Is it a direct access to the folder, or are you using a redirection,
like a junction (reparse point)? Does that folder itself have any
redirections which could run the backup program into a loop if it
doesn't specifically ignore those?

That may be it. I remember reading about junctions causing havoc if they
were in a backup scheme (I think in a folder/file backup). Amazingly,
there was some helpful information (for me, at the time) on a Microsoft
forum, about identifying junctions, found in paragraph two of darrenc1's
answer.

To the OP:

https://answers.microsoft.com/en-us/...rum/windows_7-
performance/what-is-a-reparse-point-can-anyone-reveal-the/17b9b457-6c8a-
4e83-a445-e603011a6b95

or

https://tinyurl.com/y8hssmg6

On 4/26/2020 9:32 PM, VanguardLH wrote:
Using WHAT backup software? Doing a file-based or image-based backup?

Macrium, file-based.

Is it a direct access to the folder, or are you using a redirection,
like a junction (reparse point)? Does that folder itself have any
redirections which could run the backup program into a loop if it
doesn't specifically ignore those?

No, none of that. Straightforward unredirected.

Yousuf Khan

On 4/26/2020 9:46 PM, Paul wrote:
Have you tried to "defragment" the drive ?

No, considering it's an SSD. But as you pointed out later, the optimize
option is available for both of my SSD's, but optimize recognizes them
as SSD's, so the only optimization available is trimming, no defragging.

Normally, the "optimize" dialog will not offer defragmentation
as an option in Windows 10. It's supposed to offer "TRIM" as
the option for an SSD.

However, there is a "Copy On Write" or COW issue with SSDs.
Under the right circumstances, there will be a slowdown.

Yes, likely this is exactly that circumstance. Do you know what the
symptoms of that circumstance are?

Now, consider what you're doing. Your backup software uses VSS
to make a shadow copy. It's possible some "COW activity" is happening
during the backup.

Yes, VSS is used by the software, which is Macrium Reflect 6 BTW.
Reflect's logs show that it creates the VSS shadows immediately before
beginning the backup.

This backup runs after midnight, and there is little activity while any
of the backups run. All of the backups run after midnight and they
finish relatively quickly, except this one.

The Optimize dialog knows about this, and the Optimize dialog
has some sort of metric it uses to decide what to do. While
most of the time, it will only offer TRIM, I bet in your
case, it's "going to have a COW" and defragment your drive.
This should not be as thorough as a regular defragment,
and the design of what's done, should have something to do
with whatever the root cause of "having a COW" is.

VSS is used on all of the backup jobs. None of the others exhibit this
behaviour. In fact, I've experienced this issue for nearly a decade now.
The problem started on Windows XP, continued on into Windows 7, and
continues to plague me in Windows 10. This particular folder has also
been migrated around from HDD to SSD, to a 2nd SSD, etc. So it's not a
problem that is specific to HDD's or SSD's, or to any particular version
of Windows.

I'll tell you what this folder is. It's actually my Thunderbird News
folder (exactly what I'm using to ask this question here), which exists
under the my User folder structure. The problem was discovered when I
started doing daily backups of my User folder and discovered that the
User folder was taking forever. After investigating it some, I figured
out that the problem was this particular substructure under News. Once I
excluded the News folder, backups finished 6 times faster! So I moved
the backups of the News folder to their own job, and let the rest of the
User folder get backed up separately. Before, you ask, I only backup the
News folder once a week, but it's still a pain in the ass watching it
take so long even once a week.

Some other background. When this particular backup is happening, it's
not the drives that are showing as busy, it's the CPU cores! 4 out of
the 8 cores on my FX-8300 are fluctuating between 50% to 100% busy,
while the other 4 are not that busy.

Yousuf Khan

Yousuf Khan wrote:


Some other background. When this particular backup is happening, it's
not the drives that are showing as busy, it's the CPU cores! 4 out of
the 8 cores on my FX-8300 are fluctuating between 50% to 100% busy,
while the other 4 are not that busy.

Yousuf Khan

I seem to remember at some time in the past, you offered
advice on putting an exception for an AV program,
so it does not scan that particular directory
(something in Thunderbird).

If your CPU cores are railed, I'd be tracing down the
PID of the offender.

One way to do it on a Pro SKU of OS, is

tasklist /svc # should not work on Home

and that will tell you what is inside a SVCHOST. You
can also do that with Process Explorer from Sysinternals,
running concurrently with Task Manager, and flip over
to Process Explorer to see what is in a busy PID in
Task Manager. If you elevate Process Explorer using
"Run as Administrator", it can even take a stack snapshot
of a SVCHOST, and you can get additional information.

For example, I have a SVCHOST with 15 things in it,
and one is wuauserv. If a Windows Update scan is running,
that SVCHOST lights up -- but then you have to guess
that's the guilty service, as the rest of the services
aren't normally a problem.

When Macrium is running, CPU effort goes into two things:

1) Running a checksum to stamp the .mrimg when finished.
This detects corruption later (like when restoring perhaps).

2) Compression. If the lightweight compressor is turned on,
that will use a core. I don't think Macrium uses multi-core
for its compressor.

If you were seeing more than that, I'd be looking at
MsMpEng as a culprit, as it could cause quite a penalty
if every small file involved a scan by the Windows Defender.

When I ran hashdeep64 in Windows 10, I think the calc
ran 8x slower than normal, to give some idea what a
penalty Windows Defender causes on reads.

Paul

On 4/27/2020 2:06 AM, Yousuf Khan wrote:
On 4/26/2020 9:46 PM, Paul wrote:
Have you tried to "defragment" the drive ?

No, considering it's an SSD. But as you pointed out later, the optimize
option is available for both of my SSD's, but optimize recognizes them
as SSD's, so the only optimization available is trimming, no defragging.

Normally, the "optimize" dialog will not offer defragmentation
as an option in Windows 10. It's supposed to offer "TRIM" as
the option for an SSD.

However, there is a "Copy On Write" or COW issue with SSDs.
Under the right circumstances, there will be a slowdown.

Yes, likely this is exactly that circumstance. Do you know what the
symptoms of that circumstance are?

Now, consider what you're doing. Your backup software uses VSS
to make a shadow copy. It's possible some "COW activity" is happening
during the backup.

Yes, VSS is used by the software, which is Macrium Reflect 6 BTW.
Reflect's logs show that it creates the VSS shadows immediately before
beginning the backup.

This backup runs after midnight, and there is little activity while any
of the backups run. All of the backups run after midnight and they
finish relatively quickly, except this one.

The Optimize dialog knows about this, and the Optimize dialog
has some sort of metric it uses to decide what to do. While
most of the time, it will only offer TRIM, I bet in your
case, it's "going to have a COW" and defragment your drive.
This should not be as thorough as a regular defragment,
and the design of what's done, should have something to do
with whatever the root cause of "having a COW" is.

VSS is used on all of the backup jobs. None of the others exhibit this
behaviour. In fact, I've experienced this issue for nearly a decade now.
The problem started on Windows XP, continued on into Windows 7, and
continues to plague me in Windows 10. This particular folder has also
been migrated around from HDD to SSD, to a 2nd SSD, etc. So it's not a
problem that is specific to HDD's or SSD's, or to any particular version
of Windows.

I'll tell you what this folder is. It's actually my Thunderbird News
folder (exactly what I'm using to ask this question here), which exists
under the my User folder structure. The problem was discovered when I
started doing daily backups of my User folder and discovered that the
User folder was taking forever. After investigating it some, I figured
out that the problem was this particular substructure under News. Once I
excluded the News folder, backups finished 6 times faster! So I moved
the backups of the News folder to their own job, and let the rest of the
User folder get backed up separately. Before, you ask, I only backup the
News folder once a week, but it's still a pain in the ass watching it
take so long even once a week.

Some other background. When this particular backup is happening, it's
not the drives that are showing as busy, it's the CPU cores! 4 out of
the 8 cores on my FX-8300 are fluctuating between 50% to 100% busy,
while the other 4 are not that busy.

Â*Â*Â*Â*Yousuf Khan
Not really sure, but I think TB does compression on its files. If you
don't allow it, that might be the cause.

On 4/27/2020 8:28 AM, Todesco wrote:
Not really sure, but I think TB does compression on its files.Â* If you
don't allow it, that might be the cause.

It does that only when it's active and running, in this case it's not
running. Also it doesn't compress newsgroup files, just email files.

On 4/27/2020 3:57 AM, Paul wrote:
I seem to remember at some time in the past, you offered
advice on putting an exception for an AV program,
so it does not scan that particular directory
(something in Thunderbird).

If your CPU cores are railed, I'd be tracing down the
PID of the offender.

One way to do it on a Pro SKU of OS, is

Â*Â* tasklist /svcÂ*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â*Â* # should not work on Home

Not even necessary, I can tell you right now which process is
responsible, it's the Macrium Reflect binary. Also the System process
which I assume the Reflect binary also makes heavy use of during this time.