A computer components & hardware forum. HardwareBanter

If this is your first visit, be sure to check out the FAQ by clicking the link above. You may have to register before you can post: click the register link above to proceed. To start viewing messages, select the forum that you want to visit from the selection below.

Go Back   Home » HardwareBanter forum » General Hardware & Peripherals » Storage (alternative)
Site Map Home Register Authors List Search Today's Posts Mark Forums Read Web Partners

Why is this folder so slow?



 
 
Thread Tools Display Modes
  #21  
Old April 27th 20, 11:16 PM posted to alt.comp.os.windows-10,comp.sys.ibm.pc.hardware.storage
T[_6_]
external usenet poster
 
Posts: 49
Default Why is this folder so slow?

On 2020-04-27 07:30, Ken Blake wrote:
SinceÂ*theÂ*folderÂ*isÂ*onÂ*anÂ*SSD,Â*fragmentatio nÂ*shouldn'tÂ*makeÂ*anyÂ*difference.


And you will reduce your wear life doing a defragment

  #22  
Old April 28th 20, 04:46 AM posted to alt.comp.os.windows-10,comp.sys.ibm.pc.hardware.storage
Paul[_28_]
external usenet poster
 
Posts: 1,467
Default Why is this folder so slow?

Yousuf Khan wrote:
On 4/27/2020 12:04 PM, Frank Slootweg wrote:
If there are 580,000 files in the News folder, then you've probably
configured your Thunderbird News account(s) to use one file for each
article instead of one file for each newsgroup.

If so, it's probably best to bite the bullet and convert to one file
per newsgroup. That probably needs an export and (re-)import and
probably will be time-consuming, but at least then you'll solve the
actual problem.

FYI, my setup - not Thunderbird - has nearly a million articles, but
only some 600 files.


Yes, that is exactly the problem, I was getting at. Does Thunderbird
have a new news file format available? My assumption was that
Thunderbird only does 1 file/message? What's the option to convert?

Yousuf Khan


These are examples of the setting in the Config Editor.

Note that the GUI "Server Settings" page has the choice grayed
out once the tool is running, implying these can't be switched
on the fly from the GUI.

And changing them here, doesn't mean a "converter" is going to run,
because the tool isn't going to know the "before" and "after"
and figure out what needs to be done, or whether it should
even be doing it.

mail.server.server1.storeContractID = @mozilla.org/msgstore/maildirstore;1

mail.server.server4.storeContractID = @mozilla.org/msgstore/berkeleystore;1

You could try some sort of Import/Export strategy, pulling from
an EML format setup, into an MBOX format setup.

Berkeleystore, as far as I know, is the "file per box" method.
The so-called Mork Storage Format, of which there is
a rudimentary parser available.

Maildirstore, is a file per message method, like an EML at a guess.
You can see this better than I can, as mine are all
going to be Berkeleystore.

Picture of Version 45 or so. Option not available/implemented
before Version 38.

https://i.postimg.cc/Jh7qmhzN/TBird-stores-option.gif

Paul
  #23  
Old April 28th 20, 05:02 AM posted to alt.comp.os.windows-10,comp.sys.ibm.pc.hardware.storage
Paul[_28_]
external usenet poster
 
Posts: 1,467
Default Why is this folder so slow?

T wrote:
On 2020-04-27 07:30, Ken Blake wrote:
Since the folder is on an SSD, fragmentation shouldn't make any difference.


And you will reduce your wear life doing a defragment


One way to do this, is with a Macrium backup and restore,
where you use the forward and back button, go back and
"edit" the size of the destination directory. This
causes the restoration to change restore mode, and it
seems to do a file-by-file write when challenged with
even an insignificant file system size change. You don't
have to "pinch it", and in fact pinching it is not
recommended. Like, make the partition 1MB smaller,
should be enough to trigger file-by-file write mode.

This results in a "mostly defragmented" disk. Due to the
handling of the $MFT and the reserved space for $MFT,
there is some "friction fragmentation" as the reserved
space gets squeezed. I can see a little bit of
fluff that doesn't get fixed. The end result of
changing the partition size on the Macrium restore,
is a mostly defragmented partition.

To save on your SSD while doing experiments like this,
you can test on hard drives, and evaluate using nfi.exe
(mentioned in the thread already).

This approach also places an upper bound on the number
of writes and the amount of flash life you're paying
for the privilege. The Windows 10 Defragmenter is pretty
good, but some of the other defragmenters out there,
they can run all night, and that can't be good for an
SSD.

https://i.postimg.cc/Y0TCt8K5/macrium-as-defragger.gif

I noticed that "side effect" one day after a restore,
where I'd changed the destination partition size. I couldn't
believe what I was seeing. Doing that to the 1.4TB
partition in that picture was mostly a joke, as on
a data partition, you don't really need to do that.
I wanted to see whether it would change the symptoms
of another bug I'm working on (and it didn't help).

Paul
  #24  
Old April 28th 20, 06:40 AM posted to alt.comp.os.windows-10,comp.sys.ibm.pc.hardware.storage
VanguardLH[_2_]
external usenet poster
 
Posts: 1,453
Default Why is this folder so slow?

Yousuf Khan wrote:

On 4/27/2020 2:29 PM, VanguardLH wrote:
As a test, disable your anti-virus software and run your TB data-only
backup job.


Yes, that's been done years ago too. This folder has been a major
headache for years now. And at one time, I found that the AV software
spending tons of time scanning this folder too, so I put an exclusion in
it for this folder. The AV doesn't ever scan in this folder anymore.


I just thought of something else: is that flagged as a special folder?
Right-click on the folder, and select Properties. Is there a Customize
tab? If so, select it, and check the setting for "Optimize this folder
for". Set to "General items" (instead of "Pictures").

It's not related to VSS, I've already given you the most likely cause of
the problem: there are over half million files, and each file is
inefficiently taking up little over half of the NTFS cluster, rather
than spreading a lesser number of files over many clusters. The real
question is how can we make NTFS more efficient at handling all of these
little files? NTFS is great at handling big files, but tiny little files
no so much.


Slack space is also a problem with FAT16/32, ext, or other file systems
where AUs (Allocation Units) are clusters or groups of sectors. The
file system will allocate a number of clusters that will encompass the
size of the file, but will be equal to or larger than the file's
content. Slack space is *not* just an NTFS problem.

For NTFS, files under the size for an MFT's file record are stored
inside the MFT since there is already enough space to hold the file
there. Instead of the MFT file record having a pointer to the small
file outside the MFT where there would be a lot of slack space (the
small file is nowhere the size of a cluster), the MFT file record *is*
the file.

An MFT file is 1 KB in size. If the file is smaller than that, the file
is stored in the MFT record. Actually, because the MFT file record has
a fixed 42-byte table at its start and holds file name and system
attributes.

https://hetmanrecovery.com/recovery_...ucture.htm#id4
According to specifications, MFT record size is determined by the
value of a variable in the boot sector. In practical terms, all
current versions of Microsoft Windows are using records sized 1024
bytes. The first 42 bytes store the header. The header contains 12
fields. The other 982 bytes do not have a fixed structure, and are
used to keep attributes.

The MFT is not infinite in size. NTFS has a limit of 4,294,967,295
files per disk (well, per volume). Your 580,000 files is only 0.01% of
NTFS' capacity for file count. Obviously there are lots of files
elsewhere in that volume.

NTFS doesn't have a problem between small and large files regarding
addressing them. It's the level of fragmentation that cause a problem.
Yeah, you think you don't need to defragment and should not defragment
an SSD because, after all, accessing memory at one address is the same
speed as accessing other memory. However, NTFS cannot support an
infinite chain of fragments for a file. Each fragment consumes an
extended file record in the MFT (a record outside the MFT). There are
limitations in every file system. Around 1.5 million fragments is the
limit per file under NTFS.

Doesn't Thunderbird have a compaction function? Used it yet? I don't
know if that will eliminate any fragmention of the files used to store
the messages or articles which, as I recall, are stored as seperate
files instead of inside a database, but I haven't used TB in a long
time.

Users don't think they ever need to defragment an SSD. All those extra
writes with no effective change in data content reduces the lifespan of
the SSD (writes are destructive). Sure, when there are few or dozen
fragments then the extra writes to defragment are wasting the SSD. It
takes time to chain from the MFT's record and through every external
extended record (which consumes space in the file system) to build up
the entire file. It's not one lookup in the MFT for the file. It's a
chained lookup for every fragment. IOPS will increase as fragmentation
increases, and perhaps why you are seeing high CPU usage when backing up
those files. Most users think of fragmentation as a performance issue
with moving physical media, like hard disks. Fragmentation ON ANY MEDIA
is still an I/O overhead issue and inflates the IOPS to process them
all. Yes, there is a limit in NTFS to the number of fragments that a
file may have, but the more fragments there are the more space is
consumed in the file system to track those fragments and the more CPU
consumed to process the fragments. When an OS sees a file comprised of
multiple fragments, there are more multiple I/O operations to process
the whole file. If Windows see 20 pieces at the logical layer, there
are 20 I/O operations to process the whole file as a read or write.

Fragmentation is not just a performance issue at the physical layer. It
is also a performance factor at the logical layer (file system).
Extreme fragmentation requires lots of repeated writes to a file. I
don't know what you've been doing with those files in the problematic
folder. If they are photos, you rarely edit those, just copy them.

Similarly, for a backup job, it has to perform the IOPS'es needed to
read all the files included in the backup. I have under 400,000 files
on my entire OS+app drive (which is a partition spanning the entire
SSD). You have more than that in one folder. From your description,
the backup job is CPU bound with all those IOPS. Do you really have
over 500K files in just one folder? You never considered creating a
hierarchical structure of subfolders to hold groups of those files based
on a common criteria for each subfolder? Just because you can dump
hundreds of thousands of files into a single folder doesn't mean that's
a good behavior.

By the say, in Macrium Reflect, did you configure your backup job to
throttle its use of the CPU? That's to prevent a backup job from
sucking up all the CPU while preventing the computer being usable to the
user during the backup. In a Reflect backup job, you can configure its
priority. Well, if you set it at max (which is still, I believe, less
than real-time priority), that process sucks up most of the CPU and
leave little for use by other processes making the computer unusable to
you. Even if you schedule the backup to run when you're not at the
computer, other backgrounded processes, like your startup programs, and
even the OS want some CPU slices.

The compression you select for a backup job also dictates how much it
consumes the CPU. You will find very little difference in the size of
the backup file between Medium (recommended) and High compression
levels. The backup job will take a lot longer trying to compress more
the backup file, but the result is little improvement in reduction of
the backup file, especially for non-compressible file formats, like
images, but wastes a lot of CPU time for insignificant gain.

I did not find an option in Reflect to throttle how much bandwidth it
uses on the data bus, like a limit on IOPS. Not for network traffic,
but how busy it keeps the data bus. If it is flooded, and especially
for a high[er] priority process, you have to wait to do any other data
I/O.
  #25  
Old April 28th 20, 04:13 PM posted to alt.comp.os.windows-10,comp.sys.ibm.pc.hardware.storage
Frank Slootweg
external usenet poster
 
Posts: 46
Default Why is this folder so slow?

Yousuf Khan wrote:
On 4/27/2020 12:04 PM, Frank Slootweg wrote:
If there are 580,000 files in the News folder, then you've probably
configured your Thunderbird News account(s) to use one file for each
article instead of one file for each newsgroup.

If so, it's probably best to bite the bullet and convert to one file
per newsgroup. That probably needs an export and (re-)import and
probably will be time-consuming, but at least then you'll solve the
actual problem.

FYI, my setup - not Thunderbird - has nearly a million articles, but
only some 600 files.


Yes, that is exactly the problem, I was getting at. Does Thunderbird
have a new news file format available? My assumption was that
Thunderbird only does 1 file/message? What's the option to convert?


It's not a new News file format, it's a different format.

You set the format in the News account: Tools - Account settings -
your news account in the left pane - 'Server Settings' page -
Message Storage - Message Store Type:. This field *should* be set to
'File per folder (mbox)'. Yours is probably set to 'File per message
(maildir)'.

For your account - i.e. an *existing* account - you probably cannot
change this setting, i.e. you can only set it when you create the
account. Hence my comment about exporting and (re-)importing. If you
cannot change the setting, you will have to export all the articles from
your current account and then re-import all articles into a new account
with 'Message Store Type: File per folder (mbox)'.

The basic Thunderbird program has no export facility and only very
limited import functionality.

For import of e-mail (from Windows Mail), I have used the Thunderbird
ImportExportTools [1] Extension, but I have not used it for News and not
for export.

ImportExportTools can export on a per-folder basis, so you could try
to export just one folder/newsgroup and then import it into a new
account to see if it works for News. Exporting is a copy-type operation,
i.e. the source remains untouched, and if you import to a *new* account,
the old account remains untouched. IOW, it's a totally safe operation.

If ImportExportTools can not solve your problem, you'll probably have
to search the Thunderbird support site(s)/forum(s) or/and post there.

[1]
https://addons.thunderbird.net/en-GB/thunderbird/addon/importexporttools/?src=userprofile
  #26  
Old April 28th 20, 08:14 PM posted to alt.comp.os.windows-10,comp.sys.ibm.pc.hardware.storage
Paul[_28_]
external usenet poster
 
Posts: 1,467
Default Why is this folder so slow?

Frank Slootweg wrote:
Yousuf Khan wrote:
On 4/27/2020 12:04 PM, Frank Slootweg wrote:
If there are 580,000 files in the News folder, then you've probably
configured your Thunderbird News account(s) to use one file for each
article instead of one file for each newsgroup.

If so, it's probably best to bite the bullet and convert to one file
per newsgroup. That probably needs an export and (re-)import and
probably will be time-consuming, but at least then you'll solve the
actual problem.

FYI, my setup - not Thunderbird - has nearly a million articles, but
only some 600 files.

Yes, that is exactly the problem, I was getting at. Does Thunderbird
have a new news file format available? My assumption was that
Thunderbird only does 1 file/message? What's the option to convert?


It's not a new News file format, it's a different format.

You set the format in the News account: Tools - Account settings -
your news account in the left pane - 'Server Settings' page -
Message Storage - Message Store Type:. This field *should* be set to
'File per folder (mbox)'. Yours is probably set to 'File per message
(maildir)'.

For your account - i.e. an *existing* account - you probably cannot
change this setting, i.e. you can only set it when you create the
account. Hence my comment about exporting and (re-)importing. If you
cannot change the setting, you will have to export all the articles from
your current account and then re-import all articles into a new account
with 'Message Store Type: File per folder (mbox)'.

The basic Thunderbird program has no export facility and only very
limited import functionality.

For import of e-mail (from Windows Mail), I have used the Thunderbird
ImportExportTools [1] Extension, but I have not used it for News and not
for export.

ImportExportTools can export on a per-folder basis, so you could try
to export just one folder/newsgroup and then import it into a new
account to see if it works for News. Exporting is a copy-type operation,
i.e. the source remains untouched, and if you import to a *new* account,
the old account remains untouched. IOW, it's a totally safe operation.

If ImportExportTools can not solve your problem, you'll probably have
to search the Thunderbird support site(s)/forum(s) or/and post there.

[1]
https://addons.thunderbird.net/en-GB/thunderbird/addon/importexporttools/?src=userprofile


The one I was looking at the other day, said that it didn't
handle stuff in the News folder specifically.

As for the availability of the MailboxStore option in the
Server settings, the claim is that you must use this
immediately when the installation of Thunderbird is
brand new. In my experiments yesterday, I tried to "clean out"
my profile, and tried not to leave any .msf files, then
set the prefs.js with the maildirstore preference, and
that *still* wasn't enough to make it work. I'm going
to have to nuke the damn thing and start from scratch,
to see if I can get it to work.

One other weirdness from yesterdays experiment, is after
I was finished with my failed experiment, I took the ZIP
file holding my unbroken profile, and started to restore
it to my SSD drive. I was greeted by write rates of arounf
2MB/sec on my SSD. It took forever to restore the fleet
of .msf (file per box) style files. And when I opened
Task Manager, MsMpEng was railed on one core, scanning
everything being written into the profile area. I've done
plenty of other stuff on the computer, where it doesn't
do that with quite the same level of venom. (If I unpack
an .ova on a scratch drive, it does that at several hundred
megabytes per second. As if MsMpEng didn't care.)

Paul
  #27  
Old April 28th 20, 10:08 PM posted to alt.comp.os.windows-10,comp.sys.ibm.pc.hardware.storage
Frank Slootweg
external usenet poster
 
Posts: 46
Default Why is this folder so slow?

Paul wrote:
[...]

As for the availability of the MailboxStore option in the
Server settings, the claim is that you must use this
immediately when the installation of Thunderbird is
brand new.


I think that's not correct. The *installation* doesn't have to be
brand new, the *account* in Thunderbird must be new, i.e. just created.

I added a new New account and could set 'Message Store Type:' to
either 'File per folder (mbox)' or 'File per message (maildir)'.

In my experiments yesterday, I tried to "clean out"
my profile, and tried not to leave any .msf files, then
set the prefs.js with the maildirstore preference, and
that *still* wasn't enough to make it work. I'm going
to have to nuke the damn thing and start from scratch,
to see if I can get it to work.


There's no need to fiddle with preferences as there are perfectly good
settings in the GUI. I think the fiddling might actually have been
counter-productive, because you might have been setting a global default
instead of a account-specific setting.

[...]
  #28  
Old May 1st 20, 07:57 AM posted to alt.comp.os.windows-10,comp.sys.ibm.pc.hardware.storage
Yousuf Khan[_2_]
external usenet poster
 
Posts: 1,296
Default Why is this folder so slow?

On 4/27/2020 2:09 PM, Paul wrote:
OK, show me a chunk of nfi.exe output, just
for files in the magical folder. Just enough
to capture the essence of what's going on.


What I'm looking for is filesystem tuning advice, not process tuning
advice. Looking at CPU processes is just a wild goose-chase. Yes, the
CPU is being hammered, but we know which processes are responsible and
why, it's hardly surprising which ones they are (i.e. Macrium Reflect
binaries), so it's trivial.

Anyways, since I'm not getting that advice here, as it turned out, I
just received a new SSD (as an RMA of a previous SSD, which has already
been replaced). I decided to try a few tests myself. I set up the new
SSD as the Z drive, and I formatted it into non-default NTFS settings.
This SSD is usually default formatted to 4K blocks, I tested it out by
using 0.5K and 1K blocks instead. I then restored a previous backup of
the filesystem to this drive, and tested out the backup and restore
performance. Since this is not the production drive, it's not being
accessed by any other processes like Thunderbird, so it's pristine and
not a busy drive.

I found that with both 0.5K and 1K blocks, the restore operation went
very fast, about half an hour to get fully restored, which is a big
improvement (previously used to take 1.5 hours to restore). Also the
allocation slack was greatly improved, went from 1.4GB stored and 2.4GB
allocated (42% slack), to 1.4GB stored and only 1.5GB allocated (7%
slack). However, then I tried backing up the new drive and it still took
over 8 hours! So writing to the drive is getting very fast, but reading
off of it is still slow. Still the same number of files as before, over
half-a-million.

Yousuf Khan
  #29  
Old May 1st 20, 08:15 AM posted to alt.comp.os.windows-10,comp.sys.ibm.pc.hardware.storage
Yousuf Khan[_2_]
external usenet poster
 
Posts: 1,296
Default Why is this folder so slow?

On 4/28/2020 11:13 AM, Frank Slootweg wrote:
Yousuf Khan wrote:
Yes, that is exactly the problem, I was getting at. Does Thunderbird
have a new news file format available? My assumption was that
Thunderbird only does 1 file/message? What's the option to convert?


It's not a new News file format, it's a different format.


New to me. LOL ;-)

You set the format in the News account: Tools - Account settings -
your news account in the left pane - 'Server Settings' page -
Message Storage - Message Store Type:. This field *should* be set to
'File per folder (mbox)'. Yours is probably set to 'File per message
(maildir)'.


Actually it does show the "file per folder (mbox)" but it's completely
grey-out, unchangeable. I assume that that's just it's preferred method
of accessing News, but it's being forced to use the old format anyways.

For your account - i.e. an *existing* account - you probably cannot
change this setting, i.e. you can only set it when you create the
account. Hence my comment about exporting and (re-)importing. If you
cannot change the setting, you will have to export all the articles from
your current account and then re-import all articles into a new account
with 'Message Store Type: File per folder (mbox)'.


Actually, I'm ready to completely blow out all of the files in the news
folder, and redownload from scratch, just so long as my newsgroups list
remains untouched. I obviously have backups of it, so it's not going to
be harmful to me.

If ImportExportTools can not solve your problem, you'll probably have
to search the Thunderbird support site(s)/forum(s) or/and post there.

[1]
https://addons.thunderbird.net/en-GB/thunderbird/addon/importexporttools/?src=userprofile


Thanks, but it looks like this version only works up to Thunderbird
version 60, and I'm on version 68.x. No problem, this is just News, I'll
just wipe it out and redownload.

Yousuf Khan
  #30  
Old May 1st 20, 08:20 AM posted to alt.comp.os.windows-10,comp.sys.ibm.pc.hardware.storage
Yousuf Khan[_2_]
external usenet poster
 
Posts: 1,296
Default Why is this folder so slow?

On 4/28/2020 3:14 PM, Paul wrote:
The one I was looking at the other day, said that it didn't
handle stuff in the News folder specifically.

As for the availability of the MailboxStore option in the
Server settings, the claim is that you must use this
immediately when the installation of Thunderbird is
brand new. In my experiments yesterday, I tried to "clean out"
my profile, and tried not to leave any .msf files, then
set the prefs.js with the maildirstore preference, and
that *still* wasn't enough to make it work. I'm going
to have to nuke the damn thing and start from scratch,
to see if I can get it to work.

One other weirdness from yesterdays experiment, is after
I was finished with my failed experiment, I took the ZIP
file holding my unbroken profile, and started to restore
it to my SSD drive. I was greeted by write rates of arounf
2MB/sec on my SSD. It took forever to restore the fleet
of .msf (file per box) style files. And when I opened
Task Manager, MsMpEng was railed on one core, scanning
everything being written into the profile area. I've done
plenty of other stuff on the computer, where it doesn't
do that with quite the same level of venom. (If I unpack
an .ova on a scratch drive, it does that at several hundred
megabytes per second. As if MsMpEng didn't care.)

Â*Â* Paul


Oh, it's a good thing I kept reading the replies, as it looks like you
already tried what I was about to try. So it kept using the same file
format as before, even after nuking it and starting from scratch?
 




Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
DH57dd/core i3 slow boot, slow wake up cew Intel 2 May 20th 12 12:52 AM
p5n32-sli se deluxe and Vista x64 and 2gb (slow enough) then 4gb (horribly slow issues) markm75 Asus Motherboards 1 August 26th 07 01:51 PM
scan to folder Ricky Printers 5 April 29th 05 02:35 PM
ATI folder Rob Ati Videocards 3 January 25th 05 01:04 AM
Cannot remove folder sunksnook Storage (alternative) 0 September 15th 03 02:00 AM


All times are GMT +1. The time now is 10:58 AM.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004-2024 HardwareBanter.
The comments are property of their posters.