Why is this folder so slow?

On 4/28/2020 5:08 PM, Frank Slootweg wrote:
I think that's not correct. The*installation* doesn't have to be
brand new, the*account* in Thunderbird must be new, i.e. just created.

I added a new New account and could set 'Message Store Type:' to
either 'File per folder (mbox)' or 'File per message (maildir)'.

So what if I nuke all of the old messages in the News folder, and let it
repopulate from scratch?

Yousuf Khan

On 4/27/2020 6:15 PM, T wrote:
Hi Yousuf,

When I see things like this, it is usually a failing
drive, especially when the index on teh offending
directory never finishes.

This will show up like a soar thumb if yo run your
drive through gsmartcontrol: check the error logs and
run the self tests

Brand new drive, less than a month old, hasn't had a chance to get old yet.

Yousuf Khan

Yousuf Khan wrote:
On 4/28/2020 3:14 PM, Paul wrote:
The one I was looking at the other day, said that it didn't
handle stuff in the News folder specifically.

As for the availability of the MailboxStore option in the
Server settings, the claim is that you must use this
immediately when the installation of Thunderbird is
brand new. In my experiments yesterday, I tried to "clean out"
my profile, and tried not to leave any .msf files, then
set the prefs.js with the maildirstore preference, and
that *still* wasn't enough to make it work. I'm going
to have to nuke the damn thing and start from scratch,
to see if I can get it to work.

One other weirdness from yesterdays experiment, is after
I was finished with my failed experiment, I took the ZIP
file holding my unbroken profile, and started to restore
it to my SSD drive. I was greeted by write rates of arounf
2MB/sec on my SSD. It took forever to restore the fleet
of .msf (file per box) style files. And when I opened
Task Manager, MsMpEng was railed on one core, scanning
everything being written into the profile area. I've done
plenty of other stuff on the computer, where it doesn't
do that with quite the same level of venom. (If I unpack
an .ova on a scratch drive, it does that at several hundred
megabytes per second. As if MsMpEng didn't care.)

Paul

Oh, it's a good thing I kept reading the replies, as it looks like you
already tried what I was about to try. So it kept using the same file
format as before, even after nuking it and starting from scratch?

I would refrain from working in this direction.

Sure, if you have backed up the various folders for TBird
before trying it (like I did when testing), then great.
Just don't do it, without having something to restore from.

It's pretty weird for a function to be existing in TBird
and presumably to be absorbing test time from release to
release, and then be hobbling the usage of it with
inept controls.

If you pursue this line of reasoning, what will
happen is your headers will be stripped down to
the event horizon of the server (maybe six months
retention on a free server), and if you have
years of headers (where the MID won't fetch anything
if you click), those are the kinds of headers that
will disappear if you start over again. The headers
from ten years ago, aren't on the server, and cannot
be regenerated from a small server - messing around
will significantly damage your header history.

If the damn thing had a conversion function that
converted equally between the two formats, I might
have a different opinion about doing this. It's just
that this is a feature that doesn't appear finished.

Paul

On 5/1/2020 3:56 AM, Paul wrote:
Yousuf Khan wrote:
Oh, it's a good thing I kept reading the replies, as it looks like you
already tried what I was about to try. So it kept using the same file
format as before, even after nuking it and starting from scratch?

I would refrain from working in this direction.

Sure, if you have backed up the various folders for TBird
before trying it (like I did when testing), then great.
Just don't do it, without having something to restore from.

It's pretty weird for a function to be existing in TBird
and presumably to be absorbing test time from release to
release, and then be hobbling the usage of it with
inept controls.

I just took a chance, and deleted all of the old newsgroup folders, that
contained all of the old-style messages. Left all of the rest of the
files in that news server's base folder untouched. Then I started
Thunderbird up again. It re-downloaded the messages, and it only
downloaded from where I last left off. It's now filling the data files
known as *.msf (e.g. alt.comp.os.windows-10.msf) rather than filling the
folders! Interestingly, these *.msf files used to exist in this News
folder before, but they were just trivial 1K or 2K files, with nothing
substantial inside them. They are now substantial files now, ranging
from 44 KB to 41 MB. So it looks like having those old folders there all
of this time was preventing Thunderbird from using the new style *.msf
files, even though it had long ago created them!

Yousuf Khan

Yousuf Khan wrote:
On 5/1/2020 3:56 AM, Paul wrote:
Yousuf Khan wrote:
Oh, it's a good thing I kept reading the replies, as it looks like
you already tried what I was about to try. So it kept using the same
file format as before, even after nuking it and starting from scratch?

I would refrain from working in this direction.

Sure, if you have backed up the various folders for TBird
before trying it (like I did when testing), then great.
Just don't do it, without having something to restore from.

It's pretty weird for a function to be existing in TBird
and presumably to be absorbing test time from release to
release, and then be hobbling the usage of it with
inept controls.

I just took a chance, and deleted all of the old newsgroup folders, that
contained all of the old-style messages. Left all of the rest of the
files in that news server's base folder untouched. Then I started
Thunderbird up again. It re-downloaded the messages, and it only
downloaded from where I last left off. It's now filling the data files
known as *.msf (e.g. alt.comp.os.windows-10.msf) rather than filling the
folders! Interestingly, these *.msf files used to exist in this News
folder before, but they were just trivial 1K or 2K files, with nothing
substantial inside them. They are now substantial files now, ranging
from 44 KB to 41 MB. So it looks like having those old folders there all
of this time was preventing Thunderbird from using the new style *.msf
files, even though it had long ago created them!

Yousuf Khan

What I had tested before, was TBird 45 (sufficiently newer than the
TBird 38 that launched maildir). There was no conversion claimed
in TBird 45.

I was just looking at TBird 60.9.1 in a VM here (a setup that's
only used for email testing), and I added a news server to it,
and not only did if offer the button to choose .msf versus
maildir, but when I selected maildir, it claimed to be
"doing a conversion" to the other format. Even though
at that moment, no groups existed.

I added one group, and again it claimed to be doing a
conversion, and now there's a parallel "maildir" folder
which presumes to be a copy of the .msf folder.

If you kept your original setup with the 500000 files,
you might try updating to 60.9.1 or so, and trying
to flip the control using that version. It seemed to
unsubscribe me from the one group I'd selected, but
it seems to have worked. I haven't had time to do much
other testing yet.

Paul

Yousuf Khan wrote:
[...]

I just took a chance, and deleted all of the old newsgroup folders, that
contained all of the old-style messages. Left all of the rest of the
files in that news server's base folder untouched. Then I started
Thunderbird up again. It re-downloaded the messages, and it only
downloaded from where I last left off. It's now filling the data files
known as *.msf (e.g. alt.comp.os.windows-10.msf) rather than filling the
folders! Interestingly, these *.msf files used to exist in this News
folder before, but they were just trivial 1K or 2K files, with nothing
substantial inside them. They are now substantial files now, ranging
from 44 KB to 41 MB. So it looks like having those old folders there all
of this time was preventing Thunderbird from using the new style *.msf
files, even though it had long ago created them!

boggle!

If you apparently did not mind to delete all the old articles, then
why did you keep 580,000 old articles in the first place!?

You can set global and per group retention policies, so if you do not
need so much articles, just set those to appropriate values.

On 5/1/2020 10:55 AM, Frank Slootweg wrote:
boggle!

If you apparently did not mind to delete all the old articles, then
why did you keep 580,000 old articles in the first place!?

Simple, because I had no idea what the purpose of any of these files in
this folder were for in any detail, what was important, and where
exactly data resided, so I just backed up everything. That way I
wouldn't have to recreate everything from scratch, and go through hours
of debugging. I've had situations were just 1 important file goes
missing which screws up the entire configuration, and trying to find
that one missing file among half million is a needle in a haystack.

So now after the deletion, I'm down from half million to only about 600
files. And I did a test backup, and the backup went from over 8 hours,
down to only 2.5 minutes! My feeling is that perhaps a lot of those
half-million files were just left over from decades of junk that
Thunderbird did not clear, even though it said it was clearing them.

Yousuf Khan

On 5/1/2020 5:11 AM, Paul wrote:
If you kept your original setup with the 500000 files,
you might try updating to 60.9.1 or so, and trying
to flip the control using that version. It seemed to
unsubscribe me from the one group I'd selected, but
it seems to have worked. I haven't had time to do much
other testing yet.

Well, I have been completely uptodate on the Thunderbird releases for a
while now. I was running 68.7, even before this.

I think what's happening here is that Thunderbird wasn't expecting there
to be such long-term'ers like me continuously using their product. I'd
been using Thunderbird since version 0.something, and what was around
back then, is not what is around now, and so they never expected that
I'd be around since back then, and they had no plans for how to migrate
old-timers like me. So they just kept using the old format files in my
setup, even though the new format already existed, but they just ignored it.

Yousuf Khan

Yousuf Khan wrote:
On 5/1/2020 10:55 AM, Frank Slootweg wrote:
boggle!

If you apparently did not mind to delete all the old articles, then
why did you keep 580,000 old articles in the first place!?

Simple, because I had no idea what the purpose of any of these files in
this folder were for in any detail, what was important, and where
exactly data resided, so I just backed up everything. That way I
wouldn't have to recreate everything from scratch, and go through hours
of debugging. I've had situations were just 1 important file goes
missing which screws up the entire configuration, and trying to find
that one missing file among half million is a needle in a haystack.

I can - sort of - understand that, but because these 580,000 were
giving you so much hardship, I would have expected you to look at a
few of them, see that they were just News articles and take it from
there, i.e. set/lower the News retention settings in Thunderbird.

So now after the deletion, I'm down from half million to only about 600
files. And I did a test backup, and the backup went from over 8 hours,
down to only 2.5 minutes! My feeling is that perhaps a lot of those
half-million files were just left over from decades of junk that
Thunderbird did not clear, even though it said it was clearing them.

What you saw about "clearing" (the actual term is 'Compact'(ing)) is
for e-mail, not for News. This was already mentioned in this thread,
IIRC by VanguardLH. E-mail folders need to be compacted, because you
might delete some messages from a folder, so the .msf file needs to be
compacted to recover the space occupied by the deleted messages. News
articles can be deleted as well (in Thunderbird), but most people won't,
because there's no point, because you can only delete your *copy*, not
the copies on the rest of The Net.

Anyway, you should probably set the (News) retention settings,
otherwise the storage will grow again without bounds, not not in number
of files, but in number of MBs/GBs.

On 5/1/2020 1:52 PM, Frank Slootweg wrote:
I can - sort of - understand that, but because these 580,000 were
giving you so much hardship, I would have expected you to look at a
few of them, see that they were just News articles and take it from
there, i.e. set/lower the News retention settings in Thunderbird.

No, I knew those were the message files, considering that there were so
many of them, what else could they have been? But often there are other
files interspersed among them, that can often go overlooked because it's
overwhelmed by the mass of all of the main files. Just let the backup
software handle backing all of it up.

Yousuf Khan