Recovering a folder on an LVM ext3 partition

I just had a sudden break down of my hard disk running Linux, don't
know what the reason is, but the result is that all the partitions in
an LVM group are now seriously corrupt. The HD layout looks something
like this:

hda1 - primary Windows partition
hda2 - primary ext3 for /boot
hda3 - LVM containing 4 logical ext3 partitions: / (root), /home,
/usr/local, swap

It's strange that LVM itself reports no errors, but the logical
partitions are all screwed, while the 2 partitions not inside LVM are
untouched. Anyone have a clue why that is?

Anyway, obviously the partition I care most about is /home. I booted
from a LiveCD and I tried doing fsck on the root first as a trial, and
ended up with a working, but empty, partition. :-) So I proceeded with
more caution on the /home partition, I answered "no" to any questions
about "force rewrite", but I let it move things to lost+found. At the
end, I attempted to mount the partition, it worked, but one top-level
folder which was called "data" is gone. At one point during fsck, I
noticed it asking about clearing this folder's extra inodes or somthing
like that. This folder contained tons of photo files, as well as some
other media and documents. There are lots of files in the lost+found,
and some folders, so I assumed the "data" folder just might be one of
those folders, however corrupted. However, if I try to cd into any of
them, I just get thrown back to the home folder (the home of the
LiveCD, not the one I'm recovering). Is there any way I can try to
recover these folders?

Previously microx wrote:
I just had a sudden break down of my hard disk running Linux, don't
know what the reason is, but the result is that all the partitions in
an LVM group are now seriously corrupt. The HD layout looks something
like this:

hda1 - primary Windows partition
hda2 - primary ext3 for /boot
hda3 - LVM containing 4 logical ext3 partitions: / (root), /home,
/usr/local, swap

It's strange that LVM itself reports no errors, but the logical
partitions are all screwed, while the 2 partitions not inside LVM are
untouched. Anyone have a clue why that is?

Can you be a bit more specific what "screwed" means? Also:
What kernel version, what were you doing when the corruption
occured?

Is it possible that yopu acctdentially overwrote parts of hda3? The
LVM descriptor block is at the end of the volume, so it would be
the last thing to go.

Anyway, obviously the partition I care most about is /home. I booted
from a LiveCD and I tried doing fsck on the root first as a trial, and
ended up with a working, but empty, partition. :-) So I proceeded with
more caution on the /home partition, I answered "no" to any questions
about "force rewrite", but I let it move things to lost+found. At the
end, I attempted to mount the partition, it worked, but one top-level
folder which was called "data" is gone. At one point during fsck, I
noticed it asking about clearing this folder's extra inodes or somthing
like that. This folder contained tons of photo files, as well as some
other media and documents. There are lots of files in the lost+found,
and some folders, so I assumed the "data" folder just might be one of
those folders, however corrupted. However, if I try to cd into any of
them, I just get thrown back to the home folder (the home of the
LiveCD, not the one I'm recovering). Is there any way I can try to
recover these folders?

Hmm. Since you worked on the original partitions without backup, there
is a good possibility that you destroyed additional data. Also
there will be only files in lost+found, not directories. The
behaviour you observe is correct (depending on shell settings).

Advice: Stop writing to the disk immediuately. Make a sector image anf
work on that. Then start looking for your files in lost+found. Other
steps depend on what caused the corruption, but stop writing on the
original immediately!

I am also curious what prompted the use of LVM for this setup. Usually
you would just do a conventional extended partition, using LVM does not
really make sense.

Additional advice: It is Linux and pretty corruption resistent, but
backup is non-optional for important files, no matter what.

Arno

Previously microx wrote:
I just had a sudden break down of my hard disk running Linux, don't
know what the reason is, but the result is that all the partitions in
an LVM group are now seriously corrupt. The HD layout looks something
like this:

hda1 - primary Windows partition
hda2 - primary ext3 for /boot
hda3 - LVM containing 4 logical ext3 partitions: / (root), /home,
/usr/local, swap

It's strange that LVM itself reports no errors, but the logical
partitions are all screwed, while the 2 partitions not inside LVM are
untouched. Anyone have a clue why that is?

P.S.: LVM does not know what is in the partitions. You can overwrite
the whole LVM volume except the last few blocks with zeros or random
data and it will still look fine to LVM. That behaviour is expected
and correct.

As to the reasons, since your root parttition seems not to have any
data left and since directories and files are ditributed all overt
the disk and backup superblocks are as well, you very likely
have overwritten your the beginning of hda3 somehow. If
/usr/local is intact, thet would be further evidence.

Arno

On 5 Sep 2006 02:30:40 -0700, "microx"
wrote:

I just had a sudden break down of my hard disk running Linux, don't
know what the reason is, but the result is that all the partitions in
an LVM group are now seriously corrupt. The HD layout looks something
like this:

hda1 - primary Windows partition
hda2 - primary ext3 for /boot
hda3 - LVM containing 4 logical ext3 partitions: / (root), /home,
/usr/local, swap

It's strange that LVM itself reports no errors, but the logical
partitions are all screwed, while the 2 partitions not inside LVM are
untouched. Anyone have a clue why that is?

Anyway, obviously the partition I care most about is /home. I booted
from a LiveCD and I tried doing fsck on the root first as a trial, and
ended up with a working, but empty, partition. :-) So I proceeded with
more caution on the /home partition, I answered "no" to any questions
about "force rewrite", but I let it move things to lost+found. At the
end, I attempted to mount the partition, it worked, but one top-level
folder which was called "data" is gone. At one point during fsck, I
noticed it asking about clearing this folder's extra inodes or somthing
like that. This folder contained tons of photo files, as well as some
other media and documents. There are lots of files in the lost+found,
and some folders, so I assumed the "data" folder just might be one of
those folders, however corrupted. However, if I try to cd into any of
them, I just get thrown back to the home folder (the home of the
LiveCD, not the one I'm recovering). Is there any way I can try to
recover these folders?

The fsck programs should not be used when data are lost. Anyway, you
could try ext3 data recovery programs. There is a free Windows program
Readext2 at

http://www.partitionsupport.com/utilities.htm

To use the program you need to know the 0 based cylinder number of a
superblock from the partition. If that as example is 6000, and the
disk is the first disk, the Readext2 command to examine the partition
is:

readext2 1 6000 6000

Or to search a cylinder area for a superblock to use (example):

readext2 1 6000 7000

--
Svend Olaf

Can you be a bit more specific what "screwed" means? Also:
What kernel version, what were you doing when the corruption
occured?

I can try to be more specific. :-) Basically, it won't mount saying it
has a "wrong filesystem" and after fsck, I find I've lost some files. I
wasn't doing anything special at the time, it's quite strange. I had
just clicked a button to start Firefox, and while it was opening, I
deleted 1 file that I had on the desktop and no longer needed. That's
where it all started when it gave me an error saying that I'm trying to
delete a file on a read only disk, so I thought to try logging out to
see if /home was remounted read-only for some reason, but then X
wouldn't start up again. So I tried restarting the computer, but then
it gave an error during boot-up that it can't mount the root
filesystem, so then I booted with the LiveCD to see what's going on. I
am running FC5 with a 2.6 kernel.

I am also curious what prompted the use of LVM for this setup. Usually
you would just do a conventional extended partition, using LVM does not
really make sense.

I am curious now too. :-) No particular reason, it's my personal
laptop, and I've just been seeing the LVM option during the Fedora
setup for a while, so I thought why not try it out to give me more
flexibility in resizing partitions if I needed. Learned my lesson now.
:-)

I do back up every few months, my main system is at home with a CVS
repository which I backup every month or so. But I'm on a business trip
for 2 months now, so what's not backed up is just that period, but that
still worries me because there's a lot of new email in the Thunderbird
folders and Gaim chat logs, and all the gigabytes of photos I took!

Also there will be only files in lost+found, not directories. The
behaviour you observe is correct (depending on shell settings)

Yeah, I get it. So does that mean I might be able to find individual
files from inside the lost folder directly in the lost+found? And is
there any way I can make use of these "recovered folders"?

P.S.: LVM does not know what is in the partitions. You can overwrite
the whole LVM volume except the last few blocks with zeros or random
data and it will still look fine to LVM.

I do understand that, what's wierd to me is just why it happened to all
the LVM partitions and not the others, with the LVM tools not noticing
anything wrong. If it was a head crash, I'd expect to see everything
messed up, and if it was a filesystem, I'd expect to see it on just one
partitions. But it could be, as you mentioned, that it overwrote the
start of the hda3 partition for some reason, and which spans clusters
from several of the logical volumes somehow. Why I'm asking is because
I'm not sure whether to assume that the partitions are still in the
right place or not, so that I would know whether to try playing around
with the paritioning or whether to just go ahead and try to recover
files. But I do assume it's the latter, it's just a thought. Anyway,
I'm avoiding LVM from now on. :-)

The fsck programs should not be used when data are lost. Anyway, you
could try ext3 data recovery programs.

Yeah, I'll keep the partition read-only now and try to use some data
recovery software now. I think what I'll do now is to try to copy out
any files I can currently see onto a separate disk, then I'll try the
Readext2 tool. I also read about another tool to use from Windows
called R-Linux, as well as PhotoRec which specifically tries to salvage
photos.

Thanks for the help guys. Any further tips will be greatly appreciated.
:-)

In article , Arno Wagner
writes

I am also curious what prompted the use of LVM for this setup. Usually
you would just do a conventional extended partition, using LVM does not
really make sense.

It seems to be the default in FC5 (which I note the OP is using.)
Doesn't make any sense to me either, especially since I installed FC5 on
a virgin disk and opted to create only the standard /boot, /, and swap
partitions. The installer then went and put the lot into a LVM volume
for reasons best known to itself.

My take is that it would be best to stick with standard partitioning
tools, since it's one less layer of obfuscation to work through when
things go wrong.

Similarly, I remove the LABEL= tags in /etc/fstab, replacing them with
the raw device names.

M.

Mike Tomlinson wrote:
It seems to be the default in FC5 (which I note the OP is using.)
Doesn't make any sense to me either, especially since I installed FC5 on
a virgin disk and opted to create only the standard /boot, /, and swap
partitions. The installer then went and put the lot into a LVM volume
for reasons best known to itself.

Yeah, using LVM was just being fancy I guess. But I learned to keep at
least the /home partition separate anyway (LVM or not) since it makes
it clear where the important stuff is in a recovery situation.

It's also best to keep the /home partition closer to the end of the HD,
since that's the area least likely to be affected if the software goes
wild or if you accidentally format the HD. I learned this in my
previous HD crash (9 years ago, I have a history of HD corruption and
recovery). :-) I had a completely dead drive back then, and I gave the
HD to a stupid repair workshop who managed to get the drive spinning
again, but then decided to repartition the whole thing as one partition
and format it! Luckily, my data partition was the 2nd half of the disk
and was not affected, but I had to re-write the partition table and
boot sector literally by hand byte-by-byte to get the data back. :-)

Right now I'm copying out everything from the lost+found, and I'm happy
to be finding quite a bit of my data there already without using any
data recovery software. It seems I can also cd into the recovered
folders, I was just typing it wrong. The files and folders in
lost+found are named in the form #123456, so I was doing "cd #123456"
and that was throwing me back to the home. Then I realized I should be
doing "cd \#123456" since # is a special character in the shell, so I
can actually get into the folders and there is stuff in them, even with
their original filenames. :-)

Anyway as I said, after recovering, I'll repartition using standard
types again and drop the LVM rubbish.

"Arno Wagner" wrote in message
Previously microx wrote:
I just had a sudden break down of my hard disk running Linux, don't
know what the reason is, but the result is that all the partitions in an
LVM group are now seriously corrupt. The HD layout looks something
like this:

hda1 - primary Windows partition
hda2 - primary ext3 for /boot
hda3 - LVM containing 4 logical ext3 partitions: / (root), /home,
/usr/local, swap

It's strange that LVM itself reports no errors, but the logical
partitions are all screwed, while the 2 partitions not inside LVM are
untouched.

Anyone have a clue why that is?

Because of how LVM works, maybe?


Can you be a bit more specific what "screwed" means? Also:
What kernel version, what were you doing when the corruption occured?

Is it possible that yopu acctdentially overwrote parts of hda3?
The LVM descriptor block is at the end of the volume, so it would be
the last thing to go.

Anyway, obviously the partition I care most about is /home. I booted
from a LiveCD and I tried doing fsck on the root first as a trial, and
ended up with a working, but empty, partition. :-) So I proceeded with
more caution on the /home partition, I answered "no" to any questions
about "force rewrite", but I let it move things to lost+found. At the
end, I attempted to mount the partition, it worked, but one top-level
folder which was called "data" is gone. At one point during fsck, I
noticed it asking about clearing this folder's extra inodes or somthing
like that. This folder contained tons of photo files, as well as some
other media and documents. There are lots of files in the lost+found,
and some folders, so I assumed the "data" folder just might be one of
those folders, however corrupted. However, if I try to cd into any of
them, I just get thrown back to the home folder (the home of the
LiveCD, not the one I'm recovering). Is there any way I can try to
recover these folders?

Hmm. Since you worked on the original partitions without backup, there
is a good possibility that you destroyed additional data.
Also there will be only files in lost+found, not directories.

The behaviour you observe is correct

(depending on shell settings).

So actually may not be correct.....


Advice: Stop writing to the disk immediuately. Make a sector image anf
work on that. Then start looking for your files in lost+found. Other steps
depend on what caused the corruption, but stop writing on the original
immediately!

I am also curious what prompted the use of LVM for this setup.
Usually you would just do a conventional extended partition,

using LVM does not really make sense.

So when does making use of LVM make sense if "using LVM does not really make sense", babblebot.


Additional advice: It is Linux and pretty corruption resistent,

As the above example easily illustrates.

but backup is non-optional for important files, no matter what.

Arno

Previously microx wrote:
Can you be a bit more specific what "screwed" means? Also:
What kernel version, what were you doing when the corruption
occured?
[...]
filesystem, so then I booted with the LiveCD to see what's going on. I
am running FC5 with a 2.6 kernel.

My guess would be that you actually ran either into a rare bug in
LVM or that some other part of FC5 has a serious issue. I am inclined
to believe the later since filesystem or LVM bugs are very rare
today. Might be a good idea to go to a newer kernel as well.

Arno

"Arno Wagner" wrote in message
Previously microx wrote:
Can you be a bit more specific what "screwed" means? Also:
What kernel version, what were you doing when the corruption occured?
[...]
filesystem, so then I booted with the LiveCD to see what's going on.
I am running FC5 with a 2.6 kernel.

My guess would be that you actually ran either into a rare bug in
LVM or that some other part of FC5 has a serious issue. I am inclined
to believe the later since filesystem or LVM bugs are very rare today.

So much for "It is Linux and pretty corruption resistent", babblemouth.

Might be a good idea to go to a newer kernel as well.

Arno