I have two a regular mount point /fastlocal, which is a partition on an
SSD. On top of various items in /fastlocal, I cryptmount a variety of
encrypted partitions. But this is leading to trouble. After mounting the
encrypted partition, I ran mount -a. Now, it should NOT have remounted
/fastlocal, since it was already mounted. But it did. Mount now says
that there are two mounts of fastmount
/dev/sda7 on /fastlocal type ext4 (rw,noatime)
/dev/sda7 on /fastlocal/unruhhome/unruh type ext4 (rw,noatime)
And now the
cryptmount partitions were buried. Ie, all I find in /fastlocal is the
empty mountpoints for the various cryptmount partitions. I try to
unmount the cryptmount partions but they all tell me that they are in
use and cannot be umounted. I try to unmount /fastlocal, and it tells me
that cannot do so as it is in use. So I am stuck.
If I were not manymiles away from the machine, I could reboot, but
somehow a usb dive got left in the machine, and when I try to reboot, it
tries to reboot from the usbdrive (it is a copy of the Mga8 installation
disk). And my arm is not long enough to reach it. Is there any way out
of this predicament.
And this appears to be a bug in mount and/or cryptmount. Ie, the second
mount of /fastlocal covers up the cryptmount mounts.

On 2023-12-02, William Unruh <unruh@invalid.ca> wrote:
> I have two a regular mount point /fastlocal, which is a partition on an
> SSD. On top of various items in /fastlocal, I cryptmount a variety of
> encrypted partitions. But this is leading to trouble. After mounting the
> encrypted partition, I ran mount -a. Now, it should NOT have remounted
> /fastlocal, since it was already mounted. But it did. Mount now says
> that there are two mounts of fastmount
> /dev/sda7 on /fastlocal type ext4 (rw,noatime)
> /dev/sda7 on /fastlocal/unruhhome/unruh type ext4 (rw,noatime)
> And now the
> cryptmount partitions were buried. Ie, all I find in /fastlocal is the
> empty mountpoints for the various cryptmount partitions. I try to
> unmount the cryptmount partions but they all tell me that they are in
> use and cannot be umounted. I try to unmount /fastlocal, and it tells me
> that cannot do so as it is in use. So I am stuck.
> If I were not manymiles away from the machine, I could reboot, but
> somehow a usb dive got left in the machine, and when I try to reboot, it
> tries to reboot from the usbdrive (it is a copy of the Mga8 installation
> disk). And my arm is not long enough to reach it. Is there any way out
> of this predicament.
> And this appears to be a bug in mount and/or cryptmount. Ie, the second
> mount of /fastlocal covers up the cryptmount mounts.
>
>

It is even stupider than I thought.
When I installed the SSD, a few years ago, Ihad a homedirectory on
/disk9/home/unruh
When I installed the SSD, I installed my home directory onto the SSD
whcih was mounted on /fastlocal.
This worked fine but just in case there were some old references in some
software script to /disk9/home/unruh, I made a soft link to
/fastlocal/unruh. Then sometime, I put in a bind mount from
/fastlocal/unruh to /disk9/home/unruh. Since the cryptmounts were done
by hand after the system had booted up, this did not matter, but if I
did a mount -a, that mount from /fastlocal/unruh on top of
/fastlocal/unruh covered up the cryptmount.
Now the bug in the system meant that if I tried to umount
/fastlocal/unruh, I think it tried to do that first mount, instead of
the last mount. and the cryptmounts sat on top of it. and stopped the
unmount. But the crytpmounts were now covered by the last
/fastlocal/unruh mount and the system was all tied up and unfixable.

Or is there some way I can get out of this mess without rebooting?

On 12/2/23 16:46, William Unruh wrote:
> On 2023-12-02, William Unruh <unruh@invalid.ca> wrote:
>> I have two a regular mount point /fastlocal, which is a partition on an
>> SSD. On top of various items in /fastlocal, I cryptmount a variety of
>> encrypted partitions. But this is leading to trouble. After mounting the
>> encrypted partition, I ran mount -a. Now, it should NOT have remounted
>> /fastlocal, since it was already mounted. But it did. Mount now says
>> that there are two mounts of fastmount
>> /dev/sda7 on /fastlocal type ext4 (rw,noatime)
>> /dev/sda7 on /fastlocal/unruhhome/unruh type ext4 (rw,noatime)
>> And now the
>> cryptmount partitions were buried. Ie, all I find in /fastlocal is the
>> empty mountpoints for the various cryptmount partitions. I try to
>> unmount the cryptmount partions but they all tell me that they are in
>> use and cannot be umounted. I try to unmount /fastlocal, and it tells me
>> that cannot do so as it is in use. So I am stuck.
>> If I were not manymiles away from the machine, I could reboot, but
>> somehow a usb dive got left in the machine, and when I try to reboot, it
>> tries to reboot from the usbdrive (it is a copy of the Mga8 installation
>> disk). And my arm is not long enough to reach it. Is there any way out
>> of this predicament.
>> And this appears to be a bug in mount and/or cryptmount. Ie, the second
>> mount of /fastlocal covers up the cryptmount mounts.
>>
>>
>
> It is even stupider than I thought.
> When I installed the SSD, a few years ago, Ihad a homedirectory on
> /disk9/home/unruh
> When I installed the SSD, I installed my home directory onto the SSD
> whcih was mounted on /fastlocal.
> This worked fine but just in case there were some old references in some
> software script to /disk9/home/unruh, I made a soft link to
> /fastlocal/unruh. Then sometime, I put in a bind mount from
> /fastlocal/unruh to /disk9/home/unruh. Since the cryptmounts were done
> by hand after the system had booted up, this did not matter, but if I
> did a mount -a, that mount from /fastlocal/unruh on top of
> /fastlocal/unruh covered up the cryptmount.
> Now the bug in the system meant that if I tried to umount
> /fastlocal/unruh, I think it tried to do that first mount, instead of
> the last mount. and the cryptmounts sat on top of it. and stopped the
> unmount. But the crytpmounts were now covered by the last
> /fastlocal/unruh mount and the system was all tied up and unfixable.
>
> Or is there some way I can get out of this mess without rebooting?
>
>
Can you unmount by using using umount /dev/sd???

On 2023-12-02, jim whitby <jim@afraid.org> wrote:
> On 12/2/23 16:46, William Unruh wrote:
>> On 2023-12-02, William Unruh <unruh@invalid.ca> wrote:
>>> I have two a regular mount point /fastlocal, which is a partition on an
>>> SSD. On top of various items in /fastlocal, I cryptmount a variety of
>>> encrypted partitions. But this is leading to trouble. After mounting the
>>> encrypted partition, I ran mount -a. Now, it should NOT have remounted
>>> /fastlocal, since it was already mounted. But it did. Mount now says
>>> that there are two mounts of fastmount
>>> /dev/sda7 on /fastlocal type ext4 (rw,noatime)
>>> /dev/sda7 on /fastlocal/unruhhome/unruh type ext4 (rw,noatime)
>>> And now the
>>> cryptmount partitions were buried. Ie, all I find in /fastlocal is the
>>> empty mountpoints for the various cryptmount partitions. I try to
>>> unmount the cryptmount partions but they all tell me that they are in
>>> use and cannot be umounted. I try to unmount /fastlocal, and it tells me
>>> that cannot do so as it is in use. So I am stuck.
>>> If I were not manymiles away from the machine, I could reboot, but
>>> somehow a usb dive got left in the machine, and when I try to reboot, it
>>> tries to reboot from the usbdrive (it is a copy of the Mga8 installation
>>> disk). And my arm is not long enough to reach it. Is there any way out
>>> of this predicament.
>>> And this appears to be a bug in mount and/or cryptmount. Ie, the second
>>> mount of /fastlocal covers up the cryptmount mounts.
>>>
>>>
>>
>> It is even stupider than I thought.
>> When I installed the SSD, a few years ago, Ihad a homedirectory on
>> /disk9/home/unruh
>> When I installed the SSD, I installed my home directory onto the SSD
>> whcih was mounted on /fastlocal.
>> This worked fine but just in case there were some old references in some
>> software script to /disk9/home/unruh, I made a soft link to
>> /fastlocal/unruh. Then sometime, I put in a bind mount from
>> /fastlocal/unruh to /disk9/home/unruh. Since the cryptmounts were done
>> by hand after the system had booted up, this did not matter, but if I
>> did a mount -a, that mount from /fastlocal/unruh on top of
>> /fastlocal/unruh covered up the cryptmount.
>> Now the bug in the system meant that if I tried to umount
>> /fastlocal/unruh, I think it tried to do that first mount, instead of
>> the last mount. and the cryptmounts sat on top of it. and stopped the
>> unmount. But the crytpmounts were now covered by the last
>> /fastlocal/unruh mount and the system was all tied up and unfixable.
>>
>> Or is there some way I can get out of this mess without rebooting?
>>
>>
> Can you unmount by using using umount /dev/sd???
>

Nope, does not work. Complains "Target is busy".
Not sure what it thinks the target is since in /etc/fstab, the target is
/disk9/home/unruh
which is a softlink to /fastlocal/unruh, which seems to have resulted in
/fastlocal/unruh being bind mounted on /fastlocal/unruh. Surely
mounting a directory on top of itself should be illegal and not work.
What is interesting is that other mounts onto directories inside
/fastlocal/unruh get hidden by the mount of the directory on top of
itself, and then make the directory busy.

On 2023-12-03, David W. Hodgins <dwhodgins@nomail.afraid.org> wrote:
> On Sun, 03 Dec 2023 10:41:16 -0500, William Unruh <unruh@invalid.ca> wrote:
>> Nope, does not work. Complains "Target is busy".
>> Not sure what it thinks the target is since in /etc/fstab, the target is
>> /disk9/home/unruh
>> which is a softlink to /fastlocal/unruh, which seems to have resulted in
>> /fastlocal/unruh being bind mounted on /fastlocal/unruh. Surely
>> mounting a directory on top of itself should be illegal and not work.
>> What is interesting is that other mounts onto directories inside
>> /fastlocal/unruh get hidden by the mount of the directory on top of
>> itself, and then make the directory busy.
>
> Don't try to unmount the device, as that's used by both mounts (directly or
> indirectly). umount the directory used in the second mount, to reveal the
> directory and it's contents used in the first mount.
>
> If the second mount is busy, something started after the second mount is using
> a directory or file inside of that second mount. It could be something such
> as the current directory inside of a terminal in a konsole tab, or a program
> such as tracker monitoring the file system for changes to index.
>
> The program lsof can help identify those programs.

Unfortunately neither lsof nor fuser indicate that anything is using
that directory.
Also it is not mounted on any remote nfs client. findmnt indicates that
the various cryptmounts are mounted, but mount sees none of them and
cryptmount -u says they are not mounted, presumably because they are
hidden under the second mount of /fastlocal/unruh on top of itself.
Presumably it is also because the second mount occured via the link.
(ie it was in fstab
/fastlocal/unruh /disk9/home/unruh none bind,nofail 0 0
>
> Regards, Dave Hodgins

On 12/3/23 10:41, William Unruh wrote:
<snip>>>>
>> Can you unmount by using using umount /dev/sd???
>>
>
> Nope, does not work. Complains "Target is busy".
> Not sure what it thinks the target is since in /etc/fstab, the target is
> /disk9/home/unruh
> which is a softlink to /fastlocal/unruh, which seems to have resulted in
> /fastlocal/unruh being bind mounted on /fastlocal/unruh. Surely
> mounting a directory on top of itself should be illegal and not work.
> What is interesting is that other mounts onto directories inside
> /fastlocal/unruh get hidden by the mount of the directory on top of
> itself, and then make the directory busy.
>
>
Part of man umount
....

-f, --force
Force an unmount (in case of an unreachable NFS system).

Note that this option does not guarantee that umount command
does not hang. It’s strongly
recommended to use absolute paths without symlinks to avoid
unwanted readlink(2) and stat(2) system
calls on unreachable NFS in umount.
....

-l, --lazy
Lazy unmount. Detach the filesystem from the file hierarchy
now, and clean up all references to this
filesystem as soon as it is not busy anymore.

A system reboot would be expected in near future if you’re
going to use this option for network
filesystem or local filesystem with submounts. The
recommended use-case for umount -l is to prevent
hangs on shutdown due to an unreachable network share where
a normal umount will hang due to a
downed server or a network partition. Remounts of the share
will not be possible.

Try the -l first

Jim

On 2023-12-04, jim whitby <jim@afraid.org> wrote:
> On 12/3/23 10:41, William Unruh wrote:
><snip>>>>
> >> Can you unmount by using using umount /dev/sd???
> >>
> >
> > Nope, does not work. Complains "Target is busy".
> > Not sure what it thinks the target is since in /etc/fstab, the target is
> > /disk9/home/unruh
> > which is a softlink to /fastlocal/unruh, which seems to have resulted in
> > /fastlocal/unruh being bind mounted on /fastlocal/unruh. Surely
> > mounting a directory on top of itself should be illegal and not work.
> > What is interesting is that other mounts onto directories inside
> > /fastlocal/unruh get hidden by the mount of the directory on top of
> > itself, and then make the directory busy.
> >
> >
> Part of man umount
> ...
>
> -f, --force
> Force an unmount (in case of an unreachable NFS system).
>
> Note that this option does not guarantee that umount command
> does not hang. It’s strongly
> recommended to use absolute paths without symlinks to avoid
> unwanted readlink(2) and stat(2) system
> calls on unreachable NFS in umount.

None of these are nfs filesystems. they are all local. I have also found
-f not to work even for nfs mounted filesystems.

> ...
>
> -l, --lazy
> Lazy unmount. Detach the filesystem from the file hierarchy
> now, and clean up all references to this
> filesystem as soon as it is not busy anymore.

The problem is that I have no idea what it making them busy. In this
case I suspect it is the fact that cryptmount filesystems are mounted on
the first mounting /fastlocal/unruh, and the second one hides all of he
cryptmounts. So I cannot unmount cryptmount partitions, and the system
things that there is only the second mount and the cryptmounts are
making the second one busy.
If I do
findmts
I see only the one mount onto /fastlocal/unruh, not two of them.

I finally "solved" the problem by getting someone who was able to get
into the locked server room to unplug the usb stick from the machine
which was stopping the reboot. Then I could reboot the machine making
sure I had removed the indirect mount onto the /fastlocal/unruh via the
solft link.
(Ie, the second mount was a mount to /disk9/home/unruh which was a
solftlink to /fastlocal/unruh, which the mount commend interpreted as a
mount onto /fastlocal/unruh, even though a mount of that same partition
already existed via a direct mount onto /fastlocal/unruh. Ie, my fstab
had lines like
/dev/sdb3 /fastlocal/unruh
and
/dev/sdb3 /disk9/home/unruh bind,....
and disk9/home/unruh was a softlink to /fastlocal/unruh
That second one did not occur during reboot, but did occur after I
mounted the crytmounts onto some interior directories in
/fastlocal/unruh and then ran mount -a
If I tried to mount /dev/sdb3 directly onto /fastlocal/unruh, the mount
command would complain that /dev/sdb3 was already mounted onto
/fastlocal/unruh, but did not complain, I guess because I told it to
mount onto the link /disk9/home/unruh. Ie, the error trap failed to
complete the link and realise I was trying a double mount.

Ie, mount should not allow bin mounts onto links, or should test that
the link does not result in a double mount from the same partition onto
the same mountpoint
>
> A system reboot would be expected in near future if you’re
> going to use this option for network
> filesystem or local filesystem with submounts. The
> recommended use-case for umount -l is to prevent
> hangs on shutdown due to an unreachable network share where
> a normal umount will hang due to a
> downed server or a network partition. Remounts of the share
> will not be possible.
>
>
> Try the -l first
>
> Jim

On 2023-12-05, William Unruh <unruh@invalid.ca> wrote:
> On 2023-12-04, jim whitby <jim@afraid.org> wrote:
>> On 12/3/23 10:41, William Unruh wrote:
>><snip>>>>
>> >> Can you unmount by using using umount /dev/sd???
>> >>
>> >
>> > Nope, does not work. Complains "Target is busy".
>> > Not sure what it thinks the target is since in /etc/fstab, the target is
>> > /disk9/home/unruh
>> > which is a softlink to /fastlocal/unruh, which seems to have resulted in
>> > /fastlocal/unruh being bind mounted on /fastlocal/unruh. Surely
>> > mounting a directory on top of itself should be illegal and not work.
>> > What is interesting is that other mounts onto directories inside
>> > /fastlocal/unruh get hidden by the mount of the directory on top of
>> > itself, and then make the directory busy.
>> >
>> >
>> Part of man umount
>> ...
>>
>> -f, --force
>> Force an unmount (in case of an unreachable NFS system).
>>
>> Note that this option does not guarantee that umount command
>> does not hang. It’s strongly
>> recommended to use absolute paths without symlinks to avoid
>> unwanted readlink(2) and stat(2) system
>> calls on unreachable NFS in umount.
>
> None of these are nfs filesystems. they are all local. I have also found
> -f not to work even for nfs mounted filesystems.
>
>> ...
>>
>> -l, --lazy
>> Lazy unmount. Detach the filesystem from the file hierarchy
>> now, and clean up all references to this
>> filesystem as soon as it is not busy anymore.
>
> The problem is that I have no idea what it making them busy. In this
> case I suspect it is the fact that cryptmount filesystems are mounted on
> the first mounting /fastlocal/unruh, and the second one hides all of he
> cryptmounts. So I cannot unmount cryptmount partitions, and the system
> things that there is only the second mount and the cryptmounts are
> making the second one busy.
> If I do
> findmts
> I see only the one mount onto /fastlocal/unruh, not two of them.
>
>
> I finally "solved" the problem by getting someone who was able to get
> into the locked server room to unplug the usb stick from the machine
> which was stopping the reboot. Then I could reboot the machine making
> sure I had removed the indirect mount onto the /fastlocal/unruh via the
> solft link.
> (Ie, the second mount was a mount to /disk9/home/unruh which was a
> solftlink to /fastlocal/unruh, which the mount commend interpreted as a
> mount onto /fastlocal/unruh, even though a mount of that same partition
> already existed via a direct mount onto /fastlocal/unruh. Ie, my fstab
> had lines like
> /dev/sdb3 /fastlocal/unruh
> and
> /dev/sdb3 /disk9/home/unruh bind,....
> and disk9/home/unruh was a softlink to /fastlocal/unruh
> That second one did not occur during reboot, but did occur after I
> mounted the crytmounts onto some interior directories in
> /fastlocal/unruh and then ran mount -a
> If I tried to mount /dev/sdb3 directly onto /fastlocal/unruh, the mount
> command would complain that /dev/sdb3 was already mounted onto
> /fastlocal/unruh, but did not complain, I guess because I told it to
> mount onto the link /disk9/home/unruh. Ie, the error trap failed to
> complete the link and realise I was trying a double mount.
>
> Ie, mount should not allow bin mounts onto links, or should test that
> the link does not result in a double mount from the same partition onto
> the same mountpoint
>
>>
>> A system reboot would be expected in near future if you’re
>> going to use this option for network
>> filesystem or local filesystem with submounts. The
>> recommended use-case for umount -l is to prevent
>> hangs on shutdown due to an unreachable network share where
>> a normal umount will hang due to a
>> downed server or a network partition. Remounts of the share
>> will not be possible.
>>
>>
>> Try the -l first
>>
>> Jim

Maybe Running:
find /proc -name mounts | xargs grep /dev/sdb3
then kill -9 the PID's of found offending processes/mounts.