Fixing dired hangs when NFS mount goes AWOL

Fixing dired hangs when NFS mount goes AWOL

[Michael Olson]

Would you consider adding "-l" to the list of default arguments for
`directory-free-space-args'?  This will fix an issue where attempting
to open dired buffers with ls-lisp enabled hangs due to an AWOL NFS
mount.  Considering that the check is skipped if the current directory
is on a remote filesystem, I think this is a reasonable default.

-- 
Michael Olson  |  http://mwolson.org/

Re: Fixing dired hangs when NFS mount goes AWOL

[Chong Yidong]

Michael Olson <mwolson@gnu.org> writes:

> Would you consider adding "-l" to the list of default arguments for
> `directory-free-space-args'?  This will fix an issue where attempting
> to open dired buffers with ls-lisp enabled hangs due to an AWOL NFS
> mount.  Considering that the check is skipped if the current directory
> is on a remote filesystem, I think this is a reasonable default.

If the check is skipped for remote directories, how does the hang occur?

Re: Fixing dired hangs when NFS mount goes AWOL

[Michael Olson]

The check is only skipped if the directory you're trying to insert
into dired is remote.  If the dir is local, it will attempt the check,
and the 'df' call will hang because it's waiting on info from the NFS
filesystem.

On Wed, Oct 5, 2011 at 2:27 PM, Chong Yidong <cyd@stupidchicken.com> wrote:
> Michael Olson <mwolson@gnu.org> writes:
>
>> Would you consider adding "-l" to the list of default arguments for
>> `directory-free-space-args'?  This will fix an issue where attempting
>> to open dired buffers with ls-lisp enabled hangs due to an AWOL NFS
>> mount.  Considering that the check is skipped if the current directory
>> is on a remote filesystem, I think this is a reasonable default.
>
> If the check is skipped for remote directories, how does the hang occur?
>



-- 
Michael Olson  |  http://mwolson.org/

Re: Fixing dired hangs when NFS mount goes AWOL

[Glenn Morris]

>>> Would you consider adding "-l" to the list of default arguments for
>>> `directory-free-space-args'?  This will fix an issue where attempting
>>> to open dired buffers with ls-lisp enabled hangs due to an AWOL NFS
>>> mount.

Could you explain what the issue actually is?
Normally even trying to ls a directory containing eg a link to a missing
NFS directory hangs.
Are you saying that doing dired on a local directory that contains a
link to a missing NFS mount works, but hangs in calling df to calculate
the 'total used xxx available yyy' part?
What does ls-lisp have to do with it?
I can see how adding -l might help with a plain `df' call, but how can
it help `df DIR' if DIR is local?

Re: Fixing dired hangs when NFS mount goes AWOL

[Michael Olson]

On Wed, Oct 5, 2011 at 3:14 PM, Glenn Morris <rgm@gnu.org> wrote:
>
>>>> Would you consider adding "-l" to the list of default arguments for
>>>> `directory-free-space-args'?  This will fix an issue where attempting
>>>> to open dired buffers with ls-lisp enabled hangs due to an AWOL NFS
>>>> mount.
>
> Could you explain what the issue actually is?
> Normally even trying to ls a directory containing eg a link to a missing
> NFS directory hangs.
>
> Are you saying that doing dired on a local directory that contains a
> link to a missing NFS mount works, but hangs in calling df to calculate
> the 'total used xxx available yyy' part?

The issue is that regardless of what directory you're trying to view
using dired, 'df' apparently still tries to stat *every* known
filesystem, including remote filesystems, before limiting its output
to just that of the filsystem for the given directory.  Changing the
df options by adding '-l' makes df only consider local filesystems, so
that it never tries to stat an NFS filesystem.

> What does ls-lisp have to do with it?

The call order is:
dired-insert-directory
insert-directory (version in ls-lisp.el is used)
get-free-disk-space
-> calls 'df'

> I can see how adding -l might help with a plain `df' call, but how can
> it help `df DIR' if DIR is local?

Apparently it's an issue in how df is implemented, in that it still
attempts to stat all existing mounts before narrowing the listing by
the directory param.  Possibly to resolve symlinks and get a
"truename"?

-- 
Michael Olson  |  http://mwolson.org/

Re: Fixing dired hangs when NFS mount goes AWOL

[Glenn Morris]

Michael Olson wrote:

> insert-directory (version in ls-lisp.el is used)

I'm surprised this succeeds if there is a missing NFS mount.
You're saying the NFS mount is outside the directory being listed, but
still hangs df?

> Apparently it's an issue in how df is implemented, in that it still
> attempts to stat all existing mounts before narrowing the listing by
> the directory param.

Wow, really? That seems like an odd way for it to be implemented.

Quick search gives eg

http://lists.gnu.org/archive/html/bug-coreutils/2003-03/msg00014.html

which suggests that df is aware of such issues and tries to avoid them.

Re: Fixing dired hangs when NFS mount goes AWOL

[Glenn Morris]

But, -l is probably not an option for the default, because:

cd /path/to/remote/nfs
df -l .

  -> "df: no file systems processed"

which breaks Dired's "total used" line on healthy NFS mounts.

Re: Fixing dired hangs when NFS mount goes AWOL

[Michael Olson]

On Wed, Oct 5, 2011 at 4:14 PM, Glenn Morris <rgm@gnu.org> wrote:
>
> But, -l is probably not an option for the default, because:
>
> cd /path/to/remote/nfs
> df -l .
>
>  -> "df: no file systems processed"
>
> which breaks Dired's "total used" line on healthy NFS mounts.
>

Nuts, I thought (file-remote-p '/some/nfs/mount') would return
non-nil, but it doesn't.

So it seems like the only viable solution is to give users a way to
suppress the totals line in ls-lisp's insert-directory, which would
get rid of the df call.

-- 
Michael Olson  |  http://mwolson.org/

Re: Fixing dired hangs when NFS mount goes AWOL

[Michael Olson]

On Wed, Oct 5, 2011 at 4:27 PM, Michael Olson <mwolson@gnu.org> wrote:
> On Wed, Oct 5, 2011 at 4:14 PM, Glenn Morris <rgm@gnu.org> wrote:
>>
>> But, -l is probably not an option for the default, because:
>>
>> cd /path/to/remote/nfs
>> df -l .
>>
>>  -> "df: no file systems processed"
>>
>> which breaks Dired's "total used" line on healthy NFS mounts.
>>
>
> Nuts, I thought (file-remote-p '/some/nfs/mount') would return
> non-nil, but it doesn't.
>
> So it seems like the only viable solution is to give users a way to
> suppress the totals line in ls-lisp's insert-directory, which would
> get rid of the df call.

(Or just customize directory-free-space-args in the first place, which
kind of already has this effect.)

-- 
Michael Olson  |  http://mwolson.org/

Re: Fixing dired hangs when NFS mount goes AWOL

[Stefan Monnier]

> Nuts, I thought (file-remote-p '/some/nfs/mount') would return
> non-nil, but it doesn't.

The concept of "remote" file is a very fuzzy one, which means very
different things to different people in different cases.
I tend to agree that file-remote-p should return non-nil on NFS mounted
filesystems, but that's simply not implemented.  Patch welcome,


        Stefan

Re: Fixing dired hangs when NFS mount goes AWOL

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Nuts, I thought (file-remote-p '/some/nfs/mount') would return
>> non-nil, but it doesn't.
>
> The concept of "remote" file is a very fuzzy one, which means very
> different things to different people in different cases.
> I tend to agree that file-remote-p should return non-nil on NFS mounted
> filesystems, but that's simply not implemented.  Patch welcome,

This will break Tramp completely. Please don't change its current behaviour.

>         Stefan

Best regards, Michael.

Re: Fixing dired hangs when NFS mount goes AWOL

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Date: Thu, 06 Oct 2011 15:18:36 +0200
> Cc: Chong Yidong <cyd@stupidchicken.com>, Michael Olson <mwolson@gnu.org>,
> 	Emacs Development Discussions <emacs-devel@gnu.org>
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> >> Nuts, I thought (file-remote-p '/some/nfs/mount') would return
> >> non-nil, but it doesn't.
> >
> > The concept of "remote" file is a very fuzzy one, which means very
> > different things to different people in different cases.
> > I tend to agree that file-remote-p should return non-nil on NFS mounted
> > filesystems, but that's simply not implemented.  Patch welcome,
> 
> This will break Tramp completely. Please don't change its current behaviour.

I was going to say precisely that, but Michael beat me to it.

I think past discussions of this particular calamity concluded that we
need a separate API for this, something like filesystem-mounted-p.
The uses of each API are expected to be very different.

Re: Fixing dired hangs when NFS mount goes AWOL

[Stefan Monnier]

>> The concept of "remote" file is a very fuzzy one, which means very
>> different things to different people in different cases.
>> I tend to agree that file-remote-p should return non-nil on NFS mounted
>> filesystems, but that's simply not implemented.  Patch welcome,
> This will break Tramp completely. Please don't change its current behaviour.

Why would it break Tramp?  This would only change the behavior of
file-remote-p for file names which are not handled by Tramp, so it
shouldn't break Tramp any more than adding some other file-name-handler
for other kinds of remote magic file names.


        Stefan

Re: Fixing dired hangs when NFS mount goes AWOL

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> The concept of "remote" file is a very fuzzy one, which means very
>>> different things to different people in different cases.
>>> I tend to agree that file-remote-p should return non-nil on NFS mounted
>>> filesystems, but that's simply not implemented.  Patch welcome,
>> This will break Tramp completely. Please don't change its current behaviour.
>
> Why would it break Tramp?  This would only change the behavior of
> file-remote-p for file names which are not handled by Tramp, so it
> shouldn't break Tramp any more than adding some other file-name-handler
> for other kinds of remote magic file names.

There was a very long discussion about, see
<http://thread.gmane.org/gmane.emacs.devel/87506/focus=87506>

In short:

Tramp uses file-remote-p internally. OK, this could be replaced by an
internal function.

Other packages use it as well, inside and outside core Emacs. We didn't
want to propagate tramp-* functions outside Tramp; file-remote-p is the
check to be used when necessary. Not so important for basic file name
operations, but needed for all process related things.

>         Stefan

Best regards, Michael.

Re: Fixing dired hangs when NFS mount goes AWOL

[Stefan Monnier]

>> Why would it break Tramp?
> There was a very long discussion about, see
> <http://thread.gmane.org/gmane.emacs.devel/87506/focus=87506>

I know, I participated in that thread.  I fail to see how it answers
the question.

> Tramp uses file-remote-p internally. OK, this could be replaced by an
> internal function.

It depends on whether it uses it right or not.

> Other packages use it as well, inside and outside core Emacs.

What does it mean "use it as well"?  In which way are those uses
incompatible with a file-remote-p which returns non-nil for SSHFS and
NFS mounts?

> Not so important for basic file name operations, but needed for all
> process related things.

These are the uses which should use something like
file-accessible-directory-p instead of file-remote-p, right?


        Stefan

Re: Fixing dired hangs when NFS mount goes AWOL

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> Why would it break Tramp?
>> There was a very long discussion about, see
>> <http://thread.gmane.org/gmane.emacs.devel/87506/focus=87506>
>
> I know, I participated in that thread.  I fail to see how it answers
> the question.

It was a more general discussion about changing the semantics of
file-remote-p. Caused by a similar problem (remote shares in MS Windows,
which are not responsive).

The discussion went into the direction, that maybe an opposite function,
file-local-p, might be useful. An interesting argument was given by
Andreas Schwab: what about mounted CD-ROMs, or USB devices? Are they
"local" or "remote".

>> Tramp uses file-remote-p internally. OK, this could be replaced by an
>> internal function.
>
> It depends on whether it uses it right or not.

Whatever is TRT. In this, we disagree.

>> Other packages use it as well, inside and outside core Emacs.
>
> What does it mean "use it as well"?  In which way are those uses
> incompatible with a file-remote-p which returns non-nil for SSHFS and
> NFS mounts?
>
>> Not so important for basic file name operations, but needed for all
>> process related things.
>
> These are the uses which should use something like
> file-accessible-directory-p instead of file-remote-p, right?

A needed functionality is to translate between "local" file names (on
the remote host), and "remote" file names in the sense of Tramp. The
usual implementation is

local -> remote:   (concat (file-remote-p default-directory) filename)
remote -> local:   (file-remote-p filename 'localname)

>         Stefan

Best regards, Michael.

Re: Fixing dired hangs when NFS mount goes AWOL

[Stefan Monnier]

>>> Not so important for basic file name operations, but needed for all
>>> process related things.
>> These are the uses which should use something like
>> file-accessible-directory-p instead of file-remote-p, right?

> A needed functionality is to translate between "local" file names (on
> the remote host), and "remote" file names in the sense of Tramp. The
> usual implementation is

> local -> remote:   (concat (file-remote-p default-directory) filename)
> remote -> local:   (file-remote-p filename 'localname)

Yes, file-remote-p for NFS would need to pay attention to such problems,
and it might introduce bugs, but that doesn't mean that file-remote-p
can't be used for NFS.


        Stefan

Re: Fixing dired hangs when NFS mount goes AWOL

[Michael Olson]

On Wed, Oct 5, 2011 at 3:55 PM, Glenn Morris <rgm@gnu.org> wrote:
> Michael Olson wrote:
>> Apparently it's an issue in how df is implemented, in that it still
>> attempts to stat all existing mounts before narrowing the listing by
>> the directory param.
>
> Wow, really? That seems like an odd way for it to be implemented.
>
> Quick search gives eg
>
> http://lists.gnu.org/archive/html/bug-coreutils/2003-03/msg00014.html
>
> which suggests that df is aware of such issues and tries to avoid them.

I'm very skeptical that the `df' issue is actually fixed upstream,
especially given that the patch seems to be almost entirely
documentation.  Maybe it deserves to be bumped there.  Attempts to
repro the issue have been unsuccessful, and I've spent about all the
time I care to in attempting it.

OS: Ubuntu 11.04
df (GNU coreutils) 8.5

-- 
Michael Olson  |  http://mwolson.org/