Emacs Hangs on Filesystem Operations on Stale NFS

Emacs Hangs on Filesystem Operations on Stale NFS

[Alexander Shukaev]

Hi Everyone,


I initiated a discussion back in 2015 [1] about fragility of Emacs in 
terms of filesystem operations on stale NFS.  No solution actually came 
out of this discussion.  I still find this issue very disruptive.  Yet 
another example would be `recentf-cleanup' which is in my case triggered 
on Emacs start up, when the file comes from stale NFS, the corresponding 
`file-readable-p' down the stack will hang indefinitely, and there would 
be no way to unfreeze it apart from issuing 'kill -9' to that Emacs 
instance.  Don't you people find it unacceptable for the daily usage?  
Well, I do.  Such hangs always disrupt daily work and require quite some 
time to track them down as they are not Lisp-debuggable with e.g. <C-g> 
in a straightforward way (these are dead hangs from C code, where even 
attaching a GDB does not work).

Well, enough rant.  I think I have a proposal how to fix the issue, even 
given the blocking nature of Emacs.  How about introducing a variable 
`file-access-timeout' defaulting to `nil', which would reflect a 
configurable timeout for all access operations (such as 
`file-readable-p')?  This would be achieved via `SIGALARM' in the C 
code, which would protect every such operation.  For example,

#include <sigaction.h>
#include <sys/stat.h>
#include <unistd.h>
#include <string.h>

static void alarm_handler(int sig)
{
     return;
}

int emacs_stat(const char* path, struct stat* s, unsigned int seconds)
{
     struct sigaction newact;
     struct sigaction oldact;

     memset(&newact, 0, sizeof(newact));
     memset(&oldact, 0, sizeof(oldact));

     sigemptyset(&newact.sa_mask);

     newact.sa_flags   = 0;
     newact.sa_handler = alarm_handler;
     sigaction(SIGALRM, &newact, &oldact);

     alarm(seconds);

     errno                 = 0;
     const int rc          = stat(path, s);
     const int saved_errno = errno;

     alarm(0);
     sigaction(SIGALRM, &oldact, NULL);

     errno = saved_errno;
     return rc;
}

where `seconds' should be initialized with the value of 
`file-access-timeout'.  The cool advantage of this that I see is that 
one can then also selectively `let'-bind different values for 
`file-access-timeout', thus having total control over the use cases in 
which one wants to protect oneself from indefinite hangs.

Kind regards,
Alexander

[1] 
https://lists.gnu.org/archive/html/help-gnu-emacs/2015-11/msg00251.html

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Andreas Schwab]

On Jun 11 2018, Alexander Shukaev <emacs@alexander.shukaev.name> wrote:

> I initiated a discussion back in 2015 [1] about fragility of Emacs in
> terms of filesystem operations on stale NFS.  No solution actually came
> out of this discussion.  I still find this issue very disruptive.  Yet
> another example would be `recentf-cleanup' which is in my case triggered
> on Emacs start up, when the file comes from stale NFS, the corresponding
> `file-readable-p' down the stack will hang indefinitely, and there would
> be no way to unfreeze it apart from issuing 'kill -9' to that Emacs
> instance.  Don't you people find it unacceptable for the daily usage?

How is that a bug in Emacs if you have stale NFS mounts?

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Alexander Shukaev]

On 2018-06-11 13:01, Andreas Schwab wrote:
> On Jun 11 2018, Alexander Shukaev <emacs@alexander.shukaev.name> wrote:
> 
>> I initiated a discussion back in 2015 [1] about fragility of Emacs in
>> terms of filesystem operations on stale NFS.  No solution actually 
>> came
>> out of this discussion.  I still find this issue very disruptive.  Yet
>> another example would be `recentf-cleanup' which is in my case 
>> triggered
>> on Emacs start up, when the file comes from stale NFS, the 
>> corresponding
>> `file-readable-p' down the stack will hang indefinitely, and there 
>> would
>> be no way to unfreeze it apart from issuing 'kill -9' to that Emacs
>> instance.  Don't you people find it unacceptable for the daily usage?
> 
> How is that a bug in Emacs if you have stale NFS mounts?
> 
> Andreas.

I didn't find a word "bug" in that quote.  I'd rather call it a 
robustness enhancement.  By the way, I guess you just don't realize how 
easy it is to run into that situation on daily basis.

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Andreas Schwab]

On Jun 11 2018, Alexander Shukaev <emacs@alexander.shukaev.name> wrote:

> On 2018-06-11 13:01, Andreas Schwab wrote:
>> On Jun 11 2018, Alexander Shukaev <emacs@alexander.shukaev.name> wrote:
>>
>>> I initiated a discussion back in 2015 [1] about fragility of Emacs in
>>> terms of filesystem operations on stale NFS.  No solution actually came
>>> out of this discussion.  I still find this issue very disruptive.  Yet
>>> another example would be `recentf-cleanup' which is in my case
>>> triggered
>>> on Emacs start up, when the file comes from stale NFS, the
>>> corresponding
>>> `file-readable-p' down the stack will hang indefinitely, and there
>>> would
>>> be no way to unfreeze it apart from issuing 'kill -9' to that Emacs
>>> instance.  Don't you people find it unacceptable for the daily usage?
>>
>> How is that a bug in Emacs if you have stale NFS mounts?
>>
>> Andreas.
>
> I didn't find a word "bug" in that quote.  I'd rather call it a robustness
> enhancement.  By the way, I guess you just don't realize how easy it is to
> run into that situation on daily basis.

Is it?  Looks like you have an unreliable NFS server, but that's hardly
Emacs' problem.  I have no problem with $HOME on NFS.

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Alexander Shukaev]

On 2018-06-11 13:50, Andreas Schwab wrote:
> On Jun 11 2018, Alexander Shukaev <emacs@alexander.shukaev.name> wrote:
> 
>> On 2018-06-11 13:01, Andreas Schwab wrote:
>>> On Jun 11 2018, Alexander Shukaev <emacs@alexander.shukaev.name> 
>>> wrote:
>>> 
>>>> I initiated a discussion back in 2015 [1] about fragility of Emacs 
>>>> in
>>>> terms of filesystem operations on stale NFS.  No solution actually 
>>>> came
>>>> out of this discussion.  I still find this issue very disruptive.  
>>>> Yet
>>>> another example would be `recentf-cleanup' which is in my case
>>>> triggered
>>>> on Emacs start up, when the file comes from stale NFS, the
>>>> corresponding
>>>> `file-readable-p' down the stack will hang indefinitely, and there
>>>> would
>>>> be no way to unfreeze it apart from issuing 'kill -9' to that Emacs
>>>> instance.  Don't you people find it unacceptable for the daily 
>>>> usage?
>>> 
>>> How is that a bug in Emacs if you have stale NFS mounts?
>>> 
>>> Andreas.
>> 
>> I didn't find a word "bug" in that quote.  I'd rather call it a 
>> robustness
>> enhancement.  By the way, I guess you just don't realize how easy it 
>> is to
>> run into that situation on daily basis.
> 
> Is it?  Looks like you have an unreliable NFS server, but that's hardly
> Emacs' problem.  I have no problem with $HOME on NFS.
> 
> Andreas.

Andreas, please let's finish that pointless thread branch.  I know what 
are you getting at but this has nothing to do with the fact that Emacs 
is fragile to external hang issues.  I'm very glad that you have 
reliable NFS server but the problem in my case has a completely 
different origin.  It's like you're jumping out of your skin to turn 
that feature proposal down, while I see no valid reasons for your 
defensive stance given that weak argumentation.

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Alexander Shukaev]

On 2018-06-11 12:27, Alexander Shukaev wrote:
> Hi Everyone,
> 
> 
> I initiated a discussion back in 2015 [1] about fragility of Emacs in
> terms of filesystem operations on stale NFS.  No solution actually
> came out of this discussion.  I still find this issue very disruptive.
>  Yet another example would be `recentf-cleanup' which is in my case
> triggered on Emacs start up, when the file comes from stale NFS, the
> corresponding `file-readable-p' down the stack will hang indefinitely,
> and there would be no way to unfreeze it apart from issuing 'kill -9'
> to that Emacs instance.  Don't you people find it unacceptable for the
> daily usage?  Well, I do.  Such hangs always disrupt daily work and
> require quite some time to track them down as they are not
> Lisp-debuggable with e.g. <C-g> in a straightforward way (these are
> dead hangs from C code, where even attaching a GDB does not work).
> 
> Well, enough rant.  I think I have a proposal how to fix the issue,
> even given the blocking nature of Emacs.  How about introducing a
> variable `file-access-timeout' defaulting to `nil', which would
> reflect a configurable timeout for all access operations (such as
> `file-readable-p')?  This would be achieved via `SIGALARM' in the C
> code, which would protect every such operation.  For example,
> 
> #include <sigaction.h>
> #include <sys/stat.h>
> #include <unistd.h>
> #include <string.h>
> 
> static void alarm_handler(int sig)
> {
>     return;
> }
> 
> int emacs_stat(const char* path, struct stat* s, unsigned int seconds)
> {
>     struct sigaction newact;
>     struct sigaction oldact;
> 
>     memset(&newact, 0, sizeof(newact));
>     memset(&oldact, 0, sizeof(oldact));
> 
>     sigemptyset(&newact.sa_mask);
> 
>     newact.sa_flags   = 0;
>     newact.sa_handler = alarm_handler;
>     sigaction(SIGALRM, &newact, &oldact);
> 
>     alarm(seconds);
> 
>     errno                 = 0;
>     const int rc          = stat(path, s);
>     const int saved_errno = errno;
> 
>     alarm(0);
>     sigaction(SIGALRM, &oldact, NULL);
> 
>     errno = saved_errno;
>     return rc;
> }
> 
> where `seconds' should be initialized with the value of
> `file-access-timeout'.  The cool advantage of this that I see is that
> one can then also selectively `let'-bind different values for
> `file-access-timeout', thus having total control over the use cases in
> which one wants to protect oneself from indefinite hangs.
> 
> Kind regards,
> Alexander
> 
> [1] 
> https://lists.gnu.org/archive/html/help-gnu-emacs/2015-11/msg00251.html

A couple of more ideas:
- I think it is reasonable to actually signal a dedicated error in case 
of the timeout so that API consumers can handle it accordingly to their 
needs.
- It might be worth to also factor this alarm mechanism out into a 
separate macro, e.g. similar to `condition-case', where one could wrap a 
piece of Lisp code into that macro by supplying a timeout and expect it 
to call a timeout handler code in case of timeout:

(with-system-timeout 3
     (do-something)
   (message "%s" "Timed out after 3 seconds..."))

This would also give Lisp developers full control over system related 
interactions.

Regards,
Alexander

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Noam Postavsky]

On 11 June 2018 at 06:27, Alexander Shukaev
<emacs@alexander.shukaev.name> wrote:

> Well, enough rant.  I think I have a proposal how to fix the issue, even
> given the blocking nature of Emacs.  How about introducing a variable
> `file-access-timeout' defaulting to `nil', which would reflect a
> configurable timeout for all access operations (such as `file-readable-p')?
> This would be achieved via `SIGALARM' in the C code, which would protect
> every such operation.  For example,

Does it work? According to some messages on the previous thread you
referenced [1], NFS hangs happen in the kernel so a signal would not
help.

[1]: https://lists.gnu.org/archive/html/help-gnu-emacs/2015-11/msg00266.html

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Alexander Shukaev]

On 2018-06-11 13:59, Noam Postavsky wrote:
> On 11 June 2018 at 06:27, Alexander Shukaev
> <emacs@alexander.shukaev.name> wrote:
> 
>> Well, enough rant.  I think I have a proposal how to fix the issue, 
>> even
>> given the blocking nature of Emacs.  How about introducing a variable
>> `file-access-timeout' defaulting to `nil', which would reflect a
>> configurable timeout for all access operations (such as 
>> `file-readable-p')?
>> This would be achieved via `SIGALARM' in the C code, which would 
>> protect
>> every such operation.  For example,
> 
> Does it work? According to some messages on the previous thread you
> referenced [1], NFS hangs happen in the kernel so a signal would not
> help.
> 
> [1]: 
> https://lists.gnu.org/archive/html/help-gnu-emacs/2015-11/msg00266.html

Go ahead and try:

import os
import signal
import subprocess

class Alarm(Exception):
   pass

def alarm_handler(signum, frame):
   raise Alarm

path = '/mnt/<nfs>'

signal.signal(signal.SIGALRM, alarm_handler)
signal.alarm(3)
try:
   proc = subprocess.call('stat ' + path,
                shell=True,
                stderr=subprocess.PIPE,
                stdout=subprocess.PIPE)
   stdoutdata, stderrdata = proc.communicate()
   signal.alarm(0)
except Alarm:
   print "Timed out after 3 seconds..."

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Noam Postavsky]

On 11 June 2018 at 08:11, Alexander Shukaev
<emacs@alexander.shukaev.name> wrote:

>> Does it work? According to some messages on the previous thread you
>> referenced [1], NFS hangs happen in the kernel so a signal would not
>> help.
>>
>> [1]:
>> https://lists.gnu.org/archive/html/help-gnu-emacs/2015-11/msg00266.html
>
>
> Go ahead and try:

I don't have a hanging NFS server here, so I can't do any useful testing.

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Alexander Shukaev]

On 2018-06-11 14:20, Noam Postavsky wrote:
> On 11 June 2018 at 08:11, Alexander Shukaev
> <emacs@alexander.shukaev.name> wrote:
> 
>>> Does it work? According to some messages on the previous thread you
>>> referenced [1], NFS hangs happen in the kernel so a signal would not
>>> help.
>>> 
>>> [1]:
>>> https://lists.gnu.org/archive/html/help-gnu-emacs/2015-11/msg00266.html
>> 
>> 
>> Go ahead and try:
> 
> I don't have a hanging NFS server here, so I can't do any useful 
> testing.

Oh, I have plenty of suggestions for you.  Just pull the network plug 
out...  :D

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Noam Postavsky]

On 11 June 2018 at 08:22, Alexander Shukaev
<emacs@alexander.shukaev.name> wrote:
> On 2018-06-11 14:20, Noam Postavsky wrote:
>>
>> On 11 June 2018 at 08:11, Alexander Shukaev
>> <emacs@alexander.shukaev.name> wrote:
>>
>>>> Does it work? According to some messages on the previous thread you
>>>> referenced [1], NFS hangs happen in the kernel so a signal would not
>>>> help.
>>>>
>>>> [1]:
>>>> https://lists.gnu.org/archive/html/help-gnu-emacs/2015-11/msg00266.html
>>>
>>>
>>>
>>> Go ahead and try:
>>
>>
>> I don't have a hanging NFS server here, so I can't do any useful testing.
>
>
> Oh, I have plenty of suggestions for you.  Just pull the network plug out...
> :D

I mean, I don't have an NFS server at all. I suppose I could try
installing one...

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Alexander Shukaev]

On 2018-06-11 14:34, Noam Postavsky wrote:
> On 11 June 2018 at 08:22, Alexander Shukaev
> <emacs@alexander.shukaev.name> wrote:
>> On 2018-06-11 14:20, Noam Postavsky wrote:
>>> 
>>> On 11 June 2018 at 08:11, Alexander Shukaev
>>> <emacs@alexander.shukaev.name> wrote:
>>> 
>>>>> Does it work? According to some messages on the previous thread you
>>>>> referenced [1], NFS hangs happen in the kernel so a signal would 
>>>>> not
>>>>> help.
>>>>> 
>>>>> [1]:
>>>>> https://lists.gnu.org/archive/html/help-gnu-emacs/2015-11/msg00266.html
>>>> 
>>>> 
>>>> 
>>>> Go ahead and try:
>>> 
>>> 
>>> I don't have a hanging NFS server here, so I can't do any useful 
>>> testing.
>> 
>> 
>> Oh, I have plenty of suggestions for you.  Just pull the network plug 
>> out...
>> :D
> 
> I mean, I don't have an NFS server at all. I suppose I could try
> installing one...

There are plenty of other ways why those file operations might hang.  
For example, [1].

[1] 
http://unix.stackexchange.com/questions/63071/local-regular-file-causes-stat-or-ls-l-to-hang

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Andreas Schwab]

On Jun 11 2018, Alexander Shukaev <emacs@alexander.shukaev.name> wrote:

> signal.signal(signal.SIGALRM, alarm_handler)
> signal.alarm(3)
> try:
>   proc = subprocess.call('stat ' + path,
>                shell=True,
>                stderr=subprocess.PIPE,
>                stdout=subprocess.PIPE)
>   stdoutdata, stderrdata = proc.communicate()
>   signal.alarm(0)
> except Alarm:
>   print "Timed out after 3 seconds..."

How do you know that 3 seconds is enough?

Andreas.

-- 
Andreas Schwab, SUSE Labs, schwab@suse.de
GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE  1748 E4D4 88E3 0EEA B9D7
"And now for something completely different."

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Alexander Shukaev]

On 2018-06-11 14:40, Andreas Schwab wrote:
> On Jun 11 2018, Alexander Shukaev <emacs@alexander.shukaev.name> wrote:
> 
>> signal.signal(signal.SIGALRM, alarm_handler)
>> signal.alarm(3)
>> try:
>>   proc = subprocess.call('stat ' + path,
>>                shell=True,
>>                stderr=subprocess.PIPE,
>>                stdout=subprocess.PIPE)
>>   stdoutdata, stderrdata = proc.communicate()
>>   signal.alarm(0)
>> except Alarm:
>>   print "Timed out after 3 seconds..."
> 
> How do you know that 3 seconds is enough?
> 
> Andreas.

You don't know.  You just decide that it's maximum tolerable for 
you/your setup/hardware/connection/preferences/whatever, otherwise you 
are 99.(9)% sure that something is wrong somewhere with your system, but 
you don't give up your Emacs instance for that and rather get indicated 
that there might be a potential problem.

P.S. How did people know that 2 hours would be enough for TCP keepalive?

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Eli Zaretskii]

> Date: Mon, 11 Jun 2018 14:46:35 +0200
> From: Alexander Shukaev <emacs@alexander.shukaev.name>
> Cc: Emacs-devel <emacs-devel-bounces+emacs=alexander.shukaev.name@gnu.org>,
> 	Noam Postavsky <npostavs@gmail.com>, Emacs developers <emacs-devel@gnu.org>
> 
> On 2018-06-11 14:40, Andreas Schwab wrote:
> > On Jun 11 2018, Alexander Shukaev <emacs@alexander.shukaev.name> wrote:
> > 
> >> signal.signal(signal.SIGALRM, alarm_handler)
> >> signal.alarm(3)
> >> try:
> >>   proc = subprocess.call('stat ' + path,
> >>                shell=True,
> >>                stderr=subprocess.PIPE,
> >>                stdout=subprocess.PIPE)
> >>   stdoutdata, stderrdata = proc.communicate()
> >>   signal.alarm(0)
> >> except Alarm:
> >>   print "Timed out after 3 seconds..."
> > 
> > How do you know that 3 seconds is enough?
> > 
> > Andreas.
> 
> You don't know.  You just decide that it's maximum tolerable for 
> you/your setup/hardware/connection/preferences/whatever, otherwise you 
> are 99.(9)% sure that something is wrong somewhere with your system, but 
> you don't give up your Emacs instance for that and rather get indicated 
> that there might be a potential problem.

I think there's more here than meets the eye.  Sure, it's quite easy
to come up with a toy program that uses SIGALRM to time out a system
call that went awry.  But Emacs is not a toy program, so doing that
has complications, even if we will come up with a suitable number of
seconds to wait (which ain't easy, since some I/O calls could really
need a long time, or example reading a large file or directory).

Here are some complications we should keep in mind:

  . Emacs already uses SIGALRM for different purposes, see atimer.c.
    Reusing it for this issue will need some complex logic, to avoid
    breaking the features that use SIGALRM now.
  . You tried this with a single 'stat' call, but that's just the tip
    of the iceberg.  Typically, Emacs will need to read a file after
    it found it readable, and we normally do that in a way that keeps
    looping as long as the system call was interrupted by signals, see,
    e.g., emacs_intr_read.  Then setting up an alarm clock will not
    help if 'read' hangs, we will just loop forever.
  . We usually deliver signals to the main thread, so if the code that
    hangs happens to run in a non-main thread (recall that Emacs 26
    has threads), it will be somewhat tricky, to say the least, to
    deliver signal there.
  . Even if we somehow succeed to interrupt the hang by a signal, it's
    not clear whether it's safe to continue running the session --
    there's a reason why we stopped doing non-trivial stuff in signal
    handlers.  It may be that the only sensible thing is to shut down,
    and in that case, what did we gain, exactly?
  . This technique is non-portable to MS-Windows.

There are probably other complications.

All in all, I'd be much happier if we could interrupt such hangs,
e.g. by C-g, as Stefan points out (on a TTY frame, this should already
be possible in many cases, since C-g there generates SIGINT).  But I'm
not sure this would be possible in general.  Maybe Paul will have some
ideas.

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Paul Eggert]

Eli Zaretskii wrote:
> I'd be much happier if we could interrupt such hangs,
> e.g. by C-g, as Stefan points out (on a TTY frame, this should already
> be possible in many cases, since C-g there generates SIGINT).  But I'm
> not sure this would be possible in general.  Maybe Paul will have some
> ideas.

I agree with you. Luckily, it appears that there's nothing Emacs can practically 
do to address this question for NFS mounts in general, so we needn't worry about 
this for Emacs in particular.

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Davis Herring]

> signal.alarm(3)
> try: 
>   proc = subprocess.call('stat ' + path,
>                shell=True,
>                stderr=subprocess.PIPE,
>                stdout=subprocess.PIPE)

You're sending SIGALRM to the _parent_, not the (possibly-frozen) 
stat(1) child.  Of course that works; you're just interrupting normal 
pipe I/O with (or wait(2) on) the child.

Emacs can't possibly do all of its file operations in subprocesses.  A 
complete redesign might allow the "process per tab" model used in some 
modern browsers, but that would break much of existing Lisp.  It might 
just be possible to have a "file server process" that could be killed 
and reincarnated as needed, but I wouldn't want to promise much about 
performance (and support for concurrent I/O).

Incidentally, don't mix call() and communicate(), and avoid 
"shell=True", especially when the replacement is easier:

   proc = subprocess.Popen(['stat', path],
                 stderr=subprocess.PIPE,
                 stdout=subprocess.PIPE)

[From a later message:]

> There are plenty of other ways why those file operations might hang.
> For example, [1].
> 
> [1] http://unix.stackexchange.com/questions/63071/local-regular-file-causes-stat-or-ls-l-to-hang

The (single, accepted) answer there says that it was an LDAP issue 
looking up user/group names.  That's not a file operation at all.

Davis

-- 
This product is sold by volume, not by mass.  If it appears too dense or 
too sparse, it is because mass-energy conversion has occurred during 
shipping.

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Perry E. Metzger]

I think it's a serious mistake for emacs to try to pretend it can fix
the problem of a hung hard NFS mount for people. This belongs at the
OS layer, not at the level of emacs, and if someone wants the
semantics of soft mounts, they should use soft mounts. Emacs
shouldn't be in the business of special casing this, and indeed, on
many OSes where a hard mount will hang a process in an unkillable
kernel wait, it _can't_ fix it.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Stefan Monnier]

> this discussion.  I still find this issue very disruptive.  Yet another
> example would be `recentf-cleanup' which is in my case triggered on Emacs
> start up, when the file comes from stale NFS, the corresponding
> `file-readable-p' down the stack will hang indefinitely, and there would be
> no way to unfreeze it apart from issuing 'kill -9' to that Emacs instance.

Indeed stale NFS mounts can be problematic.  As you can see from
Andreas's reaction the obvious first answer is that it's a general
problem, so I think we first need to understand what makes it different
in the context of Emacs.

I don't use NFS much these days, but IIRC there are basically two
different ways to do NFS mounts: "hard" and "soft".  Back when I used
it, "hard" was used with "intr" so you could interrupt frozen processes,
but from what I read, the linux kernel's NFS client nowadays doesn't
support this any more, so a process waiting for a hard-mounted NFS
server can only be interrupted with a SIGKILL.

So some questions, to better understand what are our options:

- It seems your unreliable NFS server is mounted "hard" rather than "soft".
  Why is that?  "man mount" on my Debian machine doesn't find any "hard"
  or "soft" options, so has the soft-mount option disappeared?  What are
  applications usually expected to do when accessing a stale NFS server?

- You can "kill -9" is the only option, yet you seem to also say that
  SIGALRM does work.  The two statements seem contradictory.
  What is the set of signals which work, really?
  E.g. Does `kill -USR1` work (with debug-on-event)?
  Maybe the issue here is that Emacs handles C-g via polling rather than
  via interrupts, and we should refine that polling such that it handles
  such "C-g while in the middle of a long-running file access syscall"?

> Well, enough rant.  I think I have a proposal how to fix the issue, even
> given the blocking nature of Emacs.  How about introducing a variable
> `file-access-timeout' defaulting to `nil', which would

If at all possible, I think I'd prefer to let the user interrupt with
C-g rather than rely on some kind of timeout.

Reading the original thread, you seem to say that this mostly affects
"dired" operation, and that not only can it hang, but it can also
be slow.

So a few more questions:

- In my experience dired-like operations over NFS servers should obey
  either normal speed or hang (if the server is unavailable) but "slow"
  is not something I'd expect.  Do you know why it's sometimes slow?
  Is your NFS server itself "far/slow"?  Is the slowness due to some
  automount (i.e. it's slow because of the time taken to perform the
  mount itself)?

- Does this slow/hanging behavior appear only in dired?  Does it only
  affect Emacs when using dired on a directory that's indeed on a NFS
  server, or does it affect accesses which don't obviously require
  NFS access?


        Stefan

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Mike Kupfer]

Stefan Monnier wrote:

> - It seems your unreliable NFS server is mounted "hard" rather than "soft".
>   Why is that?  "man mount" on my Debian machine doesn't find any "hard"
>   or "soft" options, so has the soft-mount option disappeared?

Try nfs(5) for a description of "hard" and "soft".  If you don't have
that man page installed, it should be on the web somewhere.

Hard mounts are the default because soft mounts usually cause more
problems than they solve.  

>   What are
>   applications usually expected to do when accessing a stale NFS server?

Could we start by getting a more precise definition of "stale NFS
server"?  I can think of at least 3 different situations that might be
meant:

  - the server is down temporarily
  - the server is down permanently
  - the server is up, but it no longer exports that filesystem

And if the mount is managed by the automounter, the application behavior
will depend on whether the mount has already been established.  (I
think.  I'm very familiar with the Solaris NFS implementation, but not
the Linux implementation.)

mike

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Stefan Monnier]

>> - It seems your unreliable NFS server is mounted "hard" rather than "soft".
>>   Why is that?  "man mount" on my Debian machine doesn't find any "hard"
>>   or "soft" options, so has the soft-mount option disappeared?
> Try nfs(5) for a description of "hard" and "soft".

[ Ah, indeed.  Not sure why it's not in `mount` even though that page also
  claims to have NFS-specific options.  In any case, this confirms tht
  soft mounts still exist.  Thanks.  ]

> Hard mounts are the default because soft mounts usually cause more
> problems than they solve.

Indeed, but the OP's description of the behavior he's trying to get from
Emacs suggests maybe he'd be satisfied by a `soft` mount.

>>   What are
>>   applications usually expected to do when accessing a stale NFS server?
>
> Could we start by getting a more precise definition of "stale NFS
> server"?  I can think of at least 3 different situations that might be
> meant:
>   - the server is down temporarily
>   - the server is down permanently
>   - the server is up, but it no longer exports that filesystem

- the server is up, but the connection is temporarily not available.
  [ where "temporarily" can last a long time.  ]
- buggy NFS (the server is up and exporting, the connection as well,
  yet the NFS client stalls for some unknown reason).


        Stefan

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Alexander Shukaev]

On 06/11/2018 12:27 PM, Alexander Shukaev wrote:
> Hi Everyone,
> 
> 
> I initiated a discussion back in 2015 [1] about fragility of Emacs in 
> terms of filesystem operations on stale NFS.  No solution actually came 
> out of this discussion.  I still find this issue very disruptive.  Yet 
> another example would be `recentf-cleanup' which is in my case triggered 
> on Emacs start up, when the file comes from stale NFS, the corresponding 
> `file-readable-p' down the stack will hang indefinitely, and there would 
> be no way to unfreeze it apart from issuing 'kill -9' to that Emacs 
> instance.  Don't you people find it unacceptable for the daily usage? 
> Well, I do.  Such hangs always disrupt daily work and require quite some 
> time to track them down as they are not Lisp-debuggable with e.g. <C-g> 
> in a straightforward way (these are dead hangs from C code, where even 
> attaching a GDB does not work).
> 
> Well, enough rant.  I think I have a proposal how to fix the issue, even 
> given the blocking nature of Emacs.  How about introducing a variable 
> `file-access-timeout' defaulting to `nil', which would reflect a 
> configurable timeout for all access operations (such as 
> `file-readable-p')?  This would be achieved via `SIGALARM' in the C 
> code, which would protect every such operation.  For example,
> 
> #include <sigaction.h>
> #include <sys/stat.h>
> #include <unistd.h>
> #include <string.h>
> 
> static void alarm_handler(int sig)
> {
>      return;
> }
> 
> int emacs_stat(const char* path, struct stat* s, unsigned int seconds)
> {
>      struct sigaction newact;
>      struct sigaction oldact;
> 
>      memset(&newact, 0, sizeof(newact));
>      memset(&oldact, 0, sizeof(oldact));
> 
>      sigemptyset(&newact.sa_mask);
> 
>      newact.sa_flags   = 0;
>      newact.sa_handler = alarm_handler;
>      sigaction(SIGALRM, &newact, &oldact);
> 
>      alarm(seconds);
> 
>      errno                 = 0;
>      const int rc          = stat(path, s);
>      const int saved_errno = errno;
> 
>      alarm(0);
>      sigaction(SIGALRM, &oldact, NULL);
> 
>      errno = saved_errno;
>      return rc;
> }
> 
> where `seconds' should be initialized with the value of 
> `file-access-timeout'.  The cool advantage of this that I see is that 
> one can then also selectively `let'-bind different values for 
> `file-access-timeout', thus having total control over the use cases in 
> which one wants to protect oneself from indefinite hangs.
> 
> Kind regards,
> Alexander
> 
> [1] https://lists.gnu.org/archive/html/help-gnu-emacs/2015-11/msg00251.html
> 

Today I realized that the following code

import os
import signal

class Alarm(Exception):
   pass

def alarm_handler(signum, frame):
   raise Alarm

path = '/mnt/<nfs>'

signal.signal(signal.SIGALRM, alarm_handler)
signal.alarm(3)
try:
   os.stat(path)
   signal.alarm(0)
except Alarm:
   print("Timed out after 3 seconds...")

does not time out in case of 'hard' mounting, which means that the only 
way to time this case out reliably is to spawn a child which attempts to 
perform 'stat' and is then killed by the parent if timed out (similar to 
how 'lsof' does it).  I'm sure such a complex technique would be 
unacceptable for Emacs.

Although I don't like the current behavior as it looks like an editor 
vulnerability to me, which could be (either intentionally or 
unintentionally) used as a potential attack to hang the editor merely by 
pulling a network plug out, I have to admit that we can probably wrap up 
this discussion here since there is not much that could be done by Emacs 
itself to defend.

Regards,
Alexander

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Yuri Khan]

On Wed, Jun 13, 2018 at 5:46 PM Alexander Shukaev
<emacs@alexander.shukaev.name> wrote:

> […] the only
> way to time this case out reliably is to spawn a child which attempts to
> perform 'stat' and is then killed by the parent if timed out (similar to
> how 'lsof' does it).  I'm sure such a complex technique would be
> unacceptable for Emacs.

Isn’t that what TRAMP does behind the scene, though?

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Stefan Monnier]

>> way to time this case out reliably is to spawn a child which attempts to
>> perform 'stat' and is then killed by the parent if timed out (similar to
>> how 'lsof' does it).  I'm sure such a complex technique would be
>> unacceptable for Emacs.
> Isn’t that what TRAMP does behind the scene, though?

Indeed, and someone could write an "NFS file name handler" so you could
use something like "/nfs:<host>/...".  It might even be reasonably easy
for Tramp to provide that (since I believe GVFS supports NFS access).
But it won't help when Emacs accesses to NFS files via normal OS calls
rather than via Tramp.


        Stefan

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Yuri Khan]

On Wed, Jun 13, 2018 at 9:34 PM Stefan Monnier <monnier@iro.umontreal.ca> wrote:

> someone could write an "NFS file name handler" so you could
> use something like "/nfs:<host>/...".  It might even be reasonably easy
> for Tramp to provide that (since I believe GVFS supports NFS access).

Alternatively, one could probably use TRAMP over su as oneself to
access NFS-mounted files. This would presumably enjoy the same process
separation.

Re: Emacs Hangs on Filesystem Operations on Stale NFS

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

Hi Stefan,

>> Isn’t that what TRAMP does behind the scene, though?
>
> Indeed, and someone could write an "NFS file name handler" so you could
> use something like "/nfs:<host>/...".  It might even be reasonably easy
> for Tramp to provide that (since I believe GVFS supports NFS access).

I'm not quite sure whether your proposal is just rhetoric. In case you
mean it seriously: GVFS ought to have an nfs backend, called
gvfsd-nfs. See gvfs(7). But it doesn't seem to be installed by default,
at least Ubuntu's gvfs-backends package does not contain it. (It seems
to be distributed by Debian's gvfs-backends package, 'tho).

However, I don't believe it is necessary to use GVFS. Something like the
mock method in tramp-test.el, slightly adapted, would serve already, I
believe.

> But it won't help when Emacs accesses to NFS files via normal OS calls
> rather than via Tramp.

Sigh.

>         Stefan

Best regards, Michael.