inotify-based file notifications missing sometimes

inotify-based file notifications missing sometimes

[Dima Kogan]

Hi.

I ran into an issue with inotify-base file notifications when looking at
auto-revert stuff from an earlier thread. To show the issue, I build
the latest emacs from source using

 ./configure --with-file-notification=inotify

I then run

 ./emacs --eval "`cat  /tmp/tstnotify.el`" -Q -nw

with tstnotify.el being

 (progn
   (require 'filenotify)

   (dolist (fil '("/tmp/tst1" "/tmp/tst2"))
     (file-notify-add-watch fil  '(change attribute-change)
                            (lambda (event)
                              (message "notify event %s" event)))
     (find-file fil))
   (switch-to-buffer "*Messages*"))


Here I ask for notifications for two files, and print out the events as
they come in. While emacs is running this way, I modify those two files
using an external tool. I would expect to see modification events for
both of these files, but I only see them for one of the files.

The issue is that emacs doesn't ask inotify to look at the two files, it
asks it to look at the containing directory (/tmp in both cases),
filtering out all the files we don't care about later. inotify is being
intelligent, notices that we're asking to monitor the same object twice
(/tmp), and simply returns the same descriptor a second time. Emacs is
assuming a unique descriptor, and the result is that we only get
notifications for one file.

The main question is, why are monitoring the directory instead of the
objects we care about? I'm assuming that it has something to do with the
other notification backends we could possibly use. For inotify
specifically, it'd be much easier to simply look at the files (and this
bug wouldn't exist then).

Proposed solutions? I haven't filed this into the bug tracker yet since
I presume some discussion will happen first.

dima

Re: inotify-based file notifications missing sometimes

[Eli Zaretskii]

> From: Dima Kogan <lists@dima.secretsauce.net>
> Date: Fri, 24 Oct 2014 23:17:57 -0700
> 
> The main question is, why are monitoring the directory instead of the
> objects we care about?

AFAIR, it was to cover the use case described here:

  http://lists.gnu.org/archive/html/emacs-devel/2013-01/msg00178.html

Michael (CC'ed) might remember it better and in more detail.

Re: inotify-based file notifications missing sometimes

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> Michael (CC'ed) might remember it better and in more detail.

I've read the thread and it is on my todo (like all the Tramp related
messages). I'm on a business trip, abroad, so I'm somehow
restricted. Hope to have the time for working on this over the weekend.

Best regards, Michael.

Re: inotify-based file notifications missing sometimes

[Eli Zaretskii]

> From: Dima Kogan <lists@dima.secretsauce.net>
> Date: Fri, 24 Oct 2014 23:17:57 -0700
> 
> I then run
> 
>  ./emacs --eval "`cat  /tmp/tstnotify.el`" -Q -nw
> 
> with tstnotify.el being
> 
>  (progn
>    (require 'filenotify)
> 
>    (dolist (fil '("/tmp/tst1" "/tmp/tst2"))
>      (file-notify-add-watch fil  '(change attribute-change)
>                             (lambda (event)
>                               (message "notify event %s" event)))
>      (find-file fil))
>    (switch-to-buffer "*Messages*"))
> 
> 
> Here I ask for notifications for two files, and print out the events as
> they come in. While emacs is running this way, I modify those two files
> using an external tool. I would expect to see modification events for
> both of these files, but I only see them for one of the files.

But you are not saying that having more than one file in the same
directory under auto-revert-mode doesn't work for this reason, are
you?  Because I just tried, and it does work, because autorevert.el
does handle this situation.

> Proposed solutions?

I think it would help if you state explicitly what is the problem,
that you want to be solved, and why.

Just to clarify: filenotify.el is infrastructure that currently
(AFAIK) has only one user -- autorevert.el.  At the time the file
notifications were introduced, there were many discussions about how
best to design this infrastructure.  Eventually, the conclusion was
that we should make these decisions as we go, as we add user-level
features based on the notifications.  So for now, filenotify.el
reflects the needs of its single user, and with that user it does its
job.  If there are additional needs and goals not covered by that,
they should be explicitly stated, and then filenotify.el might need to
be extended/modified to cover them.

Re: inotify-based file notifications missing sometimes

[Dima Kogan]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Dima Kogan <lists@dima.secretsauce.net>
>> Date: Fri, 24 Oct 2014 23:17:57 -0700
>> 
>> Here I ask for notifications for two files, and print out the events as
>> they come in. While emacs is running this way, I modify those two files
>> using an external tool. I would expect to see modification events for
>> both of these files, but I only see them for one of the files.
>
> But you are not saying that having more than one file in the same
> directory under auto-revert-mode doesn't work for this reason, are
> you?  Because I just tried, and it does work, because autorevert.el
> does handle this situation.

I'm not saying that largely because this thread is explicitly not about
auto-revert-mode. autorevert.el likely handles this because it was
originally written to not use inotify at all (and is still that way for
remote files), and likely makes more thorough checks. I have not looked
into that at all. The patch in the other thread makes auto-revert-mode
event based, and is broken by the issue described in this thread.


>> Proposed solutions?
>
> I think it would help if you state explicitly what is the problem,
> that you want to be solved, and why.

The problem is that we provide utility functions in filenotify.el that
claim to do something and do not do it. More to the point of what I'm
doing, I want to fix the event-based auto-revert, as implemented in the
patch I sent to the list a few hours ago. The patch is fine, I think,
but the issue described HERE breaks it.


> Just to clarify: filenotify.el is infrastructure that currently
> (AFAIK) has only one user -- autorevert.el.  At the time the file
> notifications were introduced, there were many discussions about how
> best to design this infrastructure.  Eventually, the conclusion was
> that we should make these decisions as we go, as we add user-level
> features based on the notifications.  So for now, filenotify.el
> reflects the needs of its single user, and with that user it does its
> job.  If there are additional needs and goals not covered by that,
> they should be explicitly stated, and then filenotify.el might need to
> be extended/modified to cover them.

OK. That is reasonable. I have no problems with the filenotify.el API at
all, it's just that the API description is not being respected.

By the way, the other filenotify.el backend for linux (gfile) has an
issue as well. Running the same test with it DOES produce events for
both files, but events come throught only when the user touches emacs.
So when the files are modified, nothing shows up in *Messages*, but
pressing any key in the emacs window makes them pop up. I haven't looked
into the cause of that yet, but intend to shortly.

dima

Re: inotify-based file notifications missing sometimes

[Eli Zaretskii]

> From: Dima Kogan <lists@dima.secretsauce.net>
> Cc: emacs-devel@gnu.org
> Date: Sat, 25 Oct 2014 01:27:43 -0700
> 
> > But you are not saying that having more than one file in the same
> > directory under auto-revert-mode doesn't work for this reason, are
> > you?  Because I just tried, and it does work, because autorevert.el
> > does handle this situation.
> 
> I'm not saying that largely because this thread is explicitly not about
> auto-revert-mode. autorevert.el likely handles this because it was
> originally written to not use inotify at all (and is still that way for
> remote files), and likely makes more thorough checks.

The file-notification support in autorevert _was_ added when inotify
became supported.  According to my reading of the related code, its
handling of notifications is explicitly designed to handle the
situation where notifications about several files come through the
same watch descriptor.  This code was not in autorevert.el before file
notifications were added to Emacs.

> The patch in the other thread makes auto-revert-mode event based,
> and is broken by the issue described in this thread.

Nitpicking: nothing in Emacs is event based (well, maybe except of a
few signals; but even there, signals that need non-trivial processing
just set a flag that is examined later).  Everything goes through the
Emacs input queue.  The only reason you might think file notifications
(and other events) are handled "immediately" is because Emacs checks
the input queue very frequently, at least when it's not busy with some
lengthy calculation.

> The problem is that we provide utility functions in filenotify.el that
> claim to do something and do not do it. More to the point of what I'm
> doing, I want to fix the event-based auto-revert, as implemented in the
> patch I sent to the list a few hours ago. The patch is fine, I think,
> but the issue described HERE breaks it.

I think the solution to that should be in your patch.

We could, of course, consider modifications in filenotify.el as well,
but my gut feeling, based on past discussions, is that it would be
more ugly, due to the need to support several different use cases and
several back-ends.

> > Just to clarify: filenotify.el is infrastructure that currently
> > (AFAIK) has only one user -- autorevert.el.  At the time the file
> > notifications were introduced, there were many discussions about how
> > best to design this infrastructure.  Eventually, the conclusion was
> > that we should make these decisions as we go, as we add user-level
> > features based on the notifications.  So for now, filenotify.el
> > reflects the needs of its single user, and with that user it does its
> > job.  If there are additional needs and goals not covered by that,
> > they should be explicitly stated, and then filenotify.el might need to
> > be extended/modified to cover them.
> 
> OK. That is reasonable. I have no problems with the filenotify.el API at
> all, it's just that the API description is not being respected.

I guess some documentation fixes are in order, then.

> By the way, the other filenotify.el backend for linux (gfile) has an
> issue as well. Running the same test with it DOES produce events for
> both files, but events come throught only when the user touches emacs.
> So when the files are modified, nothing shows up in *Messages*, but
> pressing any key in the emacs window makes them pop up. I haven't looked
> into the cause of that yet, but intend to shortly.

Welcome to the black art known as file notifications.  Each of the 3
available back-ends has its own "issues".  The current version of
filenotify.el tries to paper over them, but I'm quite sure it doesn't
yet do a perfect job, except when used through autorevert.el.  We
simply don't yet have enough experience with that.

Re: inotify-based file notifications missing sometimes

[Dima Kogan]

I posted this issue at the bug tracker:

http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18880