gfile-based file notifications are not immediate

gfile-based file notifications are not immediate

[Dima Kogan]

Hi.

In the last few days I started several threads about various details of
file notification and auto-revert behavior. This one is about some
incorrect behavior of notifications using the 'gfile' (gio from glib)
backend.

As in the inotify thread, I set up a simple test. I built emacs using

 ./configure --with-file-notification=inotify

I then ran

 ./emacs --eval "`cat  /tmp/tstnotify.el`" -Q -nw

with tstnotify.el being

 (progn
   (require 'filenotify)

   (dolist (fil '("/tmp/tst1" "/tmp/tst2"))
     (file-notify-add-watch fil  '(change attribute-change)
                            (lambda (event)
                              (message "notify event %s" event)))
     (find-file fil))
   (switch-to-buffer "*Messages*"))


Here I ask for notifications for two files, and print out the events as
they come in. While emacs is running this way, I modify those two files
using an external tool. I would expect to see modification events for
both of these files, as soon as these modifications happen. Instead, the
notifications come in either 30 seconds later, or whenever anything goes
through emacs's input queue. So the notification is greatly delayed if
the user is not touching their emacs.

The reason is that when emacs is idle, it's blocking in the pselect()
call in xg_select() in process.c. pselect() is an operating-system call,
not a glib one. Later on in xg_select() there's some code to kick the
glib event loop, but as long as we're sitting in the pselect(), that
secondary event loop remains untouched.

Even if you kick the event loop in time, the notification is still
delayed a few seconds (I verified this with some test code, outside of
emacs; can post if anybody wants it). THIS is a separate issue that is
likely a bug in glib. On Linux it uses inotify internally, so if one was
using Linux, would there be any reason to use this notification scheme
instead of using inotify directly? Are there OSs where emacs supports
notifications ONLY through glib?

Once again, I'm not posting to the bug tracker yet.

dima

Re: gfile-based file notifications are not immediate

[Michael Albinus]

Dima Kogan <lists@dima.secretsauce.net> writes:

> Hi.

Hi Dima,

[I will come to the other issues later when time / my environment fits better]

> On Linux it uses inotify internally, so if one was
> using Linux, would there be any reason to use this notification scheme
> instead of using inotify directly?

IIRC, inotify doesn't work on mounted directories. gfile uses what's
appropriate; on mounted directories (again, IIRC) it polls for changes
of the supervised file/directory.

> Are there OSs where emacs supports
> notifications ONLY through glib?

On BSD-like systems, including MacOS X, you can use only gfile these
days from Emacs. There exist a native library kqueue which ought to
implement file notifications, but nobody wrote an Emacs integration so
far. And AFAIU, gfile uses internally kqueue on those systems (not
tested by myself).

I also don't remember whether there is inotify support for Cygwin.

> dima

Best regards, Michael.

Re: gfile-based file notifications are not immediate

[Ken Brown]

On 10/26/2014 3:55 AM, Michael Albinus wrote:
> Dima Kogan <lists@dima.secretsauce.net> writes:
> I also don't remember whether there is inotify support for Cygwin.

No, Cygwin uses Glib for file notification.

Ken

Re: gfile-based file notifications are not immediate

[Rüdiger Sonderfeld]

On Sunday 26 October 2014 08:55:31 Michael Albinus wrote:
> > Are there OSs where emacs supports
> > notifications ONLY through glib?
> 
> On BSD-like systems, including MacOS X, you can use only gfile these
> days from Emacs. There exist a native library kqueue which ought to
> implement file notifications, but nobody wrote an Emacs integration so
> far. And AFAIU, gfile uses internally kqueue on those systems (not
> tested by myself).

I was looking at adding kqueue support as well when I wrote the inotify 
implementation.  However, the kqueue API is so bad that it makes even the w32 
API look good in comparison.  (kqueue is probably also the reason why the 
gfile API is so limited compared to inotify.  They simply can't implement a 
lot of it on BSD.  We all have to suffer because of kqueue even if we use 
saner systems.)

Maybe on GNU/Linux the default could be set to inotify or would this break 
using gfile on remote systems via tramp?

Regards,
Rüdiger

Re: gfile-based file notifications are not immediate

[Stefan Monnier]

> I was looking at adding kqueue support as well when I wrote the inotify 
> implementation.

Even if kqueue works, I think it's better not to try and support it
(just like it would be better not to support inotify) if there's
a library (such as glib) which we can use and which shields us from all
those issues.

Let's not reinvent the wheel.


        Stefan

Re: gfile-based file notifications are not immediate

[Dima Kogan]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Even if kqueue works, I think it's better not to try and support it
> (just like it would be better not to support inotify) if there's
> a library (such as glib) which we can use and which shields us from all
> those issues.

By the way, if in some hypothetical future we drop support for inotify
in favor of glib, then we don't need to do the thing where we monitor a
file's directory instead of the file itself, as described here:

  http://lists.gnu.org/archive/html/emacs-devel/2013-01/msg00178.html

glib already does this, so emacs doing too is an unnecessary
complication and a form of wheel-reinventing.

Re: gfile-based file notifications are not immediate

[Stefan Monnier]

>> Even if kqueue works, I think it's better not to try and support it
>> (just like it would be better not to support inotify) if there's
>> a library (such as glib) which we can use and which shields us from all
>> those issues.
> By the way, if in some hypothetical future we drop support for inotify
> in favor of glib, then we don't need to do the thing where we monitor a
> file's directory instead of the file itself, as described here:

>   http://lists.gnu.org/archive/html/emacs-devel/2013-01/msg00178.html

> glib already does this, so emacs doing too is an unnecessary
> complication and a form of wheel-reinventing.

Interesting.  Another point in favor of dropping inotify support,


        Stefan

Re: gfile-based file notifications are not immediate

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> By the way, if in some hypothetical future we drop support for inotify
>> in favor of glib, then we don't need to do the thing where we monitor a
>> file's directory instead of the file itself, as described here:
>
>>   http://lists.gnu.org/archive/html/emacs-devel/2013-01/msg00178.html
>
>> glib already does this, so emacs doing too is an unnecessary
>> complication and a form of wheel-reinventing.
>
> Interesting.  Another point in favor of dropping inotify support,

There are people who do not want glib linked with Emacs.

>         Stefan

Best regards, Michael.

Re: gfile-based file notifications are not immediate

[Stefan Monnier]

> There are people who do not want glib linked with Emacs.

That's not a problem: those people simply won't get file notifications
(just like they won't get Gtk menus and scrollbars, and they won't get
D-Bus support and ...).


        Stefan

Re: gfile-based file notifications are not immediate

[Óscar Fuentes]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> There are people who do not want glib linked with Emacs.
>
> That's not a problem: those people simply won't get file notifications

Are you suggesting that auto-revert and other associated features would
not be present for those who chose to avoid glib?

> (just like they won't get Gtk menus and scrollbars, and they won't get
> D-Bus support and ...).

Those are just appearence (Gtk menus/scrollbars) and external interface
(D-Bus) features. auto-revert is a core feature for any serious editor.

Re: gfile-based file notifications are not immediate

[Stefan Monnier]

> Are you suggesting that auto-revert and other associated features would
> not be present for those who chose to avoid glib?

Please remember that auto-revert-mode predates file-notifications, and
even now that we have file-notification support, auto-revert doesn't
need it to work.


        Stefan

Re: gfile-based file notifications are not immediate

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> There are people who do not want glib linked with Emacs.
>
> That's not a problem: those people simply won't get file notifications
> (just like they won't get Gtk menus and scrollbars, and they won't get
> D-Bus support and ...).

I meant to say: There are people who want file notifications but do not
want glib linked with Emacs.

>         Stefan

Best regards, Michael.

Re: gfile-based file notifications are not immediate

[Stefan Monnier]

> I meant to say: There are people who want file notifications but do not
> want glib linked with Emacs.

Yup, people want all kinds of things.  That doesn't mean we necessarily
have to satisfy their desires.


        Stefan

Re: gfile-based file notifications are not immediate

[Perry E. Metzger]

On Wed, 29 Oct 2014 09:56:41 -0400 Stefan Monnier
<monnier@iro.umontreal.ca> wrote:
> > There are people who do not want glib linked with Emacs.
> 
> That's not a problem: those people simply won't get file
> notifications (just like they won't get Gtk menus and scrollbars,
> and they won't get D-Bus support and ...).

Windows and OS X users, etc, might not want to link glib in to their
editor, since they don't need it for their GUI.

-- 
Perry E. Metzger		perry@piermont.com

Re: gfile-based file notifications are not immediate

[Stefan Monnier]

> Windows and OS X users, etc, might not want to link glib in to their
> editor, since they don't need it for their GUI.

AFAIK those user can't use the inotify support anyway, so rmeoving
inotify support would make no difference to them.


        Stefan

Re: gfile-based file notifications are not immediate

[Perry E. Metzger]

On Thu, 30 Oct 2014 12:51:06 -0400 Stefan Monnier
<monnier@IRO.UMontreal.CA> wrote:
> > Windows and OS X users, etc, might not want to link glib in to
> > their editor, since they don't need it for their GUI.
> 
> AFAIK those user can't use the inotify support anyway, so rmeoving
> inotify support would make no difference to them.

Certainly true, though both Windows and OS X have file change
monitoring APIs. Perhaps Emacs can define some sort of higher level
API that each port of Emacs can implement (or not) as part of the
system specific code for the platform.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: gfile-based file notifications are not immediate

[Michael Albinus]

"Perry E. Metzger" <perry@piermont.com> writes:

> Perhaps Emacs can define some sort of higher level
> API that each port of Emacs can implement (or not) as part of the
> system specific code for the platform.

This high level API exist and is called filenotify.el. In Emacs 24.4, it
supports inotify, gfile (glib), and the MS Windows file notification
library.

If people on BSD/OS X do not want to use glib, somebody shall write
kqueue support. But it is said to be a hassle, so I wouldn't recommend
it.

> Perry

Best regards, Michael.

Re: gfile-based file notifications are not immediate

[Perry E. Metzger]

On Thu, 30 Oct 2014 18:40:29 +0100 Michael Albinus
<michael.albinus@gmx.de> wrote:
> "Perry E. Metzger" <perry@piermont.com> writes:
> 
> > Perhaps Emacs can define some sort of higher level
> > API that each port of Emacs can implement (or not) as part of the
> > system specific code for the platform.
> 
> This high level API exist and is called filenotify.el. In Emacs
> 24.4, it supports inotify, gfile (glib), and the MS Windows file
> notification library.
> 
> If people on BSD/OS X do not want to use glib, somebody shall write
> kqueue support. But it is said to be a hassle, so I wouldn't
> recommend it.

kqueue is actually quite straightforward to use. It is just a little
"different". I don't know how it would hook in to Emacs's low level
event loop though, I'll have to look at that.

I'm mostly on OS X these days, maybe I'll have a look at doing the
work, though if someone else has an itch to do it, they'll almost
certainly finish before me...

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: gfile-based file notifications are not immediate

[Stefan Monnier]

> If people on BSD/OS X do not want to use glib, somebody shall write
> kqueue support. But it is said to be a hassle,

.. and a waste of time since glib already does it for us,


        Stefan

Re: gfile-based file notifications are not immediate

[Michael Albinus]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

>> If people on BSD/OS X do not want to use glib, somebody shall write
>> kqueue support. But it is said to be a hassle,
>
> .. and a waste of time since glib already does it for us,

I agree with you. But *if* somebody writes native kqueue support,
because he doesn't want to touch glib on OS X: would you as Emacs
maintainer reject this contribution?

>         Stefan

Best regards, Michael.

Re: gfile-based file notifications are not immediate

[Perry E. Metzger]

On Thu, 30 Oct 2014 19:11:55 +0100 Michael Albinus
<michael.albinus@gmx.de> wrote:
> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
> 
> >> If people on BSD/OS X do not want to use glib, somebody shall
> >> write kqueue support. But it is said to be a hassle,
> >
> > .. and a waste of time since glib already does it for us,
> 
> I agree with you. But *if* somebody writes native kqueue support,
> because he doesn't want to touch glib on OS X: would you as Emacs
> maintainer reject this contribution?

I've been reading the actual code. So far, I'm not sure
how glib could be conveniently used here on Mac OS X even if one
wanted to. From what I can tell, the glib code seems to depend on the
central event loop being managed by glib/gtk, which is not the case
on a port that uses a different GUI. There are also different select
replacements (ns_select vs xg_select etc) used on the different
platforms because they have differing event loop handling.

I might be mistaken, though. I've never read any of the low level
event handling code before and I'm still a bit fuzzy on all the
details.

It also *seems* like Emacs currently is very much wired to use select
and select-like things for its low level event handling. This means
using a different mechanism (say, epoll or kqueue) a challenge.
Perhaps someone can correct me if I'm mistaken.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: gfile-based file notifications are not immediate

[Dima Kogan]

Perry E. Metzger <perry@piermont.com> writes:

> On Thu, 30 Oct 2014 19:11:55 +0100 Michael Albinus
> <michael.albinus@gmx.de> wrote:
>> Stefan Monnier <monnier@IRO.UMontreal.CA> writes:
>> 
>> >> If people on BSD/OS X do not want to use glib, somebody shall
>> >> write kqueue support. But it is said to be a hassle,
>> >
>> > .. and a waste of time since glib already does it for us,
>> 
>> I agree with you. But *if* somebody writes native kqueue support,
>> because he doesn't want to touch glib on OS X: would you as Emacs
>> maintainer reject this contribution?
>
> I've been reading the actual code. So far, I'm not sure
> how glib could be conveniently used here on Mac OS X even if one
> wanted to. From what I can tell, the glib code seems to depend on the
> central event loop being managed by glib/gtk, which is not the case
> on a port that uses a different GUI. There are also different select
> replacements (ns_select vs xg_select etc) used on the different
> platforms because they have differing event loop handling.
>
> I might be mistaken, though. I've never read any of the low level
> event handling code before and I'm still a bit fuzzy on all the
> details.

Look at xg_select.c. You call g_main_context_query() to retrieve a list
of file descriptors, and then you use pselect() on those AND on other
file descriptors emacs cares about. So even on my Debian machine the
event loop is still handled by emacs.

And please note that the whole reason I started this whole kerfuffle was
to get rid of delays in auto-revert-mode, and inotify is way closer to
getting us there than glib is:

 https://bugzilla.gnome.org/show_bug.cgi?id=739322

So please do not get rid of the inotify stuff yet.

Re: gfile-based file notifications are not immediate

[Stefan Monnier]

> I agree with you. But *if* somebody writes native kqueue support,
> because he doesn't want to touch glib on OS X: would you as Emacs
> maintainer reject this contribution?

Probably, yes.  Unless there's a serious problem with using glib.


        Stefan

Re: gfile-based file notifications are not immediate

[Wolfgang Jenkner]

On Wed, Oct 29 2014, Michael Albinus wrote:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> Interesting.  Another point in favor of dropping inotify support,
>
> There are people who do not want glib linked with Emacs.

Also, glib is part of GNOME, which, as a whole, does not seem to be
interested in systems which are not commercially relevant for them.

There's an active project to implement inotify for *BSD

https://github.com/dmatveev/libinotify-kqueue

While it still lacks inotify_init1, a year or so ago I worked around
this by some voodoo and built emacs to use the library.  That basically
worked (and, IIRC, even passed the inotify-test.el of the time) but in
some situations removing a watch did not work correctly and a cpu core
remained pegged at maximum usage while emacs was running.

So, in the long run, inotify might be the more portable solution.

Wolfgang

Re: gfile-based file notifications are not immediate

[Wolfgang Jenkner]

On Wed, Oct 29 2014, Wolfgang Jenkner wrote:

> There's an active project to implement inotify for *BSD
>
> https://github.com/dmatveev/libinotify-kqueue

Below is an updated patch for using this library.

I tried it on FreeBSD 10-STABLE by configuring emacs (trunk HEAD at
c0696d2) like this

./configure --with-x-toolkit=no --without-gsettings --without-rsvg --with-file-notification=inotify

(which on my system is enough to make sure that emacs is not linked with
libglib-2.0).

Previously, I had installed libinotify from the current HEAD on the
master branch (by adapting the existing devel/libinotify port, see
below).

After building emacs, I found that "gmake check" barfed, so I just
loaded test/automated/inotify-test.el and
test/automated/file-notify-tests.el and used ert-run-tests-interactively
to run the tests in those files.

All those test succeeded with `Ran 1 tests, 1 results were as expected'
except that the first test for remote files gave `Ran 1 tests, 0 results
were as expected, 1 skipped', so I didn't try the rest of them.

I can also replicate the behaviour Dima described in bug#18880.

I've also randomly played with file notifications and so far there were
no adverse effects on this emacs instance in which I am writing this.

There are two patches here:

The first patch is meant for FreeBSD (and, perhaps, DFly) people who
want to build the current libinotify from ports (and it has to be
applied in the devel/libinotify subdirectory of the ports tree).

The second patch is for emacs.  As I said in the previous mail, it
contains a more or less bogus work-around for the currently missing
inotify_init1().

--8<---------------cut here---------------start------------->8---
Index: Makefile
===================================================================
--- Makefile	(revision 371635)
+++ Makefile	(working copy)
@@ -13,7 +13,8 @@
 USE_GITHUB=	yes
 GH_ACCOUNT=	dmatveev
 GH_PROJECT=	libinotify-kqueue
-GH_COMMIT=	d775062
+#GH_COMMIT=	d775062
+GH_COMMIT=	953b095
 GH_TAGNAME=	${GH_COMMIT}
 
 USE_LDCONFIG=	yes
Index: distinfo
===================================================================
--- distinfo	(revision 371635)
+++ distinfo	(working copy)
@@ -1,2 +1,2 @@
-SHA256 (libinotify-20140622.tar.gz) = 512ca9342ab44ab1338c46baab596bd510ba9631d8ecb3b64ef8a89aa1b90d19
-SIZE (libinotify-20140622.tar.gz) = 33447
+SHA256 (libinotify-20140622.tar.gz) = 89a5c7796f46dd31501941617687ef6cb0705011e0a723433b094ddbc0c9bedb
+SIZE (libinotify-20140622.tar.gz) = 34586
--8<---------------cut here---------------end--------------->8---

--8<---------------cut here---------------start------------->8---
Subject: [PATCH] Tentative libinotify support.

---
 configure.ac    | 20 ++++++++++++++------
 src/Makefile.in |  4 +++-
 src/inotify.c   |  7 +++++++
 3 files changed, 24 insertions(+), 7 deletions(-)

diff --git a/configure.ac b/configure.ac
index cb42ed6..1665440 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2620,19 +2620,27 @@ case $with_file_notification,$NOTIFY_OBJ in
     fi ;;
 esac
 
-dnl inotify is only available on GNU/Linux.
+dnl inotify is available on GNU/Linux or, through libinotify, on *BSD.
+LIB_INOTIFY=
 case $with_file_notification,$NOTIFY_OBJ in
   inotify, | yes,)
     AC_CHECK_HEADER(sys/inotify.h)
     if test "$ac_cv_header_sys_inotify_h" = yes ; then
-	AC_CHECK_FUNC(inotify_init1)
-	if test "$ac_cv_func_inotify_init1" = yes; then
-	  AC_DEFINE(HAVE_INOTIFY, 1, [Define to 1 to use inotify.])
-	  NOTIFY_OBJ=inotify.o
-	  NOTIFY_SUMMARY="yes -lglibc (inotify)"
+       AC_SEARCH_LIBS(inotify_init, inotify)
+       if test "$ac_cv_search_inotify_init" != no; then
+	  AC_CHECK_FUNCS(inotify_init1 inotify_init)
+	  if test "$ac_cv_func_inotify_init" = yes; then
+	     AC_DEFINE(HAVE_INOTIFY, 1, [Define to 1 to use inotify.])
+	     NOTIFY_OBJ=inotify.o
+	     if test "$ac_cv_search_inotify_init" != "none required"; then
+		LIB_INOTIFY="$ac_cv_search_inotify_init"
+	     fi
+	     NOTIFY_SUMMARY="yes $LIB_INOTIFY (inotify)"
+	  fi
        fi
     fi ;;
 esac
+AC_SUBST(LIB_INOTIFY)
 
 case $with_file_notification,$NOTIFY_OBJ in
   yes,* | no,* | *,?*) ;;
diff --git a/src/Makefile.in b/src/Makefile.in
index 270119e..34db667 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -153,6 +153,8 @@ DBUS_OBJ = @DBUS_OBJ@
 
 LIB_EXECINFO=@LIB_EXECINFO@
 
+LIB_INOTIFY=@LIB_INOTIFY@
+
 SETTINGS_CFLAGS = @SETTINGS_CFLAGS@
 SETTINGS_LIBS = @SETTINGS_LIBS@
 
@@ -429,7 +431,7 @@ LIBES = $(LIBS) $(W32_LIBS) $(LIBS_GNUSTEP) $(LIBX_BASE) $(LIBIMAGE) \
    $(LIBS_TERMCAP) $(GETLOADAVG_LIBS) $(SETTINGS_LIBS) $(LIBSELINUX_LIBS) \
    $(FREETYPE_LIBS) $(FONTCONFIG_LIBS) $(LIBOTF_LIBS) $(M17N_FLT_LIBS) \
    $(LIBGNUTLS_LIBS) $(LIB_PTHREAD) \
-   $(GFILENOTIFY_LIBS) $(LIB_MATH) $(LIBZ)
+   $(GFILENOTIFY_LIBS) $(LIB_INOTIFY) $(LIB_MATH) $(LIBZ)
 
 all: emacs$(EXEEXT) $(OTHER_FILES)
 .PHONY: all
diff --git a/src/inotify.c b/src/inotify.c
index c55b130..e321a6d 100644
--- a/src/inotify.c
+++ b/src/inotify.c
@@ -333,7 +333,14 @@ is managed internally and there is no corresponding inotify_init.  Use
 
   if (inotifyfd < 0)
     {
+#if HAVE_INOTIFY_INIT1
       inotifyfd = inotify_init1 (IN_NONBLOCK|IN_CLOEXEC);
+#else /* I have no idea what I'm doing here. */
+      inotifyfd = inotify_init ();
+#include <fcntl.h>
+      fcntl (inotifyfd, F_SETFL, O_NONBLOCK);
+      fcntl (inotifyfd, F_SETFD, FD_CLOEXEC);
+#endif
       if (inotifyfd < 0)
 	xsignal1
 	  (Qfile_notify_error,
-- 
2.1.2

--8<---------------cut here---------------end--------------->8---

Re: gfile-based file notifications are not immediate

[Rüdiger Sonderfeld]

On Wednesday 29 October 2014 15:00:23 Wolfgang Jenkner wrote:
> There's an active project to implement inotify for *BSD
> 
> https://github.com/dmatveev/libinotify-kqueue
> 
> While it still lacks inotify_init1, a year or so ago I worked around
> this by some voodoo and built emacs to use the library.  That basically
> worked (and, IIRC, even passed the inotify-test.el of the time) but in
> some situations removing a watch did not work correctly and a cpu core
> remained pegged at maximum usage while emacs was running.
> 
> So, in the long run, inotify might be the more portable solution.

Glib already seems to use several hacks to work around the limitations of 
kqueue and Glib offers far fewer features than inotify.  Trying to emulate 
inotify on top of kqueue therefore seems like a futile approach.  Unless the 
*BSD folks implement inotify (or at least something similar) in their kernels 
I don't think we can seriously expect inotify to become a portable solution.

Glib meanwhile has a portable implementation and is already used (optionally) 
in GNU Emacs.  I don't think there is a risk of Glib abandoning kqueue support 
and even if they did then we could look at the issue again.

Regards,
Rüdiger

Re: gfile-based file notifications are not immediate

[Perry E. Metzger]

On Thu, 30 Oct 2014 21:32:30 +0100 Rüdiger Sonderfeld
<ruediger@c-plusplus.de> wrote:
> Glib already seems to use several hacks to work around the
> limitations of kqueue

As a aside: people keep mentioning that they believe kqueue has some
sort of fundamental limitations. I'm curious what those are perceived
to be. So far as I can tell, the interface is fine -- it is a rough
equivalent to what you can do in Linux with epoll (which is currently
preferred over select and its successors).

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: gfile-based file notifications are not immediate

[Noam Postavsky]

On Fri, Oct 31, 2014 at 5:28 PM, Perry E. Metzger <perry@piermont.com> wrote:
> On Thu, 30 Oct 2014 21:32:30 +0100 Rüdiger Sonderfeld
> <ruediger@c-plusplus.de> wrote:
>> Glib already seems to use several hacks to work around the
>> limitations of kqueue
>
> As a aside: people keep mentioning that they believe kqueue has some
> sort of fundamental limitations. I'm curious what those are perceived
> to be. So far as I can tell, the interface is fine -- it is a rough
> equivalent to what you can do in Linux with epoll (which is currently
> preferred over select and its successors).

I don't know the details, but I gathered the limitations are relative
to *inotify*. Presumably epoll would also be a poor substitute for
inotify, which is why Linux has both.

Re: gfile-based file notifications are not immediate

[Perry E. Metzger]

On Sun, 2 Nov 2014 09:15:32 -0500 Noam Postavsky
<npostavs@users.sourceforge.net> wrote:
> On Fri, Oct 31, 2014 at 5:28 PM, Perry E. Metzger
> <perry@piermont.com> wrote:
> > On Thu, 30 Oct 2014 21:32:30 +0100 Rüdiger Sonderfeld
> > <ruediger@c-plusplus.de> wrote:
> >> Glib already seems to use several hacks to work around the
> >> limitations of kqueue
> >
> > As a aside: people keep mentioning that they believe kqueue has
> > some sort of fundamental limitations. I'm curious what those are
> > perceived to be. So far as I can tell, the interface is fine --
> > it is a rough equivalent to what you can do in Linux with epoll
> > (which is currently preferred over select and its successors).
> 
> I don't know the details, but I gathered the limitations are
> relative to *inotify*. Presumably epoll would also be a poor
> substitute for inotify, which is why Linux has both.
> 

epoll and inotify are orthogonal. epoll says when a file descriptor
has data on it. inotify provides a file descriptor that yields event
information for file changes.

kqueue implements features of both select/poll and inotify. I'm not
sure what limitations kqueue might have vs. inotify, but they are
unlikely to be significant in this context.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: gfile-based file notifications are not immediate

[Eli Zaretskii]

> From: Dima Kogan <lists@dima.secretsauce.net>
> Date: Sat, 25 Oct 2014 13:17:29 -0700
> 
> As in the inotify thread, I set up a simple test. I built emacs using
> 
>  ./configure --with-file-notification=inotify
> 
> I then ran
> 
>  ./emacs --eval "`cat  /tmp/tstnotify.el`" -Q -nw
> 
> with tstnotify.el being
> 
>  (progn
>    (require 'filenotify)
> 
>    (dolist (fil '("/tmp/tst1" "/tmp/tst2"))
>      (file-notify-add-watch fil  '(change attribute-change)
>                             (lambda (event)
>                               (message "notify event %s" event)))
>      (find-file fil))
>    (switch-to-buffer "*Messages*"))
> 
> 
> Here I ask for notifications for two files, and print out the events as
> they come in. While emacs is running this way, I modify those two files
> using an external tool. I would expect to see modification events for
> both of these files, as soon as these modifications happen. Instead, the
> notifications come in either 30 seconds later, or whenever anything goes
> through emacs's input queue. So the notification is greatly delayed if
> the user is not touching their emacs.
> 
> The reason is that when emacs is idle, it's blocking in the pselect()
> call in xg_select() in process.c. pselect() is an operating-system call,
> not a glib one. Later on in xg_select() there's some code to kick the
> glib event loop, but as long as we're sitting in the pselect(), that
> secondary event loop remains untouched.

Maybe I'm confused, but are you saying that even though you
specifically requested for inotify notifications, and even though the
configure script shows

  Does Emacs use a file notification library?       yes -lglibc (inotify)

when it finishes, Emacs still uses glib file notifications, and the
functions in inotify.c are never called?

When functions in inotify.c are called, they add a file descriptor to
the set monitored by pselect (see the call to add_read_fd in
inotofy.c), so we shouldn't be waiting for glib event loop, I think.

Oh, maybe this all happens because ytour build is with GTK, and
therefore HAVE_GLIB is defined, so we call xg_select instead (I think
because some unwanted interference between these two mechanisms).  So
perhaps try building without GTK (or any other optional features that
pull in glib).

Re: gfile-based file notifications are not immediate

[Dima Kogan]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Dima Kogan <lists@dima.secretsauce.net>
>> Date: Sat, 25 Oct 2014 13:17:29 -0700
>> 
>> As in the inotify thread, I set up a simple test. I built emacs using
>> 
>>  ./configure --with-file-notification=inotify
>> 
> Maybe I'm confused, but are you saying that even though you
> specifically requested for inotify notifications, and even though the
> configure script shows
>
>   Does Emacs use a file notification library?       yes -lglibc (inotify)

Ack! My first email in this thread had a massive error. The intent was
for this thread to be about gfile-based file notifications, so the
configure wass done with

 ./configure --with-file-notification=gfile

Sorry about the unnecessary confusion. Hope the message makes more sense
now.

Re: gfile-based file notifications are not immediate

[Eli Zaretskii]

> From: Dima Kogan <dima@secretsauce.net>
> Cc: emacs-devel@gnu.org
> Date: Sun, 26 Oct 2014 10:07:26 -0700
> 
> configure wass done with
> 
>  ./configure --with-file-notification=gfile
> 
> Sorry about the unnecessary confusion. Hope the message makes more sense
> now.

Indeed, it does, thanks for clarifying.

Re: gfile-based file notifications are not immediate

[Michael Albinus]

Dima Kogan <lists@dima.secretsauce.net> writes:

> Hi.

Hi Dima,

> The reason is that when emacs is idle, it's blocking in the pselect()
> call in xg_select() in process.c. pselect() is an operating-system call,
> not a glib one. Later on in xg_select() there's some code to kick the
> glib event loop, but as long as we're sitting in the pselect(), that
> secondary event loop remains untouched.

Stefan did propose to add another event queue for special events coming
from D-Bus or file notifications, separated from the input queue. See
<http://thread.gmane.org/gmane.emacs.devel/169268/focus=169278>.

> Once again, I'm not posting to the bug tracker yet.

Now that Emacs 24.4 has been released, we could think about
implementation. From my point of view, a bug entry would remind us.

I don't know whether I can spend sufficient time to volunteer
implementation, 'tho :-(

> dima

Best regards, Michael.

Re: gfile-based file notifications are not immediate

[Dima Kogan]

Michael Albinus <michael.albinus@gmx.de> writes:

> Now that Emacs 24.4 has been released, we could think about
> implementation. From my point of view, a bug entry would remind us.

Just made a bug tracker entry:

 http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18861

Also I dug into this a bit, and will post findings to the bug.