OSX FSEvents file watching support

OSX FSEvents file watching support

[Muir Manders]

[-- Attachment #1: Type: text/plain, Size: 1064 bytes --]

Hello

I am interested in adding support in Emacs for Mac OS "FSEvents" file watching. FSEvents allows for efficient recursive directory watching, which would be a boon for packages like lsp-mode.

I seem to have three options:

1. Write a new backend for filenotify.el. The existing backends explicitly don't support recursive watching, so I'm not sure if it makes sense to have one special backend that does support recursive watching.

2. Publish a dynamic module that wraps the FSEvents API. It seems like most OSX distributions of Emacs have module support enabled, but I'm afraid there might be other challenges with this approach I don't know yet.

3. Publish an elisp package that calls out to an existing program that supports FSEvents like "fswatch". This approach probably has the least friction, but doesn't allow for tweaking FSEvents settings, and might have performance disadvantages.

Does anyone have any guidance on which approach makes the most sense, and in particular if a new recursive filenotify.el backend would be acceptable?

Thanks,
Muir

[-- Attachment #2: Type: text/html, Size: 1501 bytes --]

Re: OSX FSEvents file watching support

[Michael Albinus]

"Muir Manders" <muir@mnd.rs> writes:

> Hello

Hi Muir,

> I am interested in adding support in Emacs for Mac OS "FSEvents" file
> watching. FSEvents allows for efficient recursive directory watching,
> which would be a boon for packages like lsp-mode.

From time to time there have been requests for an FSEvents backend of
filenotify.el. Until now, nobody has taken the ball. So it would be
appreciated if you could do this.

> I seem to have three options:
>
> 1. Write a new backend for filenotify.el. The existing backends
> explicitly don't support recursive watching, so I'm not sure if it
> makes sense to have one special backend that does support recursive
> watching.

An FSEvents backend even for the existing filenotify.el interface could
be a win. Additionally, one could think about extending the
filenotify.el functions to support recursivity. For the other backends,
this interface returns either kinda of "not implemented", or we mimic
the recursivity by an own implementation, traversing the directory in
question.

> 2. Publish a dynamic module that wraps the FSEvents API. It seems like
> most OSX distributions of Emacs have module support enabled, but I'm
> afraid there might be other challenges with this approach I don't know
> yet.

This would reduce the usability. Packages interested in recursive
watching would be limited to macOS. They might be less enthusiastic to
use this.

> 3. Publish an elisp package that calls out to an existing program that
> supports FSEvents like "fswatch". This approach probably has the least
> friction, but doesn't allow for tweaking FSEvents settings, and might
> have performance disadvantages.

This is still useful for watching remote directories via Tramp. Should
be implemented in addition to 1.

> Does anyone have any guidance on which approach makes the most sense,
> and in particular if a new recursive filenotify.el backend would be
> acceptable?

I would say yes, and yes. But let's see what other people say.

> Thanks,
> Muir

Best regards, Michael.

Re: OSX FSEvents file watching support

[Eli Zaretskii]

> Date: Tue, 16 Jul 2019 21:18:29 -0700
> From: "Muir Manders" <muir@mnd.rs>
> 
> I am interested in adding support in Emacs for Mac OS "FSEvents" file watching. FSEvents allows for
> efficient recursive directory watching, which would be a boon for packages like lsp-mode.
> 
> I seem to have three options:
> 
> 1. Write a new backend for filenotify.el. The existing backends explicitly don't support recursive watching, so
> I'm not sure if it makes sense to have one special backend that does support recursive watching.

What do you mean by "recursive watching"?  w32notify.c supports
watching changes in all the subdirectories, recursively, of a
specified directory.  Is that what you mean?

> 2. Publish a dynamic module that wraps the FSEvents API. It seems like most OSX distributions of Emacs
> have module support enabled, but I'm afraid there might be other challenges with this approach I don't know
> yet.

I don't think modules can currently inject events into the Emacs event
queue.

> 3. Publish an elisp package that calls out to an existing program that supports FSEvents like "fswatch". This
> approach probably has the least friction, but doesn't allow for tweaking FSEvents settings, and might have
> performance disadvantages.

How would that work, implementation-wise?  Filesystem events are
supposed to be eventually injected into the input queue; how would you
go about doing that when an external porgram does the watching?

Re: OSX FSEvents file watching support

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> 3. Publish an elisp package that calls out to an existing program
>> that supports FSEvents like "fswatch". This
>> approach probably has the least friction, but doesn't allow for
>> tweaking FSEvents settings, and might have
>> performance disadvantages.
>
> How would that work, implementation-wise?  Filesystem events are
> supposed to be eventually injected into the input queue; how would you
> go about doing that when an external porgram does the watching?

Tramp mimics this by calling the respective special event filter function
file-notify-handle-event. See tramp-sh-inotifywait-process-filter and
tramp-sh-gio-monitor-process-filter.

Best regards, Michael.

Re: OSX FSEvents file watching support

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: "Muir Manders" <muir@mnd.rs>,  emacs-devel@gnu.org
> Date: Thu, 18 Jul 2019 13:31:10 +0200
> 
> > How would that work, implementation-wise?  Filesystem events are
> > supposed to be eventually injected into the input queue; how would you
> > go about doing that when an external porgram does the watching?
> 
> Tramp mimics this by calling the respective special event filter function
> file-notify-handle-event.

That needs a callback function, and the callback needs to be written
in C, no?  Or am I missing something?

To make myself clear, I was responding to the "publish an elisp
package" part of the OP's proposal, and my point was that I didn't
see how can this be done entirely in Lisp.

Re: OSX FSEvents file watching support

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

>> > How would that work, implementation-wise?  Filesystem events are
>> > supposed to be eventually injected into the input queue; how would you
>> > go about doing that when an external porgram does the watching?
>>
>> Tramp mimics this by calling the respective special event filter function
>> file-notify-handle-event.
>
> That needs a callback function, and the callback needs to be written
> in C, no?  Or am I missing something?

You were responding to

--8<---------------cut here---------------start------------->8---
3. Publish an elisp package that calls out to an existing program
that supports FSEvents like "fswatch".
--8<---------------cut here---------------end--------------->8---

> To make myself clear, I was responding to the "publish an elisp
> package" part of the OP's proposal, and my point was that I didn't
> see how can this be done entirely in Lisp.

The crucial point of this is "calls out an existing program". That's
what Tramp does in the process filters I've mentioned before.

Best regards, Michael.

Re: OSX FSEvents file watching support

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: muir@mnd.rs,  emacs-devel@gnu.org
> Date: Thu, 18 Jul 2019 14:16:29 +0200
> 
> You were responding to
> 
> --8<---------------cut here---------------start------------->8---
> 3. Publish an elisp package that calls out to an existing program
> that supports FSEvents like "fswatch".
> --8<---------------cut here---------------end--------------->8---
> 
> > To make myself clear, I was responding to the "publish an elisp
> > package" part of the OP's proposal, and my point was that I didn't
> > see how can this be done entirely in Lisp.
> 
> The crucial point of this is "calls out an existing program". That's
> what Tramp does in the process filters I've mentioned before.

Yes.  But once Tramp has the event description from the remote, how
does it inject it into Emacs?

Re: OSX FSEvents file watching support

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> The crucial point of this is "calls out an existing program". That's
>> what Tramp does in the process filters I've mentioned before.
>
> Yes.  But once Tramp has the event description from the remote, how
> does it inject it into Emacs?

Tramp doesn't. Injection the event would trigger the call of the
respective handler from special-event-map. Instead, Tramp calls this
handler directly.

Not pure doctrine, but it works.

Best regards, Michael.

Re: OSX FSEvents file watching support

[Eli Zaretskii]

> From: Michael Albinus <michael.albinus@gmx.de>
> Cc: muir@mnd.rs,  emacs-devel@gnu.org
> Date: Thu, 18 Jul 2019 16:14:26 +0200
> 
> > Yes.  But once Tramp has the event description from the remote, how
> > does it inject it into Emacs?
> 
> Tramp doesn't. Injection the event would trigger the call of the
> respective handler from special-event-map. Instead, Tramp calls this
> handler directly.

That only works if (a) Tramp knows in advance what handler to call,
and (b) that handler is callable from Lisp.

I don't think it's reasonable to base a native notification back-end
on these assumptions, they are too shaky, and it's unnecessary in the
first place.

Re: OSX FSEvents file watching support

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

>> Tramp doesn't. Injection the event would trigger the call of the
>> respective handler from special-event-map. Instead, Tramp calls this
>> handler directly.
>
> That only works if (a) Tramp knows in advance what handler to call,
> and (b) that handler is callable from Lisp.
>
> I don't think it's reasonable to base a native notification back-end
> on these assumptions, they are too shaky, and it's unnecessary in the
> first place.

We can change it, of course. But see also the comment in both filter
functions directly before the call of the handler: I tried it the other
way, and there were problems.

And yes, both (a) and (b) are fulfilled. Likely (a) is fulfilled
because it is the same guy who has written Tramp and the filenotify
event handler. (b) is fulfilled, because it is a Lisp function, which is
given by special-event-map as handler for file notification events.

I don't recommend to apply this technique for other events. But for file
notifications, it works for years.

Best regards, Michael.

Re: OSX FSEvents file watching support

[Stefan Monnier]

> That only works if (a) Tramp knows in advance what handler to call,
> and (b) that handler is callable from Lisp.

By virtue of being found via special-event-map, (b) should be satisfied
in all cases: we have no way to save in Lisp data structures functions
that are only callable from C.

As for (a) you don't need to know it in advance: you can look it up in
special-event-map, just like the C code does.


        Stefan

Re: OSX FSEvents file watching support

[Michael Albinus]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> As for (a) you don't need to know it in advance: you can look it up in
> special-event-map, just like the C code does.

Will change it in Tramp. There will be days, somebody else is
maintaining Tramp and/or filenotify.el.

>         Stefan

Best regards, Michael.

Re: OSX FSEvents file watching support

[Eli Zaretskii]

Btw, just between us: I'm much less enthusiastic today about file
notifications than I was a few years ago.  Since this feature was
introduced, we have learned about too many problematic cases that
together spell out one simple truth: these facilities are not scalable
enough.  For me, the last straw was when I tried to watch a log file
of building the Boost library in auto-revert-tail-mode -- this
completely broke down because there were many events each second, both
due to changes to the log itself and due to files being
created/modified in the same directory.  Polling, OTOH, had no problem
dealing with this situation.

I'm the last one to discourage someone from hacking Emacs, of course,
but I don't consider adding yet another file notification back-end
such a cool feature, given our somewhat disappointing experience.

Re: OSX FSEvents file watching support

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

Hi Eli,

> Btw, just between us: I'm much less enthusiastic today about file
> notifications than I was a few years ago.  Since this feature was
> introduced, we have learned about too many problematic cases that
> together spell out one simple truth: these facilities are not scalable
> enough.

Sadly, I must agree. One proof of this is, that even after years we have
no stable test case in filenotify-tests.el for arriving of mass events,
especially file-notify-test09-watched-file-in-watched-dir. Also other
test cases are flaky; I have adjusted them again and again over years.

Maybe we shall emphasize much more, what is written in the man page of
inotify:

--8<---------------cut here---------------start------------->8---
       With careful programming, an application can use inotify to efficiently
       monitor and cache the state of a set of filesystem  objects.   However,
       robust applications should allow for the fact that bugs in the monitor‐
       ing logic or races of the kind described  below  may  leave  the  cache
       inconsistent with the filesystem state.  It is probably wise to do some
       consistency checking, and rebuild the cache  when  inconsistencies  are
       detected.
--8<---------------cut here---------------end--------------->8---

> For me, the last straw was when I tried to watch a log file
> of building the Boost library in auto-revert-tail-mode -- this
> completely broke down because there were many events each second, both
> due to changes to the log itself and due to files being
> created/modified in the same directory.  Polling, OTOH, had no problem
> dealing with this situation.

We have already some defence mechanisms to handle such situations (I
believe). Maybe we shall implement better, in order to fall back to
polling automatically in case something unexpected happens.

However, this is nat basic file notification, but the upper level, auto
reverting using file notifications.

> I'm the last one to discourage someone from hacking Emacs, of course,
> but I don't consider adding yet another file notification back-end
> such a cool feature, given our somewhat disappointing experience.

OTOH, file notification is good if it acts over a limited set of file
changes. Whether recursive directory tracking belongs to it - I don't
know. But the proposed use case, lsp-mode, could fall into that category.

Best regards, Michael.

Re: OSX FSEvents file watching support

[Muir Manders]

> file notification is good if it acts over a limited set of file
> changes. Whether recursive directory tracking belongs to it - I don't
> know. But the proposed use case, lsp-mode, could fall into that category

lsp-mode in particular needs to watch your entire source code project for file changes and then relay the file names that were created/modified/deleted fo the LSP server. This is to handle when files change outside the editor, like when changing branches. The LSP server registers file globs for the types of files it wants to receive notifications for, which is normally just the source code files and maybe some project metadata files (e.g. "**/*.go" to watch Go files). The LSP spec delegates file watching to the editor because file watching is hard, and in general there are fewer LSP clients (editors) than LSP servers.

The current (kqueue, at least) backends must open many file descriptors to watch a large directory tree. FSEvents however can do this task efficiently.

Muir

Re: OSX FSEvents file watching support

[Eli Zaretskii]

> Date: Thu, 18 Jul 2019 08:14:33 -0700
> From: "Muir Manders" <muir@mnd.rs>
> Cc: emacs-devel@gnu.org
> 
> lsp-mode in particular needs to watch your entire source code project for file changes and then relay the file names that were created/modified/deleted fo the LSP server. This is to handle when files change outside the editor, like when changing branches. The LSP server registers file globs for the types of files it wants to receive notifications for, which is normally just the source code files and maybe some project metadata files (e.g. "**/*.go" to watch Go files). The LSP spec delegates file watching to the editor because file watching is hard, and in general there are fewer LSP clients (editors) than LSP servers.
> 
> The current (kqueue, at least) backends must open many file descriptors to watch a large directory tree. FSEvents however can do this task efficiently.

Beware: like I said earlier, I don't expect these facilities to be
scalable enough for watching large trees that get a lot of frequent
changes.  AFAIK, lsp-mode developers recommend to turn off file
notifications when there are a lot of them.

Re: OSX FSEvents file watching support

[Michael Albinus]

Eli Zaretskii <eliz@gnu.org> writes:

> Beware: like I said earlier, I don't expect these facilities to be
> scalable enough for watching large trees that get a lot of frequent
> changes.  AFAIK, lsp-mode developers recommend to turn off file
> notifications when there are a lot of them.

It shouldn't hurt, if Muir demonstrates how it works in a branch.

Best regards, Michael.

Re: OSX FSEvents file watching support

[Muir Manders]

> What do you mean by "recursive watching"?  ... watching changes in all the subdirectories ...?

Yes, I mean specifying a directory and watching for changes in all subdirectories. I didn't realize w32notify.c already supported that (the filenotify.el docs for "file-notify-add-watch" says "If FILE is a directory, `change' watches for file creation or deletion in that directory.  This does not work recursively." which made me think none of them do).

> I don't think modules can currently inject events into the Emacs event queue.

Good to know, hand't considered that. This is my first foray into lower level Emacs work.

> How would [calls out to an existing program that supports FSEvents like "fswatch"] work, implementation-wise?

I hadn't considered that Lisp can't inject the events into the input queue. My thought was to just read the file changes line by line over stdout and then feed the events into wherever they need to go.

Muir

On Thu, Jul 18, 2019, at 2:36 AM, Eli Zaretskii wrote:
> > Date: Tue, 16 Jul 2019 21:18:29 -0700
> > From: "Muir Manders" <muir@mnd.rs>
> > 
> > I am interested in adding support in Emacs for Mac OS "FSEvents" file watching. FSEvents allows for
> > efficient recursive directory watching, which would be a boon for packages like lsp-mode.
> > 
> > I seem to have three options:
> > 
> > 1. Write a new backend for filenotify.el. The existing backends explicitly don't support recursive watching, so
> > I'm not sure if it makes sense to have one special backend that does support recursive watching.
> 
> What do you mean by "recursive watching"?  w32notify.c supports
> watching changes in all the subdirectories, recursively, of a
> specified directory.  Is that what you mean?
> 
> > 2. Publish a dynamic module that wraps the FSEvents API. It seems like most OSX distributions of Emacs
> > have module support enabled, but I'm afraid there might be other challenges with this approach I don't know
> > yet.
> 
> I don't think modules can currently inject events into the Emacs event
> queue.
> 
> > 3. Publish an elisp package that calls out to an existing program that supports FSEvents like "fswatch". This
> > approach probably has the least friction, but doesn't allow for tweaking FSEvents settings, and might have
> > performance disadvantages.
> 
> How would that work, implementation-wise?  Filesystem events are
> supposed to be eventually injected into the input queue; how would you
> go about doing that when an external porgram does the watching?
>

Re: OSX FSEvents file watching support

[Mattias Engdegård]

17 juli 2019 kl. 06.18 skrev Muir Manders <muir@mnd.rs>:
> 
> I am interested in adding support in Emacs for Mac OS "FSEvents" file watching. FSEvents allows for efficient recursive directory watching, which would be a boon for packages like lsp-mode.

Good!

> 1. Write a new backend for filenotify.el. The existing backends explicitly don't support recursive watching, so I'm not sure if it makes sense to have one special backend that does support recursive watching.

This is probably the way to go, but some careful interface design is necessary. For example, FSEvents only reports the directory where changes occurred. For some applications this may be sufficient, but for others, the exact file names are required. What about lsp-mode? Without a specific application in mind, we risk over-engineering.

FSEvents can probably co-exist with kqueue which fits the filenotify model better for watching individual files, but it would be nice to have something that scales better (kqueue uses up file descriptors). On the other hand, FSEvents may require careful filtering to avoid being swamped with notifications during periods of intensive file system change.

Re: OSX FSEvents file watching support

[Michael Albinus]

Mattias Engdegård <mattiase@acm.org> writes:

> FSEvents can probably co-exist with kqueue which fits the filenotify
> model better for watching individual files,

We have no support for two backends running in parallel. Don't know
whether this is really wanted.

What we could think about is activating/deactivating backends during
runtime. Something like inotify or gfile, kqueue of FSEvents,
inotifywait or gio.

Best regards, Michael.

Re: OSX FSEvents file watching support

[Mattias Engdegård]

18 juli 2019 kl. 13.26 skrev Michael Albinus <michael.albinus@gmx.de>:
> 
> We have no support for two backends running in parallel. Don't know
> whether this is really wanted.

There is nothing sacred about the current filenotify/backend structure; if it needs changing, then so be it.
Either FSEvents turns out to be able to supplant kqueue on macOS entirely, in which case the point is moot, or we use FSEvent for  watching subtrees and keep kqueue for single files and directories.

Re: OSX FSEvents file watching support

[Muir Manders]

> FSEvents only reports the directory where changes occurred.

Based on my testing, FSEvents reports the full file path for each change along with metadata about the file and the change.

Here is the test program I was using to play around with it: https://gist.github.com/muirrn/ea8c63c22ca7624f304f805ba77b25f4

Muir

On Thu, Jul 18, 2019, at 2:54 AM, Mattias Engdegård wrote:
> 17 juli 2019 kl. 06.18 skrev Muir Manders <muir@mnd.rs>:
> > 
> > I am interested in adding support in Emacs for Mac OS "FSEvents" file watching. FSEvents allows for efficient recursive directory watching, which would be a boon for packages like lsp-mode.
> 
> Good!
> 
> > 1. Write a new backend for filenotify.el. The existing backends explicitly don't support recursive watching, so I'm not sure if it makes sense to have one special backend that does support recursive watching.
> 
> This is probably the way to go, but some careful interface design is 
> necessary. For example, FSEvents only reports the directory where 
> changes occurred. For some applications this may be sufficient, but for 
> others, the exact file names are required. What about lsp-mode? Without 
> a specific application in mind, we risk over-engineering.
> 
> FSEvents can probably co-exist with kqueue which fits the filenotify 
> model better for watching individual files, but it would be nice to 
> have something that scales better (kqueue uses up file descriptors). On 
> the other hand, FSEvents may require careful filtering to avoid being 
> swamped with notifications during periods of intensive file system 
> change.
> 
>

Re: OSX FSEvents file watching support

[Stefan Monnier]

>> FSEvents only reports the directory where changes occurred.
> Based on my testing, FSEvents reports the full file path for each change
                                            ^^^^^^^^^^^^^^
The GNU convention (which we use in Emacs) is to only use "path" for
lists of directories as in $PATH and `load-path`, so we'd call the above
"full file name" (or maybe "absolute file name").

[ Of course, you're free to use the terminology you want, but I'm just
  pointing this out because I feel that there's a risk of confusion
  here.  ]


        Stefan

Re: OSX FSEvents file watching support

[Mattias Engdegård]

18 juli 2019 kl. 16.53 skrev Muir Manders <muir@mnd.rs>:
> 
> Based on my testing, FSEvents reports the full file path for each change along with metadata about the file and the change.

Look at that, kFSEventStreamCreateFlagFileEvents is new (to me). Thanks!
On the other hand, using that flag would increase load on Emacs if there is a lot of irrelevant changes, so I'm not sure it is useful unless changes are sporadic and/or the subtrees watched carefully constrained.

FSEvents does have a rate limiter of sorts (the 'latency' parameter), which works better when that flag isn't used. Since filenotify normally knows what files are of interest anyway, that may be more efficient.

For monitoring an entire subtree for changes, FSEvents is clearly useful. A sensible interface would allow a filter to be specified (globs or regexp) so that this could be handled by the back-end directly, avoiding the rather expensive call-out to elisp.