* OSX FSEvents file watching support @ 2019-07-17 4:18 Muir Manders 2019-07-17 19:26 ` Michael Albinus ` (2 more replies) 0 siblings, 3 replies; 24+ messages in thread From: Muir Manders @ 2019-07-17 4:18 UTC (permalink / raw) To: emacs-devel [-- Attachment #1: Type: text/plain, Size: 1064 bytes --] Hello I am interested in adding support in Emacs for Mac OS "FSEvents" file watching. FSEvents allows for efficient recursive directory watching, which would be a boon for packages like lsp-mode. I seem to have three options: 1. Write a new backend for filenotify.el. The existing backends explicitly don't support recursive watching, so I'm not sure if it makes sense to have one special backend that does support recursive watching. 2. Publish a dynamic module that wraps the FSEvents API. It seems like most OSX distributions of Emacs have module support enabled, but I'm afraid there might be other challenges with this approach I don't know yet. 3. Publish an elisp package that calls out to an existing program that supports FSEvents like "fswatch". This approach probably has the least friction, but doesn't allow for tweaking FSEvents settings, and might have performance disadvantages. Does anyone have any guidance on which approach makes the most sense, and in particular if a new recursive filenotify.el backend would be acceptable? Thanks, Muir [-- Attachment #2: Type: text/html, Size: 1501 bytes --] ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-17 4:18 OSX FSEvents file watching support Muir Manders @ 2019-07-17 19:26 ` Michael Albinus 2019-07-18 9:35 ` Eli Zaretskii 2019-07-18 9:54 ` Mattias Engdegård 2 siblings, 0 replies; 24+ messages in thread From: Michael Albinus @ 2019-07-17 19:26 UTC (permalink / raw) To: Muir Manders; +Cc: emacs-devel "Muir Manders" <muir@mnd.rs> writes: > Hello Hi Muir, > I am interested in adding support in Emacs for Mac OS "FSEvents" file > watching. FSEvents allows for efficient recursive directory watching, > which would be a boon for packages like lsp-mode. From time to time there have been requests for an FSEvents backend of filenotify.el. Until now, nobody has taken the ball. So it would be appreciated if you could do this. > I seem to have three options: > > 1. Write a new backend for filenotify.el. The existing backends > explicitly don't support recursive watching, so I'm not sure if it > makes sense to have one special backend that does support recursive > watching. An FSEvents backend even for the existing filenotify.el interface could be a win. Additionally, one could think about extending the filenotify.el functions to support recursivity. For the other backends, this interface returns either kinda of "not implemented", or we mimic the recursivity by an own implementation, traversing the directory in question. > 2. Publish a dynamic module that wraps the FSEvents API. It seems like > most OSX distributions of Emacs have module support enabled, but I'm > afraid there might be other challenges with this approach I don't know > yet. This would reduce the usability. Packages interested in recursive watching would be limited to macOS. They might be less enthusiastic to use this. > 3. Publish an elisp package that calls out to an existing program that > supports FSEvents like "fswatch". This approach probably has the least > friction, but doesn't allow for tweaking FSEvents settings, and might > have performance disadvantages. This is still useful for watching remote directories via Tramp. Should be implemented in addition to 1. > Does anyone have any guidance on which approach makes the most sense, > and in particular if a new recursive filenotify.el backend would be > acceptable? I would say yes, and yes. But let's see what other people say. > Thanks, > Muir Best regards, Michael. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-17 4:18 OSX FSEvents file watching support Muir Manders 2019-07-17 19:26 ` Michael Albinus @ 2019-07-18 9:35 ` Eli Zaretskii 2019-07-18 11:31 ` Michael Albinus 2019-07-18 14:38 ` Muir Manders 2019-07-18 9:54 ` Mattias Engdegård 2 siblings, 2 replies; 24+ messages in thread From: Eli Zaretskii @ 2019-07-18 9:35 UTC (permalink / raw) To: Muir Manders; +Cc: emacs-devel > Date: Tue, 16 Jul 2019 21:18:29 -0700 > From: "Muir Manders" <muir@mnd.rs> > > I am interested in adding support in Emacs for Mac OS "FSEvents" file watching. FSEvents allows for > efficient recursive directory watching, which would be a boon for packages like lsp-mode. > > I seem to have three options: > > 1. Write a new backend for filenotify.el. The existing backends explicitly don't support recursive watching, so > I'm not sure if it makes sense to have one special backend that does support recursive watching. What do you mean by "recursive watching"? w32notify.c supports watching changes in all the subdirectories, recursively, of a specified directory. Is that what you mean? > 2. Publish a dynamic module that wraps the FSEvents API. It seems like most OSX distributions of Emacs > have module support enabled, but I'm afraid there might be other challenges with this approach I don't know > yet. I don't think modules can currently inject events into the Emacs event queue. > 3. Publish an elisp package that calls out to an existing program that supports FSEvents like "fswatch". This > approach probably has the least friction, but doesn't allow for tweaking FSEvents settings, and might have > performance disadvantages. How would that work, implementation-wise? Filesystem events are supposed to be eventually injected into the input queue; how would you go about doing that when an external porgram does the watching? ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 9:35 ` Eli Zaretskii @ 2019-07-18 11:31 ` Michael Albinus 2019-07-18 12:00 ` Eli Zaretskii 2019-07-18 14:38 ` Muir Manders 1 sibling, 1 reply; 24+ messages in thread From: Michael Albinus @ 2019-07-18 11:31 UTC (permalink / raw) To: Eli Zaretskii; +Cc: Muir Manders, emacs-devel Eli Zaretskii <eliz@gnu.org> writes: >> 3. Publish an elisp package that calls out to an existing program >> that supports FSEvents like "fswatch". This >> approach probably has the least friction, but doesn't allow for >> tweaking FSEvents settings, and might have >> performance disadvantages. > > How would that work, implementation-wise? Filesystem events are > supposed to be eventually injected into the input queue; how would you > go about doing that when an external porgram does the watching? Tramp mimics this by calling the respective special event filter function file-notify-handle-event. See tramp-sh-inotifywait-process-filter and tramp-sh-gio-monitor-process-filter. Best regards, Michael. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 11:31 ` Michael Albinus @ 2019-07-18 12:00 ` Eli Zaretskii 2019-07-18 12:16 ` Michael Albinus 0 siblings, 1 reply; 24+ messages in thread From: Eli Zaretskii @ 2019-07-18 12:00 UTC (permalink / raw) To: Michael Albinus; +Cc: muir, emacs-devel > From: Michael Albinus <michael.albinus@gmx.de> > Cc: "Muir Manders" <muir@mnd.rs>, emacs-devel@gnu.org > Date: Thu, 18 Jul 2019 13:31:10 +0200 > > > How would that work, implementation-wise? Filesystem events are > > supposed to be eventually injected into the input queue; how would you > > go about doing that when an external porgram does the watching? > > Tramp mimics this by calling the respective special event filter function > file-notify-handle-event. That needs a callback function, and the callback needs to be written in C, no? Or am I missing something? To make myself clear, I was responding to the "publish an elisp package" part of the OP's proposal, and my point was that I didn't see how can this be done entirely in Lisp. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 12:00 ` Eli Zaretskii @ 2019-07-18 12:16 ` Michael Albinus 2019-07-18 12:30 ` Eli Zaretskii 0 siblings, 1 reply; 24+ messages in thread From: Michael Albinus @ 2019-07-18 12:16 UTC (permalink / raw) To: Eli Zaretskii; +Cc: muir, emacs-devel Eli Zaretskii <eliz@gnu.org> writes: Hi Eli, >> > How would that work, implementation-wise? Filesystem events are >> > supposed to be eventually injected into the input queue; how would you >> > go about doing that when an external porgram does the watching? >> >> Tramp mimics this by calling the respective special event filter function >> file-notify-handle-event. > > That needs a callback function, and the callback needs to be written > in C, no? Or am I missing something? You were responding to --8<---------------cut here---------------start------------->8--- 3. Publish an elisp package that calls out to an existing program that supports FSEvents like "fswatch". --8<---------------cut here---------------end--------------->8--- > To make myself clear, I was responding to the "publish an elisp > package" part of the OP's proposal, and my point was that I didn't > see how can this be done entirely in Lisp. The crucial point of this is "calls out an existing program". That's what Tramp does in the process filters I've mentioned before. Best regards, Michael. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 12:16 ` Michael Albinus @ 2019-07-18 12:30 ` Eli Zaretskii 2019-07-18 14:14 ` Michael Albinus 0 siblings, 1 reply; 24+ messages in thread From: Eli Zaretskii @ 2019-07-18 12:30 UTC (permalink / raw) To: Michael Albinus; +Cc: muir, emacs-devel > From: Michael Albinus <michael.albinus@gmx.de> > Cc: muir@mnd.rs, emacs-devel@gnu.org > Date: Thu, 18 Jul 2019 14:16:29 +0200 > > You were responding to > > --8<---------------cut here---------------start------------->8--- > 3. Publish an elisp package that calls out to an existing program > that supports FSEvents like "fswatch". > --8<---------------cut here---------------end--------------->8--- > > > To make myself clear, I was responding to the "publish an elisp > > package" part of the OP's proposal, and my point was that I didn't > > see how can this be done entirely in Lisp. > > The crucial point of this is "calls out an existing program". That's > what Tramp does in the process filters I've mentioned before. Yes. But once Tramp has the event description from the remote, how does it inject it into Emacs? ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 12:30 ` Eli Zaretskii @ 2019-07-18 14:14 ` Michael Albinus 2019-07-18 14:18 ` Eli Zaretskii 0 siblings, 1 reply; 24+ messages in thread From: Michael Albinus @ 2019-07-18 14:14 UTC (permalink / raw) To: Eli Zaretskii; +Cc: muir, emacs-devel Eli Zaretskii <eliz@gnu.org> writes: >> The crucial point of this is "calls out an existing program". That's >> what Tramp does in the process filters I've mentioned before. > > Yes. But once Tramp has the event description from the remote, how > does it inject it into Emacs? Tramp doesn't. Injection the event would trigger the call of the respective handler from special-event-map. Instead, Tramp calls this handler directly. Not pure doctrine, but it works. Best regards, Michael. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 14:14 ` Michael Albinus @ 2019-07-18 14:18 ` Eli Zaretskii 2019-07-18 14:29 ` Michael Albinus 2019-07-18 14:30 ` Eli Zaretskii 0 siblings, 2 replies; 24+ messages in thread From: Eli Zaretskii @ 2019-07-18 14:18 UTC (permalink / raw) To: Michael Albinus; +Cc: muir, emacs-devel > From: Michael Albinus <michael.albinus@gmx.de> > Cc: muir@mnd.rs, emacs-devel@gnu.org > Date: Thu, 18 Jul 2019 16:14:26 +0200 > > > Yes. But once Tramp has the event description from the remote, how > > does it inject it into Emacs? > > Tramp doesn't. Injection the event would trigger the call of the > respective handler from special-event-map. Instead, Tramp calls this > handler directly. That only works if (a) Tramp knows in advance what handler to call, and (b) that handler is callable from Lisp. I don't think it's reasonable to base a native notification back-end on these assumptions, they are too shaky, and it's unnecessary in the first place. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 14:18 ` Eli Zaretskii @ 2019-07-18 14:29 ` Michael Albinus 2019-07-18 14:55 ` Stefan Monnier 2019-07-18 14:30 ` Eli Zaretskii 1 sibling, 1 reply; 24+ messages in thread From: Michael Albinus @ 2019-07-18 14:29 UTC (permalink / raw) To: Eli Zaretskii; +Cc: muir, emacs-devel Eli Zaretskii <eliz@gnu.org> writes: >> Tramp doesn't. Injection the event would trigger the call of the >> respective handler from special-event-map. Instead, Tramp calls this >> handler directly. > > That only works if (a) Tramp knows in advance what handler to call, > and (b) that handler is callable from Lisp. > > I don't think it's reasonable to base a native notification back-end > on these assumptions, they are too shaky, and it's unnecessary in the > first place. We can change it, of course. But see also the comment in both filter functions directly before the call of the handler: I tried it the other way, and there were problems. And yes, both (a) and (b) are fulfilled. Likely (a) is fulfilled because it is the same guy who has written Tramp and the filenotify event handler. (b) is fulfilled, because it is a Lisp function, which is given by special-event-map as handler for file notification events. I don't recommend to apply this technique for other events. But for file notifications, it works for years. Best regards, Michael. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 14:29 ` Michael Albinus @ 2019-07-18 14:55 ` Stefan Monnier 2019-07-18 15:03 ` Michael Albinus 0 siblings, 1 reply; 24+ messages in thread From: Stefan Monnier @ 2019-07-18 14:55 UTC (permalink / raw) To: Michael Albinus; +Cc: Eli Zaretskii, muir, emacs-devel > That only works if (a) Tramp knows in advance what handler to call, > and (b) that handler is callable from Lisp. By virtue of being found via special-event-map, (b) should be satisfied in all cases: we have no way to save in Lisp data structures functions that are only callable from C. As for (a) you don't need to know it in advance: you can look it up in special-event-map, just like the C code does. Stefan ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 14:55 ` Stefan Monnier @ 2019-07-18 15:03 ` Michael Albinus 0 siblings, 0 replies; 24+ messages in thread From: Michael Albinus @ 2019-07-18 15:03 UTC (permalink / raw) To: Stefan Monnier; +Cc: Eli Zaretskii, muir, emacs-devel Stefan Monnier <monnier@iro.umontreal.ca> writes: > As for (a) you don't need to know it in advance: you can look it up in > special-event-map, just like the C code does. Will change it in Tramp. There will be days, somebody else is maintaining Tramp and/or filenotify.el. > Stefan Best regards, Michael. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 14:18 ` Eli Zaretskii 2019-07-18 14:29 ` Michael Albinus @ 2019-07-18 14:30 ` Eli Zaretskii 2019-07-18 14:46 ` Michael Albinus 1 sibling, 1 reply; 24+ messages in thread From: Eli Zaretskii @ 2019-07-18 14:30 UTC (permalink / raw) To: michael.albinus; +Cc: muir, emacs-devel Btw, just between us: I'm much less enthusiastic today about file notifications than I was a few years ago. Since this feature was introduced, we have learned about too many problematic cases that together spell out one simple truth: these facilities are not scalable enough. For me, the last straw was when I tried to watch a log file of building the Boost library in auto-revert-tail-mode -- this completely broke down because there were many events each second, both due to changes to the log itself and due to files being created/modified in the same directory. Polling, OTOH, had no problem dealing with this situation. I'm the last one to discourage someone from hacking Emacs, of course, but I don't consider adding yet another file notification back-end such a cool feature, given our somewhat disappointing experience. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 14:30 ` Eli Zaretskii @ 2019-07-18 14:46 ` Michael Albinus 2019-07-18 15:14 ` Muir Manders 0 siblings, 1 reply; 24+ messages in thread From: Michael Albinus @ 2019-07-18 14:46 UTC (permalink / raw) To: Eli Zaretskii; +Cc: muir, emacs-devel Eli Zaretskii <eliz@gnu.org> writes: Hi Eli, > Btw, just between us: I'm much less enthusiastic today about file > notifications than I was a few years ago. Since this feature was > introduced, we have learned about too many problematic cases that > together spell out one simple truth: these facilities are not scalable > enough. Sadly, I must agree. One proof of this is, that even after years we have no stable test case in filenotify-tests.el for arriving of mass events, especially file-notify-test09-watched-file-in-watched-dir. Also other test cases are flaky; I have adjusted them again and again over years. Maybe we shall emphasize much more, what is written in the man page of inotify: --8<---------------cut here---------------start------------->8--- With careful programming, an application can use inotify to efficiently monitor and cache the state of a set of filesystem objects. However, robust applications should allow for the fact that bugs in the monitor‐ ing logic or races of the kind described below may leave the cache inconsistent with the filesystem state. It is probably wise to do some consistency checking, and rebuild the cache when inconsistencies are detected. --8<---------------cut here---------------end--------------->8--- > For me, the last straw was when I tried to watch a log file > of building the Boost library in auto-revert-tail-mode -- this > completely broke down because there were many events each second, both > due to changes to the log itself and due to files being > created/modified in the same directory. Polling, OTOH, had no problem > dealing with this situation. We have already some defence mechanisms to handle such situations (I believe). Maybe we shall implement better, in order to fall back to polling automatically in case something unexpected happens. However, this is nat basic file notification, but the upper level, auto reverting using file notifications. > I'm the last one to discourage someone from hacking Emacs, of course, > but I don't consider adding yet another file notification back-end > such a cool feature, given our somewhat disappointing experience. OTOH, file notification is good if it acts over a limited set of file changes. Whether recursive directory tracking belongs to it - I don't know. But the proposed use case, lsp-mode, could fall into that category. Best regards, Michael. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 14:46 ` Michael Albinus @ 2019-07-18 15:14 ` Muir Manders 2019-07-18 15:28 ` Eli Zaretskii 0 siblings, 1 reply; 24+ messages in thread From: Muir Manders @ 2019-07-18 15:14 UTC (permalink / raw) To: Michael Albinus, Eli Zaretskii; +Cc: emacs-devel > file notification is good if it acts over a limited set of file > changes. Whether recursive directory tracking belongs to it - I don't > know. But the proposed use case, lsp-mode, could fall into that category lsp-mode in particular needs to watch your entire source code project for file changes and then relay the file names that were created/modified/deleted fo the LSP server. This is to handle when files change outside the editor, like when changing branches. The LSP server registers file globs for the types of files it wants to receive notifications for, which is normally just the source code files and maybe some project metadata files (e.g. "**/*.go" to watch Go files). The LSP spec delegates file watching to the editor because file watching is hard, and in general there are fewer LSP clients (editors) than LSP servers. The current (kqueue, at least) backends must open many file descriptors to watch a large directory tree. FSEvents however can do this task efficiently. Muir ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 15:14 ` Muir Manders @ 2019-07-18 15:28 ` Eli Zaretskii 2019-07-18 18:04 ` Michael Albinus 0 siblings, 1 reply; 24+ messages in thread From: Eli Zaretskii @ 2019-07-18 15:28 UTC (permalink / raw) To: Muir Manders; +Cc: michael.albinus, emacs-devel > Date: Thu, 18 Jul 2019 08:14:33 -0700 > From: "Muir Manders" <muir@mnd.rs> > Cc: emacs-devel@gnu.org > > lsp-mode in particular needs to watch your entire source code project for file changes and then relay the file names that were created/modified/deleted fo the LSP server. This is to handle when files change outside the editor, like when changing branches. The LSP server registers file globs for the types of files it wants to receive notifications for, which is normally just the source code files and maybe some project metadata files (e.g. "**/*.go" to watch Go files). The LSP spec delegates file watching to the editor because file watching is hard, and in general there are fewer LSP clients (editors) than LSP servers. > > The current (kqueue, at least) backends must open many file descriptors to watch a large directory tree. FSEvents however can do this task efficiently. Beware: like I said earlier, I don't expect these facilities to be scalable enough for watching large trees that get a lot of frequent changes. AFAIK, lsp-mode developers recommend to turn off file notifications when there are a lot of them. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 15:28 ` Eli Zaretskii @ 2019-07-18 18:04 ` Michael Albinus 0 siblings, 0 replies; 24+ messages in thread From: Michael Albinus @ 2019-07-18 18:04 UTC (permalink / raw) To: Eli Zaretskii; +Cc: Muir Manders, emacs-devel Eli Zaretskii <eliz@gnu.org> writes: > Beware: like I said earlier, I don't expect these facilities to be > scalable enough for watching large trees that get a lot of frequent > changes. AFAIK, lsp-mode developers recommend to turn off file > notifications when there are a lot of them. It shouldn't hurt, if Muir demonstrates how it works in a branch. Best regards, Michael. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 9:35 ` Eli Zaretskii 2019-07-18 11:31 ` Michael Albinus @ 2019-07-18 14:38 ` Muir Manders 1 sibling, 0 replies; 24+ messages in thread From: Muir Manders @ 2019-07-18 14:38 UTC (permalink / raw) To: Eli Zaretskii; +Cc: emacs-devel > What do you mean by "recursive watching"? ... watching changes in all the subdirectories ...? Yes, I mean specifying a directory and watching for changes in all subdirectories. I didn't realize w32notify.c already supported that (the filenotify.el docs for "file-notify-add-watch" says "If FILE is a directory, `change' watches for file creation or deletion in that directory. This does not work recursively." which made me think none of them do). > I don't think modules can currently inject events into the Emacs event queue. Good to know, hand't considered that. This is my first foray into lower level Emacs work. > How would [calls out to an existing program that supports FSEvents like "fswatch"] work, implementation-wise? I hadn't considered that Lisp can't inject the events into the input queue. My thought was to just read the file changes line by line over stdout and then feed the events into wherever they need to go. Muir On Thu, Jul 18, 2019, at 2:36 AM, Eli Zaretskii wrote: > > Date: Tue, 16 Jul 2019 21:18:29 -0700 > > From: "Muir Manders" <muir@mnd.rs> > > > > I am interested in adding support in Emacs for Mac OS "FSEvents" file watching. FSEvents allows for > > efficient recursive directory watching, which would be a boon for packages like lsp-mode. > > > > I seem to have three options: > > > > 1. Write a new backend for filenotify.el. The existing backends explicitly don't support recursive watching, so > > I'm not sure if it makes sense to have one special backend that does support recursive watching. > > What do you mean by "recursive watching"? w32notify.c supports > watching changes in all the subdirectories, recursively, of a > specified directory. Is that what you mean? > > > 2. Publish a dynamic module that wraps the FSEvents API. It seems like most OSX distributions of Emacs > > have module support enabled, but I'm afraid there might be other challenges with this approach I don't know > > yet. > > I don't think modules can currently inject events into the Emacs event > queue. > > > 3. Publish an elisp package that calls out to an existing program that supports FSEvents like "fswatch". This > > approach probably has the least friction, but doesn't allow for tweaking FSEvents settings, and might have > > performance disadvantages. > > How would that work, implementation-wise? Filesystem events are > supposed to be eventually injected into the input queue; how would you > go about doing that when an external porgram does the watching? > ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-17 4:18 OSX FSEvents file watching support Muir Manders 2019-07-17 19:26 ` Michael Albinus 2019-07-18 9:35 ` Eli Zaretskii @ 2019-07-18 9:54 ` Mattias Engdegård 2019-07-18 11:26 ` Michael Albinus 2019-07-18 14:53 ` Muir Manders 2 siblings, 2 replies; 24+ messages in thread From: Mattias Engdegård @ 2019-07-18 9:54 UTC (permalink / raw) To: Muir Manders; +Cc: emacs-devel 17 juli 2019 kl. 06.18 skrev Muir Manders <muir@mnd.rs>: > > I am interested in adding support in Emacs for Mac OS "FSEvents" file watching. FSEvents allows for efficient recursive directory watching, which would be a boon for packages like lsp-mode. Good! > 1. Write a new backend for filenotify.el. The existing backends explicitly don't support recursive watching, so I'm not sure if it makes sense to have one special backend that does support recursive watching. This is probably the way to go, but some careful interface design is necessary. For example, FSEvents only reports the directory where changes occurred. For some applications this may be sufficient, but for others, the exact file names are required. What about lsp-mode? Without a specific application in mind, we risk over-engineering. FSEvents can probably co-exist with kqueue which fits the filenotify model better for watching individual files, but it would be nice to have something that scales better (kqueue uses up file descriptors). On the other hand, FSEvents may require careful filtering to avoid being swamped with notifications during periods of intensive file system change. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 9:54 ` Mattias Engdegård @ 2019-07-18 11:26 ` Michael Albinus 2019-07-18 11:52 ` Mattias Engdegård 2019-07-18 14:53 ` Muir Manders 1 sibling, 1 reply; 24+ messages in thread From: Michael Albinus @ 2019-07-18 11:26 UTC (permalink / raw) To: Mattias Engdegård; +Cc: Muir Manders, emacs-devel Mattias Engdegård <mattiase@acm.org> writes: > FSEvents can probably co-exist with kqueue which fits the filenotify > model better for watching individual files, We have no support for two backends running in parallel. Don't know whether this is really wanted. What we could think about is activating/deactivating backends during runtime. Something like inotify or gfile, kqueue of FSEvents, inotifywait or gio. Best regards, Michael. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 11:26 ` Michael Albinus @ 2019-07-18 11:52 ` Mattias Engdegård 0 siblings, 0 replies; 24+ messages in thread From: Mattias Engdegård @ 2019-07-18 11:52 UTC (permalink / raw) To: Michael Albinus; +Cc: Muir Manders, emacs-devel 18 juli 2019 kl. 13.26 skrev Michael Albinus <michael.albinus@gmx.de>: > > We have no support for two backends running in parallel. Don't know > whether this is really wanted. There is nothing sacred about the current filenotify/backend structure; if it needs changing, then so be it. Either FSEvents turns out to be able to supplant kqueue on macOS entirely, in which case the point is moot, or we use FSEvent for watching subtrees and keep kqueue for single files and directories. ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 9:54 ` Mattias Engdegård 2019-07-18 11:26 ` Michael Albinus @ 2019-07-18 14:53 ` Muir Manders 2019-07-18 15:06 ` Stefan Monnier 2019-07-18 16:47 ` Mattias Engdegård 1 sibling, 2 replies; 24+ messages in thread From: Muir Manders @ 2019-07-18 14:53 UTC (permalink / raw) To: Mattias Engdegård; +Cc: emacs-devel > FSEvents only reports the directory where changes occurred. Based on my testing, FSEvents reports the full file path for each change along with metadata about the file and the change. Here is the test program I was using to play around with it: https://gist.github.com/muirrn/ea8c63c22ca7624f304f805ba77b25f4 Muir On Thu, Jul 18, 2019, at 2:54 AM, Mattias Engdegård wrote: > 17 juli 2019 kl. 06.18 skrev Muir Manders <muir@mnd.rs>: > > > > I am interested in adding support in Emacs for Mac OS "FSEvents" file watching. FSEvents allows for efficient recursive directory watching, which would be a boon for packages like lsp-mode. > > Good! > > > 1. Write a new backend for filenotify.el. The existing backends explicitly don't support recursive watching, so I'm not sure if it makes sense to have one special backend that does support recursive watching. > > This is probably the way to go, but some careful interface design is > necessary. For example, FSEvents only reports the directory where > changes occurred. For some applications this may be sufficient, but for > others, the exact file names are required. What about lsp-mode? Without > a specific application in mind, we risk over-engineering. > > FSEvents can probably co-exist with kqueue which fits the filenotify > model better for watching individual files, but it would be nice to > have something that scales better (kqueue uses up file descriptors). On > the other hand, FSEvents may require careful filtering to avoid being > swamped with notifications during periods of intensive file system > change. > > ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 14:53 ` Muir Manders @ 2019-07-18 15:06 ` Stefan Monnier 2019-07-18 16:47 ` Mattias Engdegård 1 sibling, 0 replies; 24+ messages in thread From: Stefan Monnier @ 2019-07-18 15:06 UTC (permalink / raw) To: Muir Manders; +Cc: Mattias Engdegård, emacs-devel >> FSEvents only reports the directory where changes occurred. > Based on my testing, FSEvents reports the full file path for each change ^^^^^^^^^^^^^^ The GNU convention (which we use in Emacs) is to only use "path" for lists of directories as in $PATH and `load-path`, so we'd call the above "full file name" (or maybe "absolute file name"). [ Of course, you're free to use the terminology you want, but I'm just pointing this out because I feel that there's a risk of confusion here. ] Stefan ^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: OSX FSEvents file watching support 2019-07-18 14:53 ` Muir Manders 2019-07-18 15:06 ` Stefan Monnier @ 2019-07-18 16:47 ` Mattias Engdegård 1 sibling, 0 replies; 24+ messages in thread From: Mattias Engdegård @ 2019-07-18 16:47 UTC (permalink / raw) To: Muir Manders; +Cc: emacs-devel 18 juli 2019 kl. 16.53 skrev Muir Manders <muir@mnd.rs>: > > Based on my testing, FSEvents reports the full file path for each change along with metadata about the file and the change. Look at that, kFSEventStreamCreateFlagFileEvents is new (to me). Thanks! On the other hand, using that flag would increase load on Emacs if there is a lot of irrelevant changes, so I'm not sure it is useful unless changes are sporadic and/or the subtrees watched carefully constrained. FSEvents does have a rate limiter of sorts (the 'latency' parameter), which works better when that flag isn't used. Since filenotify normally knows what files are of interest anyway, that may be more efficient. For monitoring an entire subtree for changes, FSEvents is clearly useful. A sensible interface would allow a filter to be specified (globs or regexp) so that this could be handled by the back-end directly, avoiding the rather expensive call-out to elisp. ^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2019-07-18 18:04 UTC | newest] Thread overview: 24+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2019-07-17 4:18 OSX FSEvents file watching support Muir Manders 2019-07-17 19:26 ` Michael Albinus 2019-07-18 9:35 ` Eli Zaretskii 2019-07-18 11:31 ` Michael Albinus 2019-07-18 12:00 ` Eli Zaretskii 2019-07-18 12:16 ` Michael Albinus 2019-07-18 12:30 ` Eli Zaretskii 2019-07-18 14:14 ` Michael Albinus 2019-07-18 14:18 ` Eli Zaretskii 2019-07-18 14:29 ` Michael Albinus 2019-07-18 14:55 ` Stefan Monnier 2019-07-18 15:03 ` Michael Albinus 2019-07-18 14:30 ` Eli Zaretskii 2019-07-18 14:46 ` Michael Albinus 2019-07-18 15:14 ` Muir Manders 2019-07-18 15:28 ` Eli Zaretskii 2019-07-18 18:04 ` Michael Albinus 2019-07-18 14:38 ` Muir Manders 2019-07-18 9:54 ` Mattias Engdegård 2019-07-18 11:26 ` Michael Albinus 2019-07-18 11:52 ` Mattias Engdegård 2019-07-18 14:53 ` Muir Manders 2019-07-18 15:06 ` Stefan Monnier 2019-07-18 16:47 ` Mattias Engdegård
Code repositories for project(s) associated with this public inbox https://git.savannah.gnu.org/cgit/emacs.git This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).