Landing feature/more-fds for Emacs 30?

Landing feature/more-fds for Emacs 30?

[Elliott Shugerman]

[-- Attachment #1: Type: text/plain, Size: 1039 bytes --]

Hi,

(Context:
https://lists.gnu.org/archive/html/emacs-devel/2022-05/msg00115.html)

I've been running this patch (rebased on emacs-29) on macOS ever since we added
it to homebrew-emacs-plus
<https://github.com/d12frosted/homebrew-emacs-plus/pull/510> (off by
default) a few months back. It's made a huge difference for me!

I work on a large codebase with LSP and used to run into the "too many open
files" error all the time unless I disabled desired functionality or gave
up native-comp. Now, with this patch, not only is that issue gone entirely,
but I find Emacs is generally more responsive and stable on macOS.

More recently I installed this patch on my GNU/Linux box as well and have
noticed no differences at all there (which is good/expected because I've
only ever run into such issues on macOS, in part because I make heavier use
of it these days (reluctantly), but also due to technical differences IIUC).

Anyway, that's all to say, thanks for this patch! And... can it be merged
now that emacs-29 is cut? :)

Best,
Elliott

[-- Attachment #2: Type: text/html, Size: 1668 bytes --]

Re: Landing feature/more-fds for Emacs 30?

[Robert Pluim]

>>>>> On Tue, 14 Mar 2023 12:09:37 -0400, Elliott Shugerman <eeshugerman@gmail.com> said:

    Elliott> Hi,
    Elliott> (Context:
    Elliott> https://lists.gnu.org/archive/html/emacs-devel/2022-05/msg00115.html)

    Elliott> I've been running this patch (rebased on emacs-29) on macOS ever since we added
    Elliott> it to homebrew-emacs-plus
    Elliott> <https://github.com/d12frosted/homebrew-emacs-plus/pull/510> (off by
    Elliott> default) a few months back. It's made a huge difference for me!

    Elliott> I work on a large codebase with LSP and used to run into the "too many open
    Elliott> files" error all the time unless I disabled desired functionality or gave
    Elliott> up native-comp. Now, with this patch, not only is that issue gone entirely,
    Elliott> but I find Emacs is generally more responsive and stable on macOS.

Thatʼs good to know. Iʼve not had a great deal of feedback on it.

    Elliott> More recently I installed this patch on my GNU/Linux box as well and have
    Elliott> noticed no differences at all there (which is good/expected because I've
    Elliott> only ever run into such issues on macOS, in part because I make heavier use
    Elliott> of it these days (reluctantly), but also due to technical differences IIUC).

The issue tends to happen when you have file notification turned on,
and the Linux kernel implementation of that scales better than the
macOS one (unless we switch to the new macOS notification api, but we
already have 3 different ones...)

    Elliott> Anyway, that's all to say, thanks for this patch! And... can it be merged
    Elliott> now that emacs-29 is cut? :)

Not as is. There are a number of other parts of Emacs that need
adapting to the possibility if having >1024 open filedescriptors. I
have most of that done, but as usual Real Life™ interferes.

Iʼll see if I can schedule some time to finish it off. After all,
Emacs 30 needs some changes that can cause instability in fundamental
code ;-)

Robert
-- 

Re: Landing feature/more-fds for Emacs 30?

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Tue, 14 Mar 2023 18:08:25 +0100
> 
> After all, Emacs 30 needs some changes that can cause instability in
> fundamental code ;-)

Yes, definitely.

Re: Landing feature/more-fds for Emacs 30?

[Michael Albinus]

Robert Pluim <rpluim@gmail.com> writes:

Hi,

> The issue tends to happen when you have file notification turned on,
> and the Linux kernel implementation of that scales better than the
> macOS one (unless we switch to the new macOS notification api, but we
> already have 3 different ones...)

Btw, patches welcome to replace kqueue by FSEvents(?) for macOS.

> Robert

Best regards, Michael.

Re: Landing feature/more-fds for Emacs 30?

[Robert Pluim]

>>>>> On Wed, 15 Mar 2023 09:48:15 +0100, Michael Albinus <michael.albinus@gmx.de> said:

    Michael> Robert Pluim <rpluim@gmail.com> writes:
    Michael> Hi,

    >> The issue tends to happen when you have file notification turned on,
    >> and the Linux kernel implementation of that scales better than the
    >> macOS one (unless we switch to the new macOS notification api, but we
    >> already have 3 different ones...)

    Michael> Btw, patches welcome to replace kqueue by FSEvents(?) for macOS.

That fixes it for macOS, but then *BSD users still have problems.

Robert
-- 

Re: Landing feature/more-fds for Emacs 30?

[Michael Albinus]

Robert Pluim <rpluim@gmail.com> writes:

Hi Robert,

>     Michael> Btw, patches welcome to replace kqueue by FSEvents(?) for macOS.
>
> That fixes it for macOS, but then *BSD users still have problems.

Sure, but this cannot be solved by us. Or does there exist an
alternative to kqueue on *BSD, which we could integrate?

> Robert

Best regards, Michael.

Re: Landing feature/more-fds for Emacs 30?

[Robert Pluim]

>>>>> On Wed, 15 Mar 2023 10:31:32 +0100, Michael Albinus <michael.albinus@gmx.de> said:

    Michael> Robert Pluim <rpluim@gmail.com> writes:
    Michael> Hi Robert,

    Michael> Btw, patches welcome to replace kqueue by FSEvents(?) for macOS.
    >> 
    >> That fixes it for macOS, but then *BSD users still have problems.

    Michael> Sure, but this cannot be solved by us. Or does there exist an
    Michael> alternative to kqueue on *BSD, which we could integrate?

Not that I know of. But moving to `poll' fixes it for everyone

Robert
-- 

Re: Landing feature/more-fds for Emacs 30?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

Hi Elliott,

On Tue, Mar 14, 2023, 16:10 Elliott Shugerman <eeshugerman@gmail.com> wrote:

>
> I work on a large codebase with LSP and used to run into the "too many
> open files" error all the time unless I disabled desired functionality or
> gave up native-comp. Now, with this patch, not only is that issue gone
> entirely, but I find Emacs is generally more responsive and stable on macOS.
>

What LSP server are you using? Is it perhaps requesting many file watchers?
Or maybe the client is overwatching on the Emacs side (what client are you
using?). Do you have a JSONRPC transcript that you can share with me?

João

[-- Attachment #2: Type: text/html, Size: 1182 bytes --]

Re: Landing feature/more-fds for Emacs 30?

[Robert Pluim]

>>>>> On Wed, 15 Mar 2023 17:12:19 +0000, João Távora <joaotavora@gmail.com> said:

    João> Hi Elliott,
    João> On Tue, Mar 14, 2023, 16:10 Elliott Shugerman <eeshugerman@gmail.com> wrote:

    >> 
    >> I work on a large codebase with LSP and used to run into the "too many
    >> open files" error all the time unless I disabled desired functionality or
    >> gave up native-comp. Now, with this patch, not only is that issue gone
    >> entirely, but I find Emacs is generally more responsive and stable on macOS.
    >> 

    João> What LSP server are you using? Is it perhaps requesting many file watchers?
    João> Or maybe the client is overwatching on the Emacs side (what client are you
    João> using?). Do you have a JSONRPC transcript that you can share with me?

Itʼs a macOS thing: file watching there uses a file descriptor
per-file, unlike Linux, which uses one per-directory.

Robert
-- 

Re: Landing feature/more-fds for Emacs 30?

[João Távora]

[-- Attachment #1: Type: text/plain, Size: 1420 bytes --]

On Wed, Mar 15, 2023, 17:37 Robert Pluim <rpluim@gmail.com> wrote:

> >>>>> On Wed, 15 Mar 2023 17:12:19 +0000, João Távora <
> joaotavora@gmail.com> said:
>
>     João> Hi Elliott,
>     João> On Tue, Mar 14, 2023, 16:10 Elliott Shugerman <
> eeshugerman@gmail.com> wrote:
>
>     >>
>     >> I work on a large codebase with LSP and used to run into the "too
> many
>     >> open files" error all the time unless I disabled desired
> functionality or
>     >> gave up native-comp. Now, with this patch, not only is that issue
> gone
>     >> entirely, but I find Emacs is generally more responsive and stable
> on macOS.
>     >>
>
>     João> What LSP server are you using? Is it perhaps requesting many
> file watchers?
>     João> Or maybe the client is overwatching on the Emacs side (what
> client are you
>     João> using?). Do you have a JSONRPC transcript that you can share
> with me?
>
> Itʼs a macOS thing: file watching there uses a file descriptor
> per-file, unlike Linux, which uses one per-directory.
>

Yes, for sure that should be improved.

But the watches that the client are could also be reduced, perhaps. Maybe,
depending on the use case, it makes sense to only watch files on behalf of
the LSP that Emacs is actually visiting (Eglot doesn't do that, but maybe
it could). All depends on the use case, and that's what I'm interested in.

João

>

[-- Attachment #2: Type: text/html, Size: 2260 bytes --]

Re: Landing feature/more-fds for Emacs 30?

[Elliott Shugerman]

[-- Attachment #1: Type: text/plain, Size: 4012 bytes --]

Hi João,

Thanks for your interest in this issue and apologies for my slow replies.

> > I always figured the purpose of these watchers is to account for
switching git branches and the like, which is functionality I appreciate,
FWIW.
> I'm not sure I understand what you mean.  In rust-analyzer, for
> example, it watches the Cargo file (similar to a Makefile) to
> know if you're adding new libraries to it, that the language
> server must link and compile.  To know if a git branch has been
> switched, the server need to watch only one or two files under .git.

Sorry I wasn't very clear. The functionality I was alluding to is not
specific to Git, it goes for any changes to project files that are not
visited by buffers in Emacs.
Example:
- I have a single source file, "A", open in an Emacs buffer. The flycheck
icon in my modeline is green, indicating no LSP errors in the buffer (I
have lsp-mode/flycheck integration configured).
- Now, by some means outside Emacs -- a git or sed command, say -- I make a
change to source file "B", which is not visited by an Emacs buffer, in such
a way that there is now a type error in buffer "A" (maybe I've added a
method to an interface).
- Momentarily, I see the flycheck icon in my modeline turn red.

I assume it's to support this kind of magic that lsp-mode watches so many
files (all .ts files in the project, I believe).

However I just tried this with lsp-mode + rust-analyzer, and it doesn't
seem to work there -- I have restart the server for the icon to turn red .
But it does work with lsp-mode + typescript-language-server. So maybe it's
the server watching the source files after all?

Best,
Elliott




On Sun, Mar 19, 2023 at 9:17 PM João Távora <joaotavora@gmail.com> wrote:

> On Mon, Mar 20, 2023 at 1:09 AM Elliott Shugerman <eeshugerman@gmail.com>
> wrote:
> >
> > Hi João,
> >
> >
> > > What LSP server are you using?
> >
> > typescript-language-server (ts-ls)
> >
> > > Is it perhaps requesting many file watchers?
> >
> > I don't know but are you suggesting the "too many open files" in Emacs
> is related to some global OS limit? I don't _think_ that's the case as
> other apps remain stable when I run into this, it's only Emacs starts
> acting up, but I could be wrong.
>
> No, I'm not suggesting that.
>
> Servers like to watch files to known about things like build system
> changes and preferences in advance, by means of interfaces that LSP
> itself doesn't give them.  I was merely conjecturing that the server
> could be overzealous and watching files that really don't matter that
> much, like watching all the files in a directory when it really
> only needs one file to know what it wants to know.
>
> >  > Or maybe the client is overwatching on the Emacs side (what client
> are you using?).
> >
> > I'm using lsp-mode
>
> If you have the possibility, give Eglot ago.  It's in the
> Emacs 29 pretests.  Once you have it working with M-x
> eglot stably, turn off event logging (which is on be default)
> This will reduce the memory footprint dramatically and
> improve performance,
>
> > > Do you have a JSONRPC transcript that you can share with me?
> >
> > I don't know, where would I find or generate this?
>
> I wouldn't know in lsp-mode.  In Eglot it's M-x eglot-events-buffer.
>
> >
> > > Maybe, depending on the use case, it makes sense to only watch files
> on behalf of the LSP that Emacs is actually visiting.
> >
> > I always figured the purpose of these watchers is to account for
> switching git branches and the like, which is functionality I appreciate,
> FWIW.
>
> I'm not sure I understand what you mean.  In rust-analyzer, for
> example, it watches the Cargo file (similar to a Makefile) to
> know if you're adding new libraries to it, that the language
> server must link and compile.  To know if a git branch has been
> switched, the server need to watch only one or two files under .git.
>
> João
>

[-- Attachment #2: Type: text/html, Size: 5485 bytes --]

Re: Landing feature/more-fds for Emacs 30?

[Stefan Kangas]

Robert Pluim <rpluim@gmail.com> writes:

> Not as is. There are a number of other parts of Emacs that need
> adapting to the possibility if having >1024 open filedescriptors. I
> have most of that done, but as usual Real Life™ interferes.
>
> Iʼll see if I can schedule some time to finish it off. After all,
> Emacs 30 needs some changes that can cause instability in fundamental
> code ;-)

What is the current status of merging feature/more-fds?

Re: Landing feature/more-fds for Emacs 30?

[Robert Pluim]

>>>>> On Mon, 2 Oct 2023 23:30:16 -0700, Stefan Kangas <stefankangas@gmail.com> said:

    Stefan> Robert Pluim <rpluim@gmail.com> writes:
    >> Not as is. There are a number of other parts of Emacs that need
    >> adapting to the possibility if having >1024 open filedescriptors. I
    >> have most of that done, but as usual Real Life™ interferes.
    >> 
    >> Iʼll see if I can schedule some time to finish it off. After all,
    >> Emacs 30 needs some changes that can cause instability in fundamental
    >> code ;-)

    Stefan> What is the current status of merging feature/more-fds?

Iʼm rebasing it to master now. Give me a week or so to test it.

Iʼm tempted to turn it on by default on macOS, but I donʼt want to
cause the maintainers any grief, so only with permission 😇

Robert
-- 

Re: Landing feature/more-fds for Emacs 30?

[Stefan Kangas]

Robert Pluim <rpluim@gmail.com> writes:

> Iʼm rebasing it to master now. Give me a week or so to test it.

Great, thanks!

> Iʼm tempted to turn it on by default on macOS, but I donʼt want to
> cause the maintainers any grief, so only with permission 😇

I'd be in favor of enabling it by default, perhaps only provisionally.
We still have time to turn it off again before Emacs 30.1 if it turns
out to cause any major stability issues.

The performance increase on macOS is substantial enough that we'll want
it enabled by default at some point anyways, right?

Eli, WDYT?

Re: Landing feature/more-fds for Emacs 30?

[Robert Pluim]

>>>>> On Fri, 6 Oct 2023 11:23:16 +0000, Stefan Kangas <stefankangas@gmail.com> said:

    Stefan> Robert Pluim <rpluim@gmail.com> writes:
    >> Iʼm rebasing it to master now. Give me a week or so to test it.

    Stefan> Great, thanks!

    >> Iʼm tempted to turn it on by default on macOS, but I donʼt want to
    >> cause the maintainers any grief, so only with permission 😇

    Stefan> I'd be in favor of enabling it by default, perhaps only provisionally.
    Stefan> We still have time to turn it off again before Emacs 30.1 if it turns
    Stefan> out to cause any major stability issues.

Sure

    Stefan> The performance increase on macOS is substantial enough that we'll want
    Stefan> it enabled by default at some point anyways, right?

It makes various lsp and file watching use cases usable, but I make no
representations about whether Emacs is faster or slower as a result.

Gripping hand option: increase FD_SETSIZE on macOS only, which is a
minimally invasive change that doesnʼt affect nor help other
platforms, but does not have the same fun potential to break stuff :-)

Robert
-- 

Re: Landing feature/more-fds for Emacs 30?

[Eli Zaretskii]

> From: Stefan Kangas <stefankangas@gmail.com>
> Date: Fri, 6 Oct 2023 11:23:16 +0000
> Cc: Elliott Shugerman <eeshugerman@gmail.com>, emacs-devel@gnu.org, eliz@gnu.org
> 
> Robert Pluim <rpluim@gmail.com> writes:
> 
> > Iʼm tempted to turn it on by default on macOS, but I donʼt want to
> > cause the maintainers any grief, so only with permission 😇
> 
> I'd be in favor of enabling it by default, perhaps only provisionally.
> We still have time to turn it off again before Emacs 30.1 if it turns
> out to cause any major stability issues.
> 
> The performance increase on macOS is substantial enough that we'll want
> it enabled by default at some point anyways, right?
> 
> Eli, WDYT?

Assuming it passes all the tests, yes, we could turn it on by default
on master.