bug#40023: 26.3 Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Robert Pluim <rpluim@gmail.com>]

>>>>> On Sun, 12 Apr 2020 12:19:44 +0200, Michael Albinus <michael.albinus@gmx.de> said:

    Michael> Mattias Engdegård <mattiase@acm.org> writes:
    >> Actually reducing RLIMIT_NOFILE doesn't sound like a terrific idea --
    >> if anything, we need as many descriptors as we can get, in particular
    >> on macOS where kqueue uses up one per monitored file. If we are
    >> limited by FD_SETSIZE, then we shouldn't use select. Or am I missing
    >> something?

With lsp-mode I think itʼs one fd per directory containing a monitored
file, but either way itʼs a limitation, and one that people are
running into.

    Michael> kqueue limits itself to (RLIMIT_NOFILE - 50) file descriptors. If 50
    Michael> file descriptors aren't sufficient, we could increase that number, or
    Michael> make it a defvar changeable via Lisp.

50 is enough. The issue is that ccls uses up the other FD_SETSIZE - 50
descriptors, and then other libraries end up passing descriptors >
FD_SETSIZE back to emacs because RLIMIT_NOFILE has been changed behind
our back.

I only see two solutions:

- the patch from
  <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40023#32>, which
  reduces RLIMIT_NOFILE to FD_SETSIZE. It works for me, but I have to
  artificially generate the crashing conditions, so some feedback from
  others would be nice (especially if we want to put it in
  emacs-27). Plus it limits the number of open file descriptors even
  on systems where crazy users want to monitor 10k files (maybe thatʼs
  a good thing :-) ).

- select() -> poll(). GNU/Linux, macOS and *BSD all have poll, plus
  thereʼs a gnulib module for it that we could perhaps use on
  MS-Windows. As Eli says, not a small change.

(increasing FD_SETSIZE is a no-go: it works on macOS, maybe on *BSD,
and is on very shaky standards-compliance ground).

Robert