bug#40023: 26.3; Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

bug#40023: 26.3; Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Holger Pirk]

Hi folks,

I use LSP/CCLS on a project with a lot of dependencies. When opening a
file, LSP asks me if I want to watch all of their files (I do). Now, when I try
to compile the project from within emacs, emacs simply crashes. I have
attached a debugger (lldb). Here is a trace of the session that should
illustrate what is happening:

(lldb) r
Process 30575 launched: '/usr/local/Cellar/emacs-plus/26.3/bin/emacs' (x86_64)
2020-03-11 13:37:18.644400+0000 emacs[30575:1729696] Failed to initialize color list unarchiver: Error Domain=NSCocoaErrorDomain Code=4864 "*** -[NSKeyedUnarchiver _initForReadingFromData:error:throwLegacyExceptions:]: non-keyed archive cannot be decoded by NSKeyedUnarchiver" UserInfo={NSDebugDescription=*** -[NSKeyedUnarchiver _initForReadingFromData:error:throwLegacyExceptions:]: non-keyed archive cannot be decoded by NSKeyedUnarchiver}
2020-03-11 13:37:20.444106+0000 emacs[30575:1729754] flock failed to lock maps file: errno = 35
2020-03-11 13:37:20.444386+0000 emacs[30575:1729754] flock failed to lock maps file: errno = 35
process.c:459: Emacs fatal error: assertion failed: fd >= 0 && fd < FD_SETSIZE
Process 30575 stopped
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
    #0: `__pthread_kill
libsystem_kernel.dylib`__pthread_kill:
->  0x7fff6ac5f7fa <+10>: jae    0x7fff6ac5f804            ; <+20>
    0x7fff6ac5f7fc <+12>: movq   %rax, %rdi
    0x7fff6ac5f7ff <+15>: jmp    0x7fff6ac59a89            ; cerror_nocancel
    0x7fff6ac5f804 <+20>: retq
Target 0: (emacs) stopped.
(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT
  * #0: `__pthread_kill
    #1: `pthread_kill
    #2: `raise
    #3: `terminate_due_to_signal
    #4: `emacs_abort
    #5: `ns_term_shutdown
    #6: `shut_down_emacs
    #7: `terminate_due_to_signal
    #8: `die
    #9: `add_non_keyboard_read_fd at process.c:459
    #10: `add_process_read_fd at process.c:471
    #11: `create_process at process.c:2050
    #12: `Fmake_process at process.c:1920
    #13: `Ffuncall
    #14: `Fapply
    #15: `Ffuncall
    #16: `exec_byte_code
    #17: `Ffuncall
    #18: `Fapply
    #19: `Ffuncall
    #20: `exec_byte_code
    #21: `Ffuncall
    #22: `exec_byte_code
    #23: `Ffuncall
    #24: `exec_byte_code
    #25: `Ffuncall
    #26: `exec_byte_code
    #27: `Ffuncall
    #28: `exec_byte_code
    #29: `Ffuncall
    #30: `exec_byte_code
    #31: `Ffuncall
    #32: `exec_byte_code
    #33: `Ffuncall
    #34: `Ffuncall_interactively
    #35: `Ffuncall
    #36: `Fcall_interactively
    #37: `funcall_subr
    #38: `Ffuncall
    #39: `exec_byte_code
    #40: `Ffuncall
    #41: `call1
    #42: `command_loop_1
    #43: `internal_condition_case
    #44: `command_loop_2
    #45: `internal_catch
    #46: `command_loop
    #47: `recursive_edit_1
    #48: `Frecursive_edit
    #49: `main
    #50: `start
(lldb) f 9
#9: `add_non_keyboard_read_fd at process.c:459
   456 	static void
   457 	add_non_keyboard_read_fd (int fd)
   458 	{
-> 459 	  eassert (fd >= 0 && fd < FD_SETSIZE);
   460 	  eassert (fd_callback_info[fd].func == NULL);
   461
   462 	  fd_callback_info[fd].flags &= ~KEYBOARD_FD;
(lldb) frame variable fd
(int) fd = 1805
(lldb) f 11
#11: `create_process at process.c:2050
   2047	  pset_status (p, Qrun);
   2048
   2049	  if (!EQ (p->command, Qt))
-> 2050	    add_process_read_fd (inchannel);
   2051
   2052	  /* This may signal an error.  */
   2053	  setup_process_coding_systems (process);
(lldb) frame variable *p
(Lisp_Process) *p = {
  header = (size = 4611686018461020181, gcaligned = '\x15')
  tty_name = (i = 0)
  name = (i = 4664306260)
  command = (i = 4741329507)
  filter = (i = 28608)
  sentinel = (i = 28656)
  log = (i = 0)
  buffer = (i = 4664509813)
  childp = (i = 46464)
  plist = (i = 0)
  type = (i = 40944)
  mark = (i = 4664307705)
  status = (i = 42384)
  decode_coding_system = (i = 126929848)
  decoding_buf = (i = 4297587076)
  encode_coding_system = (i = 126929848)
  encoding_buf = (i = 4297587076)
  write_queue = (i = 0)
  gnutls_cred_type = (i = 0)
  gnutls_boot_parameters = (i = 0)
  stderrproc = (i = 0)
  thread = (i = 4302432989)
  pid = 0
  infd = 1805
  nbytes_read = 0
  outfd = 1805
  open_fd = ([0] = 1806, [1] = -1, [2] = 1805, [3] = -1, [4] = 1807, [5] = 1808)
  tick = 0
  update_tick = 0
  decoding_carryover = 0
  read_output_delay = 0
  adaptive_read_buffering = 1
  read_output_skip = false
  kill_without_query = false
  pty_flag = true
  inherit_coding_system_flag = false
  alive = false
  raw_status_new = false
  is_non_blocking_client = false
  is_server = false
  raw_status = 0
  backlog = 0
  port = 0
  socktype = 0
  gnutls_initstage = GNUTLS_STAGE_EMPTY
  gnutls_state = 0x0000000000000000
  gnutls_x509_cred = 0x0000000000000000
  gnutls_anon_cred = 0x0000000000000000
  gnutls_certificate = 0x0000000000000000
  gnutls_peer_verification = 0
  gnutls_extra_peer_verification = 0
  gnutls_log_level = 0
  gnutls_handshakes_tried = 0
  gnutls_p = false
  gnutls_complete_negotiation_p = false
}
(lldb)

I have spent some time digging around the code and couldn't quite figure
out the logic behind the `fd < FD_SETSIZE`-assert. I suspect the CCLS
opens files without going through emacs' infrastructure which leads to
high file descriptors which emacs cannot handle. I would say that CCLS
is, in part, to blame but emacs shouldn't simply crash.



In GNU Emacs 26.3 (build 2, x86_64-apple-darwin19.3.0, NS appkit-1894.30 Version 10.15.3 (Build 19D76))
of 2020-03-11 built on veloceraptor.local
Windowing system distributor 'Apple', version 10.3.1894
Recent messages:
Restarting server
Loading holgerstuff...done
Created message link.
ad-handle-definition: ‘imenu’ got redefined
ad-handle-definition: ‘xref-find-definitions’ got redefined
Starting new Ispell process /usr/local/bin/ispell with en_US dictionary...
Error enabling Flyspell mode:
(Can't open /usr/local/Cellar/ispell/3.4.00/lib/ispell/en_US.hash)
For information about GNU Emacs and the GNU system, type C-h C-a.
Mark activated

Configured using:
'configure --disable-dependency-tracking --disable-silent-rules
--enable-locallisppath=/usr/local/share/emacs/site-lisp
--infodir=/usr/local/Cellar/emacs-plus/26.3/share/info/emacs
--prefix=/usr/local/Cellar/emacs-plus/26.3 --with-xml2 --without-dbus
--with-gnutls --with-imagemagick --with-modules --with-rsvg --with-ns
--disable-ns-self-contained --enable-checking=yes,glyphs
--enable-check-lisp-object-type 'CFLAGS=-O0 -g' 'LDFLAGS=-O0 -g''

Configured features:
JPEG RSVG IMAGEMAGICK GLIB NOTIFY ACL GNUTLS LIBXML2 ZLIB
TOOLKIT_SCROLL_BARS NS MODULES THREADS LCMS2

Important settings:
  value of $LC_ALL: en_US.UTF-8
  value of $LANG: en_GB.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Org

Minor modes in effect:
  eyebrowse-mode: t
  shackle-mode: t
  spaceline-helm-mode: t
  yas-minor-mode: t
  async-bytecomp-package-mode: t
  override-global-mode: t
  straight-use-package-mode: t
  straight-package-neutering-mode: t
  server-mode: t
  save-place-mode: t
  history-mode: t
  helm-mode: t
  helm--remap-mouse-mode: t
  global-subword-mode: t
  subword-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  global-prettify-symbols-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  line-number-mode: t
  transient-mark-mode: t

Load-path shadows:
/Users/hlgr/.emacs.d/elpa/lispy-20200305.1858/elpa hides /Users/hlgr/.emacs.d/elpa/ivy-20200309.2006/elpa
/Users/hlgr/.emacs.d/straight/build/use-package/use-package-jump hides /Users/hlgr/.emacs.d/elpa/use-package-20191126.2034/use-package-jump
/Users/hlgr/.emacs.d/straight/build/use-package/use-package-diminish hides /Users/hlgr/.emacs.d/elpa/use-package-20191126.2034/use-package-diminish
/Users/hlgr/.emacs.d/straight/build/use-package/use-package hides /Users/hlgr/.emacs.d/elpa/use-package-20191126.2034/use-package
/Users/hlgr/.emacs.d/straight/build/use-package/use-package-delight hides /Users/hlgr/.emacs.d/elpa/use-package-20191126.2034/use-package-delight
/Users/hlgr/.emacs.d/straight/build/use-package/use-package-lint hides /Users/hlgr/.emacs.d/elpa/use-package-20191126.2034/use-package-lint
/Users/hlgr/.emacs.d/straight/build/use-package/use-package-autoloads hides /Users/hlgr/.emacs.d/elpa/use-package-20191126.2034/use-package-autoloads
/Users/hlgr/.emacs.d/straight/build/use-package/use-package-core hides /Users/hlgr/.emacs.d/elpa/use-package-20191126.2034/use-package-core
/Users/hlgr/.emacs.d/straight/build/use-package/use-package-ensure hides /Users/hlgr/.emacs.d/elpa/use-package-20191126.2034/use-package-ensure
/Users/hlgr/.emacs.d/straight/build/use-package/use-package-bind-key hides /Users/hlgr/.emacs.d/elpa/use-package-20191126.2034/use-package-bind-key
/Users/hlgr/.emacs.d/straight/build/bind-key/bind-key-autoloads hides /Users/hlgr/.emacs.d/elpa/bind-key-20191110.416/bind-key-autoloads
/Users/hlgr/.emacs.d/straight/build/bind-key/bind-key hides /Users/hlgr/.emacs.d/elpa/bind-key-20191110.416/bind-key
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-stan hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-stan
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-exp hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-exp
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-J hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-J
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-emacs-lisp hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-emacs-lisp
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-css hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-css
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-lob hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-lob
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-forth hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-forth
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-macs hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-macs
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-version hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-version
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-scheme hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-scheme
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ox hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ox
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-abc hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-abc
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-C hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-C
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-capture hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-capture
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-ref hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-ref
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-clojure hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-clojure
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-mouse hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-mouse
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-ledger hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-ledger
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-ctags hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-ctags
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-entities hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-entities
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-archive hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-archive
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-screen hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-screen
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-haskell hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-haskell
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-asymptote hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-asymptote
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-table hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-table
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ox-org hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ox-org
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-plot hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-plot
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-awk hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-awk
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-groovy hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-groovy
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-octave hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-octave
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-faces hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-faces
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-colview hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-colview
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-R hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-R
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-timer hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-timer
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-ebnf hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-ebnf
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-mobile hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-mobile
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-fortran hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-fortran
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-shell hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-shell
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-perl hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-perl
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-sqlite hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-sqlite
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-sed hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-sed
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-list hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-list
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-ruby hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-ruby
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-eval hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-eval
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-habit hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-habit
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-clock hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-clock
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ox-html hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ox-html
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-src hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-src
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-lisp hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-lisp
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-ditaa hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-ditaa
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-pcomplete hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-pcomplete
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-lint hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-lint
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ox-latex hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ox-latex
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-sass hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-sass
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-io hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-io
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-tangle hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-tangle
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-calc hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-calc
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-java hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-java
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ox-icalendar hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ox-icalendar
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ox-md hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ox-md
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ox-beamer hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ox-beamer
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-element hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-element
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-protocol hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-protocol
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-mscgen hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-mscgen
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-gnuplot hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-gnuplot
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-latex hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-latex
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-id hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-id
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-vala hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-vala
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ox-man hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ox-man
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-feed hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-feed
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-lua hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-lua
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-table hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-table
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-ocaml hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-ocaml
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-coq hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-coq
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-picolisp hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-picolisp
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-indent hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-indent
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-lilypond hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-lilypond
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-matlab hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-matlab
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-datetree hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-datetree
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-python hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-python
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-makefile hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-makefile
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-duration hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-duration
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-agenda hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-agenda
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-dot hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-dot
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-js hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-js
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ox-publish hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ox-publish
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-inlinetask hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-inlinetask
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-org hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-org
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-core hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-core
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-compat hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-compat
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ox-odt hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ox-odt
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-plantuml hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-plantuml
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ox-ascii hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ox-ascii
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-loaddefs hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-loaddefs
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-hledger hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-hledger
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-maxima hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-maxima
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-macro hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-macro
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-sql hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-sql
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-attach hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-attach
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-processing hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-processing
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ox-texinfo hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ox-texinfo
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-crypt hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-crypt
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-footnote hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-footnote
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/org-install hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/org-install
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-comint hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-comint
/Users/hlgr/.emacs.d/elpa/org-plus-contrib-20200309/ob-shen hides /usr/local/Cellar/emacs-plus/26.3/share/emacs/26.3/lisp/org/ob-shen

Features:
(shadow sort mail-extr emacsbug sendmail winner helm-command helm-elisp
helm-eval edebug helm-info flyspell ispell org-ref org-ref-helm-bibtex
org-ref-helm helm-bibtex bibtex-completion biblio biblio-download
biblio-dissemin biblio-ieee biblio-hal biblio-dblp biblio-crossref
biblio-arxiv timezone biblio-doi biblio-core url-queue ido helm-net
browse-url xml org-ref-core reftex-cite reftex reftex-loaddefs
reftex-vars org-ref-glossary org-ref-bibtex org-ref-citeproc key-chord
doi-utils org-ref-utils org-ref-pdf url-http url-auth url-gw ol-bibtex
htmlize noxml-fold rng-nxml rng-valid nxml-mode nxml-outln nxml-rap
sgml-mode dom request autorevert url url-proxy url-privacy url-expand
url-methods url-history url-cookie url-domsuf mailcap company-prescient
prescient company cl elmacro ccls ccls-member-hierarchy
ccls-inheritance-hierarchy ccls-call-hierarchy ccls-tree ccls-code-lens
ccls-semantic-highlight ccls-common lsp lsp-ui-flycheck lsp-ui-doc
goto-addr lsp-ui-imenu lsp-ui-peek lsp-ui-sideline lsp-mode xref project
url-util tree-widget spinner network-stream nsm starttls markdown-mode
inline imenu ht filenotify f ewoc em-glob bindat lsp-ui flymake-cppcheck
flymake-easy flymake-proc flymake compile warnings helm-rg
ob-mathematica wolfram-mode smie eyebrowse hydra lv shackle trace pcase
spaceline-config spaceline-segments s spaceline powerline
powerline-separators powerline-themes org-eshell esh-var esh-io esh-cmd
esh-opt esh-ext esh-proc esh-arg esh-groups eshell esh-module esh-mode
esh-util org-gnus nnir gnus-sum gnus-group gnus-undo gnus-start
gnus-cloud nnimap nnmail mail-source tls gnutls utf7 netrc nnoo
parse-time gnus-spec gnus-int gnus-range message rmc puny rfc822 mml
mml-sec epa derived epg mm-decode mm-bodies mm-encode mail-parse rfc2231
mailabbrev gmm-utils mailheader gnus-win gnus nnheader gnus-util rmail
rmail-loaddefs rfc2047 rfc2045 ietf-drums mail-utils mm-util mail-prsvr
org-eww org-docview doc-view jka-compr image-mode dired dired-loaddefs
org-bibtex org-bbdb ample-light-theme ample-flat-theme ample-theme
dash-functional ebib ivy delsel colir color ivy-overlay
ebib-reading-list ebib-notes ebib-filters ebib-keywords ebib-utils
ebib-db parsebib pp crm bibtex ox-slimhtml ox-odt rng-loc rng-uri
rng-parse rng-match rng-dt rng-util rng-pttrn nxml-parse nxml-ns
nxml-enc xmltok nxml-util ox-latex ox-icalendar ox-html table ox-ascii
ox-publish ox org-element org ob ob-tangle ob-ref ob-lob ob-table
org-macro org-footnote org-src ob-comint org-pcomplete pcomplete comint
org-list org-faces org-entities noutline outline org-version
ob-emacs-lisp org-table org-keys org-loaddefs cal-menu calendar
cal-loaddefs avl-tree generator ol ob-exp ob-core org-compat ob-eval
org-macs fringe-current-line cc-mode cc-fonts cc-guess cc-menus cc-cmds
cc-styles cc-align cc-engine cc-vars cc-defs window-purpose
window-purpose-fixes window-purpose-prefix-overload
window-purpose-switch let-alist window-purpose-layout ring
window-purpose-core window-purpose-configuration window-purpose-utils
yasnippet-snippets yasnippet elec-pair cus-edit wid-edit flycheck
ansi-color json map find-func rx dash helm-config helm-easymenu
async-bytecomp use-package-bind-key bind-key use-package-ensure
use-package-core use-package-autoloads bind-key-autoloads
straight-autoloads cl-extra help-mode straight subr-x server saveplace
history thingatpt helm-mode helm-files helm-buffers helm-occur helm-tags
helm-locate helm-grep helm-regexp format-spec helm-utils helm-help
helm-types helm easy-mmode edmacro kmacro helm-source eieio-compat
helm-multi-match helm-lib advice async cap-words superword subword
hl-line cus-start cus-load finder-inf tex-site info package easymenu
epg-config url-handlers url-parse auth-source cl-seq eieio eieio-core
cl-macs eieio-loaddefs password-cache url-vars seq byte-opt gv bytecomp
byte-compile cconv cl-loaddefs cl-lib time-date tooltip eldoc electric
uniquify ediff-hook vc-hooks lisp-float-type mwheel term/ns-win ns-win
ucs-normalize mule-util term/common-win tool-bar dnd fontset image
regexp-opt fringe tabulated-list replace newcomment text-mode elisp-mode
lisp-mode prog-mode register page menu-bar rfn-eshadow isearch timer
select scroll-bar mouse jit-lock font-lock syntax facemenu font-core
term/tty-colors frame cl-generic cham georgian utf-8-lang misc-lang
vietnamese tibetan thai tai-viet lao korean japanese eucjp-ms cp51932
hebrew greek romanian slovak czech european ethiopic indian cyrillic
chinese composite charscript charprop case-table epa-hook jka-cmpr-hook
help simple abbrev obarray minibuffer cl-preloaded nadvice loaddefs
button faces cus-face macroexp files text-properties overlay sha1 md5
base64 format env code-pages mule custom widget hashtable-print-readable
backquote threads kqueue cocoa ns lcms2 multi-tty make-network-process
emacs)

Memory information:
((conses 16 634628 326410)
(symbols 48 60322 2)
(miscs 40 112 560)
(strings 32 163812 50088)
(string-bytes 1 5593472)
(vectors 16 90885)
(vector-slots 8 1338691 257952)
(floats 8 577 798)
(intervals 56 734 167)
(buffers 992 13))

bug#40023: 26.3; Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Robert Pluim]

>>>>> On Wed, 11 Mar 2020 13:50:22 +0000, Holger Pirk <holger@pirk.name> said:

    Holger> I have spent some time digging around the code and couldn't quite figure
    Holger> out the logic behind the `fd < FD_SETSIZE`-assert. I suspect the CCLS
    Holger> opens files without going through emacs' infrastructure which leads to
    Holger> high file descriptors which emacs cannot handle. I would say that CCLS
    Holger> is, in part, to blame but emacs shouldn't simply crash.

Emacs is limited by the limits of select, which supports a maximum of
FD_SETSIZE file descriptors, which on macOS is 1024

At least on macOS, itʼs possible to increase that value to eg 8192 by
setting FD_SETSIZE, which might alleviate this, but then you'd
probably run into the 'ulimit -n' maximum, so you'd have to remember
to increase that.

Using 'poll' would help, but as far as I know nobody is working on
converting emacs to use it.

It would be interesting to see how CCLS opens files. How does it pass
the descriptor back to emacs?

Robert

bug#40023: 26.3; Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Wed, 11 Mar 2020 16:32:14 +0100
> Cc: 40023@debbugs.gnu.org
> 
> >>>>> On Wed, 11 Mar 2020 13:50:22 +0000, Holger Pirk <holger@pirk.name> said:
> 
>     Holger> I have spent some time digging around the code and couldn't quite figure
>     Holger> out the logic behind the `fd < FD_SETSIZE`-assert. I suspect the CCLS
>     Holger> opens files without going through emacs' infrastructure which leads to
>     Holger> high file descriptors which emacs cannot handle. I would say that CCLS
>     Holger> is, in part, to blame but emacs shouldn't simply crash.
> 
> Emacs is limited by the limits of select, which supports a maximum of
> FD_SETSIZE file descriptors, which on macOS is 1024
> 
> At least on macOS, itʼs possible to increase that value to eg 8192 by
> setting FD_SETSIZE, which might alleviate this, but then you'd
> probably run into the 'ulimit -n' maximum, so you'd have to remember
> to increase that.

Wasn't this fixed lately by Yamamoto-san?

bug#40023: 26.3; Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Holger Pirk]

[-- Attachment #1: Type: text/plain, Size: 1319 bytes --]

I think it is fine if this is an open problem but I would like more graceful failure. Isn’t it possible to catch this behavior and turn it into a lisp-level error?

On 11 Mar 2020, at 16:17, Eli Zaretskii <eliz@gnu.org<mailto:eliz@gnu.org>> wrote:

From: Robert Pluim <rpluim@gmail.com<mailto:rpluim@gmail.com>>
Date: Wed, 11 Mar 2020 16:32:14 +0100
Cc: 40023@debbugs.gnu.org<mailto:40023@debbugs.gnu.org>

On Wed, 11 Mar 2020 13:50:22 +0000, Holger Pirk <holger@pirk.name<mailto:holger@pirk.name>> said:

   Holger> I have spent some time digging around the code and couldn't quite figure
   Holger> out the logic behind the `fd < FD_SETSIZE`-assert. I suspect the CCLS
   Holger> opens files without going through emacs' infrastructure which leads to
   Holger> high file descriptors which emacs cannot handle. I would say that CCLS
   Holger> is, in part, to blame but emacs shouldn't simply crash.

Emacs is limited by the limits of select, which supports a maximum of
FD_SETSIZE file descriptors, which on macOS is 1024

At least on macOS, itʼs possible to increase that value to eg 8192 by
setting FD_SETSIZE, which might alleviate this, but then you'd
probably run into the 'ulimit -n' maximum, so you'd have to remember
to increase that.

Wasn't this fixed lately by Yamamoto-san?


[-- Attachment #2: Type: text/html, Size: 3548 bytes --]

bug#40023: 26.3; Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Robert Pluim]

>>>>> On Wed, 11 Mar 2020 18:17:02 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Date: Wed, 11 Mar 2020 16:32:14 +0100
    >> Cc: 40023@debbugs.gnu.org
    >> 
    >> >>>>> On Wed, 11 Mar 2020 13:50:22 +0000, Holger Pirk <holger@pirk.name> said:
    >> 
    Holger> I have spent some time digging around the code and couldn't quite figure
    Holger> out the logic behind the `fd < FD_SETSIZE`-assert. I suspect the CCLS
    Holger> opens files without going through emacs' infrastructure which leads to
    Holger> high file descriptors which emacs cannot handle. I would say that CCLS
    Holger> is, in part, to blame but emacs shouldn't simply crash.
    >> 
    >> Emacs is limited by the limits of select, which supports a maximum of
    >> FD_SETSIZE file descriptors, which on macOS is 1024
    >> 
    >> At least on macOS, itʼs possible to increase that value to eg 8192 by
    >> setting FD_SETSIZE, which might alleviate this, but then you'd
    >> probably run into the 'ulimit -n' maximum, so you'd have to remember
    >> to increase that.

    Eli> Wasn't this fixed lately by Yamamoto-san?

You mean Bug#39164? I donʼt think there was a commit for that.

Robert

bug#40023: 26.3; Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Robert Pluim]

>>>>> On Wed, 11 Mar 2020 16:45:45 +0000, Holger Pirk <holger@pirk.name> said:

    Holger> I think it is fine if this is an open problem but I would
    Holger> like more graceful failure. Isn’t it possible to catch
    Holger> this behavior and turn it into a lisp-level error?

Probably. Iʼm still curious as to what ccls is doing that results in
these file descriptors (or maybe itʼs not ccls' fault, but the
underlying OS).

Robert

bug#40023: 26.3; Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: holger@pirk.name,  40023@debbugs.gnu.org
> Date: Thu, 12 Mar 2020 08:26:03 +0100
> 
> >>>>> On Wed, 11 Mar 2020 18:17:02 +0200, Eli Zaretskii <eliz@gnu.org> said:
> 
>     Eli> Wasn't this fixed lately by Yamamoto-san?
> 
> You mean Bug#39164? I donʼt think there was a commit for that.

That's too bad.  I think we should revive that bug and fix it in Emacs
27, since I understand the recent versions of macOS will have that
effect more and more frequently.

bug#40023: 26.3; Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Robert Pluim]

>>>>> On Thu, 12 Mar 2020 17:11:39 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: holger@pirk.name,  40023@debbugs.gnu.org
    >> Date: Thu, 12 Mar 2020 08:26:03 +0100
    >> 
    >> >>>>> On Wed, 11 Mar 2020 18:17:02 +0200, Eli Zaretskii <eliz@gnu.org> said:
    >> 
    Eli> Wasn't this fixed lately by Yamamoto-san?
    >> 
    >> You mean Bug#39164? I donʼt think there was a commit for that.

    Eli> That's too bad.  I think we should revive that bug and fix it in Emacs
    Eli> 27, since I understand the recent versions of macOS will have that
    Eli> effect more and more frequently.

If someone has an 'emacs -Q' testcase I can look at it. Soon we will
all be spending lots of time at home here :-(

Robert

bug#40023: 26.3; Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: holger@pirk.name,  40023@debbugs.gnu.org
> Date: Thu, 12 Mar 2020 16:24:44 +0100
> 
>     >> You mean Bug#39164? I donʼt think there was a commit for that.
> 
>     Eli> That's too bad.  I think we should revive that bug and fix it in Emacs
>     Eli> 27, since I understand the recent versions of macOS will have that
>     Eli> effect more and more frequently.
> 
> If someone has an 'emacs -Q' testcase I can look at it. Soon we will
> all be spending lots of time at home here :-(

We could simply install the proposed workaround on the emacs-27 branch
for now, I think.

bug#40023: 26.3; Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Robert Pluim]

>>>>> On Thu, 12 Mar 2020 17:50:14 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: holger@pirk.name,  40023@debbugs.gnu.org
    >> Date: Thu, 12 Mar 2020 16:24:44 +0100
    >> 
    >> >> You mean Bug#39164? I donʼt think there was a commit for that.
    >> 
    Eli> That's too bad.  I think we should revive that bug and fix it in Emacs
    Eli> 27, since I understand the recent versions of macOS will have that
    Eli> effect more and more frequently.
    >> 
    >> If someone has an 'emacs -Q' testcase I can look at it. Soon we will
    >> all be spending lots of time at home here :-(

    Eli> We could simply install the proposed workaround on the emacs-27 branch
    Eli> for now, I think.

Holger, is it possible for you to rebuild emacs with the following
patch and see if Emacs still crashes? If it fixes things Iʼll install
to emacs-27

diff --git a/src/nsterm.m b/src/nsterm.m
index aefbb2721e..90f63bc182 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -5805,6 +5805,22 @@ - (void)applicationDidFinishLaunching: (NSNotification *)notification
 #endif
 
 #ifdef NS_IMPL_COCOA
+  /* Some functions/methods in CoreFoundation/Foundation increase the
+     maximum number of open files for the process in their first call.
+     We make dummy calls to them and then reduce the resource limit
+     here, since pselect cannot handle file descriptors that are
+     greater than or equal to FD_SETSIZE.  */
+  CFSocketGetTypeID ();
+  CFFileDescriptorGetTypeID ();
+  [[NSFileHandle alloc] init];
+  struct rlimit rlim;
+  if (getrlimit (RLIMIT_NOFILE, &rlim) == 0
+      && rlim.rlim_cur > FD_SETSIZE)
+    {
+      rlim.rlim_cur = FD_SETSIZE;
+      setrlimit (RLIMIT_NOFILE, &rlim);
+    }
+
   if ([NSApp activationPolicy] == NSApplicationActivationPolicyProhibited) {
     /* Set the app's activation policy to regular when we run outside
        of a bundle.  This is already done for us by Info.plist when we

bug#40023: 26.3; Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Robert Pluim]

>>>>> On Thu, 12 Mar 2020 17:46:58 +0100, Robert Pluim <rpluim@gmail.com> said:
    >>> If someone has an 'emacs -Q' testcase I can look at it. Soon we will
    >>> all be spending lots of time at home here :-(

    Eli> We could simply install the proposed workaround on the emacs-27 branch
    Eli> for now, I think.

So I installed lsp-mode and ccls, added lsp to c-mode-hook, visited
emacs/src/font.c and then said yes when it asked me if I wanted to
watch all the files (there are ~6500). I then recompiled emacs using
M-x compile, and saw no crash.

This is with emacs-master. I checked with emacs-26 and saw no crash. I
did look in *lsp-log*, and saw a bunch of messages saying:

Failed to create a watch for File watching not possible, no file descriptor left: 975: message

(this is with 'ulimit -n 4096')

so I think this is very much dependent on which language server youʼre
using. Note also that itʼs watching directories, not files, I had to
create a bunch of extra directories to get that message.

I have:

    ccls --version
    ccls version 8.0.0 (tags/RELEASE_800/final)

Robert

bug#40023: 26.3; Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: 40023@debbugs.gnu.org,  holger@pirk.name
> Date: Thu, 19 Mar 2020 11:41:37 +0100
> 
> So I installed lsp-mode and ccls, added lsp to c-mode-hook, visited
> emacs/src/font.c and then said yes when it asked me if I wanted to
> watch all the files (there are ~6500). I then recompiled emacs using
> M-x compile, and saw no crash.

I thought this only happens on some (recent enough) versions of the
OS?

Anyway, thanks for trying.

bug#40023: 26.3 Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Mattias Engdegård]

Actually reducing RLIMIT_NOFILE doesn't sound like a terrific idea -- if anything, we need as many descriptors as we can get, in particular on macOS where kqueue uses up one per monitored file. If we are limited by FD_SETSIZE, then we shouldn't use select. Or am I missing something?

bug#40023: 26.3 Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Sat, 11 Apr 2020 20:15:37 +0200
> Cc: Robert Pluim <rpluim@gmail.com>, Eli Zaretskii <eliz@gnu.org>
> 
> Actually reducing RLIMIT_NOFILE doesn't sound like a terrific idea -- if anything, we need as many descriptors as we can get, in particular on macOS where kqueue uses up one per monitored file. If we are limited by FD_SETSIZE, then we shouldn't use select. Or am I missing something?

We are talking about the emacs-27 branch, where such deep changes are
out of the question.

bug#40023: 26.3 Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Michael Albinus]

Mattias Engdegård <mattiase@acm.org> writes:

> Actually reducing RLIMIT_NOFILE doesn't sound like a terrific idea --
> if anything, we need as many descriptors as we can get, in particular
> on macOS where kqueue uses up one per monitored file. If we are
> limited by FD_SETSIZE, then we shouldn't use select. Or am I missing
> something?

kqueue limits itself to (RLIMIT_NOFILE - 50) file descriptors. If 50
file descriptors aren't sufficient, we could increase that number, or
make it a defvar changeable via Lisp.

See line 397 of kqueue.c

Best regards, Michael.

bug#40023: 26.3 Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Robert Pluim]

>>>>> On Sun, 12 Apr 2020 12:19:44 +0200, Michael Albinus <michael.albinus@gmx.de> said:

    Michael> Mattias Engdegård <mattiase@acm.org> writes:
    >> Actually reducing RLIMIT_NOFILE doesn't sound like a terrific idea --
    >> if anything, we need as many descriptors as we can get, in particular
    >> on macOS where kqueue uses up one per monitored file. If we are
    >> limited by FD_SETSIZE, then we shouldn't use select. Or am I missing
    >> something?

With lsp-mode I think itʼs one fd per directory containing a monitored
file, but either way itʼs a limitation, and one that people are
running into.

    Michael> kqueue limits itself to (RLIMIT_NOFILE - 50) file descriptors. If 50
    Michael> file descriptors aren't sufficient, we could increase that number, or
    Michael> make it a defvar changeable via Lisp.

50 is enough. The issue is that ccls uses up the other FD_SETSIZE - 50
descriptors, and then other libraries end up passing descriptors >
FD_SETSIZE back to emacs because RLIMIT_NOFILE has been changed behind
our back.

I only see two solutions:

- the patch from
  <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40023#32>, which
  reduces RLIMIT_NOFILE to FD_SETSIZE. It works for me, but I have to
  artificially generate the crashing conditions, so some feedback from
  others would be nice (especially if we want to put it in
  emacs-27). Plus it limits the number of open file descriptors even
  on systems where crazy users want to monitor 10k files (maybe thatʼs
  a good thing :-) ).

- select() -> poll(). GNU/Linux, macOS and *BSD all have poll, plus
  thereʼs a gnulib module for it that we could perhaps use on
  MS-Windows. As Eli says, not a small change.

(increasing FD_SETSIZE is a no-go: it works on macOS, maybe on *BSD,
and is on very shaky standards-compliance ground).

Robert

bug#40023: 26.3 Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Mattias Engdegård]

14 apr. 2020 kl. 15.58 skrev Robert Pluim <rpluim@gmail.com>:

> With lsp-mode I think itʼs one fd per directory containing a monitored
> file, but either way itʼs a limitation, and one that people are
> running into.

Are you sure? In contrast to inotify, kqueue can't detect changes to the contents of files (or to their inodes) by monitoring their containing directory.

> - the patch from
>  <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40023#32>, which
>  reduces RLIMIT_NOFILE to FD_SETSIZE.

It's perhaps the least bad option for emacs-27. In addition to the file monitoring requirement, there's the concern that the macOS libs raise RLIMIT_NOFILE for reasons of their own, and that more things will break when they no longer get what they want.

> - select() -> poll(). GNU/Linux, macOS and *BSD all have poll, plus
>  thereʼs a gnulib module for it that we could perhaps use on
>  MS-Windows. As Eli says, not a small change.

Right. macOS doesn't have ppoll but Emacs doesn't seem to make any use of the sigmask argument.

> (increasing FD_SETSIZE is a no-go: it works on macOS, maybe on *BSD,
> and is on very shaky standards-compliance ground).

Agreed.

bug#40023: 26.3 Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Robert Pluim]

>>>>> On Tue, 14 Apr 2020 18:02:47 +0200, Mattias Engdegård <mattiase@acm.org> said:

    Mattias> 14 apr. 2020 kl. 15.58 skrev Robert Pluim <rpluim@gmail.com>:
    >> With lsp-mode I think itʼs one fd per directory containing a monitored
    >> file, but either way itʼs a limitation, and one that people are
    >> running into.

    Mattias> Are you sure? In contrast to inotify, kqueue can't detect changes to
    Mattias> the contents of files (or to their inodes) by monitoring their
    Mattias> containing directory.

C-x C-f my-dodgy-memory-disclaimer.txt :-)

    >> - the patch from
    >> <https://debbugs.gnu.org/cgi/bugreport.cgi?bug=40023#32>, which
    >> reduces RLIMIT_NOFILE to FD_SETSIZE.

    Mattias> It's perhaps the least bad option for emacs-27. In addition to the
    Mattias> file monitoring requirement, there's the concern that the macOS libs
    Mattias> raise RLIMIT_NOFILE for reasons of their own, and that more things
    Mattias> will break when they no longer get what they want.

Libraries have no business messing with RLIMIT_NOFILE, but this is
macOS weʼre talking about.

    >> - select() -> poll(). GNU/Linux, macOS and *BSD all have poll, plus
    >> thereʼs a gnulib module for it that we could perhaps use on
    >> MS-Windows. As Eli says, not a small change.

    Mattias> Right. macOS doesn't have ppoll but Emacs doesn't seem to make any use of the sigmask argument.

Right, we always pass NULL.

Itʼs unfortunately very much a flag-day change that would need testing
on loads of different platforms. Or we #ifdef it to death.

Robert

bug#40023: 26.3 Emacs crashes when creating process if many file handles are in use (e.g., when using ccls)

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Date: Tue, 14 Apr 2020 18:14:19 +0200
> Cc: Michael Albinus <michael.albinus@gmx.de>, 40023@debbugs.gnu.org
> 
> Itʼs unfortunately very much a flag-day change that would need testing
> on loads of different platforms. Or we #ifdef it to death.

The user who reported bug#40555 said the patch we have here fixed the
problem, so I think we should install that patch on the emacs-27
branch (I hope we will have another pretest soon).

Or is there something else that's needed before we can install that
patch?

Thanks.