1024 file descriptors should be enough for anyone

1024 file descriptors should be enough for anyone

[Robert Pluim]

[-- Attachment #1: Type: text/plain, Size: 794 bytes --]


But for those people who insist on using LSP with large codebases,
Emacs's use of 'select' is somewhat limiting.

Attached an initial attempt at using poll() instead of select(), and
bumping the open filed descriptor limit to 10240. It passes 'make
check' on GNU/Linux, FreeBSD 13, and macOS, and M-x eww
works. MS-Windows should still build with no change in functionality.

I haven't tried --with-pgtk or a lucid build, but I donʼt anticipate
any major issues there.

Comments welcome (I can stick it on a branch if needed). Things that
could be done:

- make the upper limit either statically or dynamically configurable
  (on GNU/Linux in theory you could have 1e6 open descriptors)
- a feature flag
- dynamic switching between poll and select

Thanks

Robert
-- 

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Allow-the-use-of-poll-instead-of-select.patch --]
[-- Type: text/x-diff, Size: 35853 bytes --]

From d32be58131e424450a621ffa23726dcbac090812 Mon Sep 17 00:00:00 2001
From: Robert Pluim <rpluim@gmail.com>
Date: Wed, 4 May 2022 17:09:07 +0200
Subject: [PATCH] Allow the use of poll instead of select
To: emacs-devel@gnu.org

As a bonus, increase the maximum number of open files allowed to
10 x FD_SETSIZE (which is what poll is limited to on macOS).

* configure.ac (--with-poll): New option.
(USE_POLL): New variable.
(EMACS_CONFIG_FEATURES): Add USE_POLL.
* etc/NEWS: Document --with-poll.
---
 configure.ac    |  17 +++-
 etc/NEWS        |   9 ++
 src/nsterm.m    |  11 ++-
 src/process.c   | 249 ++++++++++++++++++++++++++++++++++--------------
 src/sysdep.c    |   4 +-
 src/syspoll.h   |  34 +++++++
 src/sysselect.h |  36 ++++++-
 src/thread.c    |   3 +
 src/xgselect.c  |   4 +
 src/xmenu.c     |   3 +
 src/xterm.c     |   3 +
 11 files changed, 291 insertions(+), 82 deletions(-)
 create mode 100644 src/syspoll.h

diff --git a/configure.ac b/configure.ac
index 484ce980a5..4785f7365e 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1668,6 +1668,18 @@ AC_DEFUN
   [The type of system you are compiling for; sets 'system-type'.])
 AC_SUBST([SYSTEM_TYPE])
 
+OPTION_DEFAULT_OFF([poll],
+  [(experimental) use poll() instead of select() for checking file
+  descriptor status.])
+if test "$with_poll" = yes; then
+  AC_CHECK_HEADERS(sys/poll.h)
+  if test "x$ac_cv_header_sys_poll_h" = xyes; then
+    AC_DEFINE([USE_POLL], 1, [Define to 1 if you use poll.])
+    USE_POLL=yes
+  else
+    USE_POLL=no
+  fi
+fi
 
 pre_PKG_CONFIG_CFLAGS=$CFLAGS
 pre_PKG_CONFIG_LIBS=$LIBS
@@ -6345,7 +6357,7 @@ AC_DEFUN
 emacs_config_features=
 for opt in ACL BE_APP CAIRO DBUS FREETYPE GCONF GIF GLIB GMP GNUTLS GPM GSETTINGS \
  HARFBUZZ IMAGEMAGICK JPEG JSON LCMS2 LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 \
- M17N_FLT MODULES NATIVE_COMP NOTIFY NS OLDXMENU PDUMPER PGTK PNG RSVG SECCOMP \
+ M17N_FLT MODULES NATIVE_COMP NOTIFY NS OLDXMENU PDUMPER PGTK PNG POLL RSVG SECCOMP \
  SOUND SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS \
  UNEXEC WEBP X11 XAW3D XDBE XFT XIM XINPUT2 XPM XWIDGETS X_TOOLKIT \
  ZLIB; do
@@ -6355,7 +6367,7 @@ AC_DEFUN
       UNEXEC) val=${with_unexec} ;;
       GLIB) val=${emacs_cv_links_glib} ;;
       NOTIFY|ACL) eval val=\${${opt}_SUMMARY} ;;
-      TOOLKIT_SCROLL_BARS|X_TOOLKIT) eval val=\${USE_$opt} ;;
+      TOOLKIT_SCROLL_BARS|X_TOOLKIT|POLL) eval val=\${USE_$opt} ;;
       THREADS) val=${threads_enabled} ;;
       *) eval val=\${HAVE_$opt} ;;
     esac
@@ -6426,6 +6438,7 @@ AC_DEFUN
   Which dumping strategy does Emacs use?                  ${with_dumping}
   Does Emacs have native lisp compiler?                   ${HAVE_NATIVE_COMP}
   Does Emacs use version 2 of the the X Input Extension?  ${HAVE_XINPUT2}
+  Does Emacs use poll()?                                  ${USE_POLL}
 "])
 
 if test -n "${EMACSDATA}"; then
diff --git a/etc/NEWS b/etc/NEWS
index 6637eda00c..f689bc8d8d 100644
--- a/etc/NEWS
+++ b/etc/NEWS
@@ -39,6 +39,15 @@ C++ compiler to be present on your system.  If Emacs is not built with
 the option '--with-be-app', the resulting Emacs will only run in
 text-mode terminals.
 
+** Emacs can now use poll() and larger file descriptor sets.
+There is experimental support for using poll() instead of select() to
+check file descriptor statuses, which can be requested by using the
+'--with-poll' option to the 'configure' script.  As part of this
+feature, the maximum number of open files supported has been increased
+to 10xFD_SETSIZE (which is typically 1024).  Note that there may be
+ulimit or kernel limits that still restrict the number of simultaneous
+open files.  This feature is not supported on MS-Windows.
+
 +++
 ** Cairo drawing support has been enabled for Haiku builds.
 To enable Cairo support, ensure that the Cairo and FreeType
diff --git a/src/nsterm.m b/src/nsterm.m
index dfb7c5d202..31b13bc570 100644
--- a/src/nsterm.m
+++ b/src/nsterm.m
@@ -46,6 +46,9 @@ Updated by Christian Limpach (chris@nice.ch)
 #include "lisp.h"
 #include "blockinput.h"
 #include "sysselect.h"
+#ifdef USE_POLL
+#include "syspoll.h"
+#endif
 #include "nsterm.h"
 #include "systime.h"
 #include "character.h"
@@ -4469,7 +4472,7 @@ in certain situations (rapid incoming events).
       return -1;
     }
 
-  eassert (nfds <= FD_SETSIZE);
+  eassert (nfds <= EMACS_MAX_FD);
   for (k = 0; k < nfds; k++)
     {
       if (readfds && FD_ISSET(k, readfds)) ++nr;
@@ -5704,15 +5707,15 @@ - (void)applicationDidFinishLaunching: (NSNotification *)notification
      maximum number of open files for the process in their first call.
      We make dummy calls to them and then reduce the resource limit
      here, since pselect cannot handle file descriptors that are
-     greater than or equal to FD_SETSIZE.  */
+     greater than or equal to EMACS_MAX_FD.  */
   CFSocketGetTypeID ();
   CFFileDescriptorGetTypeID ();
   [[NSFileHandle alloc] init];
   struct rlimit rlim;
   if (getrlimit (RLIMIT_NOFILE, &rlim) == 0
-      && rlim.rlim_cur > FD_SETSIZE)
+      && rlim.rlim_cur > EMACS_MAX_FD)
     {
-      rlim.rlim_cur = FD_SETSIZE;
+      rlim.rlim_cur = EMACS_MAX_FD;
       setrlimit (RLIMIT_NOFILE, &rlim);
     }
   if ([NSApp activationPolicy] == NSApplicationActivationPolicyProhibited) {
diff --git a/src/process.c b/src/process.c
index 08a02ad942..80d9cc461f 100644
--- a/src/process.c
+++ b/src/process.c
@@ -48,7 +48,7 @@ #define PIPECONN1_P(p) false
 #ifdef HAVE_SETRLIMIT
 # include <sys/resource.h>
 
-/* If NOFILE_LIMIT.rlim_cur is greater than FD_SETSIZE, then
+/* If NOFILE_LIMIT.rlim_cur is greater than EMACS_MAX_FD, then
    NOFILE_LIMIT is the initial limit on the number of open files,
    which should be restored in child processes.  */
 static struct rlimit nofile_limit;
@@ -113,6 +113,9 @@ #define HAVE_LOCAL_SOCKETS
 #include "blockinput.h"
 #include "atimer.h"
 #include "sysselect.h"
+#ifdef USE_POLL
+#include "syspoll.h"
+#endif
 #include "syssignal.h"
 #include "syswait.h"
 #ifdef HAVE_GNUTLS
@@ -299,7 +302,7 @@ network_lookup_address_info_1 (Lisp_Object host, const char *service,
 static void child_signal_notify (void);
 
 /* Indexed by descriptor, gives the process (if any) for that descriptor.  */
-static Lisp_Object chan_process[FD_SETSIZE];
+static Lisp_Object chan_process[EMACS_MAX_FD];
 static void wait_for_socket_fds (Lisp_Object, char const *);
 
 /* Alist of elements (NAME . PROCESS).  */
@@ -311,18 +314,18 @@ network_lookup_address_info_1 (Lisp_Object host, const char *service,
    output from the process is to read at least one char.
    Always -1 on systems that support FIONREAD.  */
 
-static int proc_buffered_char[FD_SETSIZE];
+static int proc_buffered_char[EMACS_MAX_FD];
 
 /* Table of `struct coding-system' for each process.  */
-static struct coding_system *proc_decode_coding_system[FD_SETSIZE];
-static struct coding_system *proc_encode_coding_system[FD_SETSIZE];
+static struct coding_system *proc_decode_coding_system[EMACS_MAX_FD];
+static struct coding_system *proc_encode_coding_system[EMACS_MAX_FD];
 
 #ifdef DATAGRAM_SOCKETS
 /* Table of `partner address' for datagram sockets.  */
 static struct sockaddr_and_len {
   struct sockaddr *sa;
   ptrdiff_t len;
-} datagram_address[FD_SETSIZE];
+} datagram_address[EMACS_MAX_FD];
 #define DATAGRAM_CHAN_P(chan)	(datagram_address[chan].sa != 0)
 #define DATAGRAM_CONN_P(proc)                                           \
   (PROCESSP (proc) &&                                                   \
@@ -460,8 +463,112 @@ make_lisp_proc (struct Lisp_Process *p)
   /* If this fd is currently being selected on by a thread, this
      points to the thread.  Otherwise it is NULL.  */
   struct thread_state *waiting_thread;
-} fd_callback_info[FD_SETSIZE];
+} fd_callback_info[EMACS_MAX_FD];
+
+#ifdef USE_POLL
+struct pollfd pollfds[EMACS_MAX_FD];
+
+/* Convert a read set and a write set to the corresponding array of
+   struct pollfd.  maxfds is the highest fd set in the sets.  Returns
+   the number of file descriptors set in the array.  rset and wset can
+   be NULL, in which case they will be treated as if they were empty
+   sets.  */
+
+int
+fd_sets_to_pollfds (fd_set *rset, fd_set *wset, int maxfds)
+{
+  int poll_idx = 0;
+  fd_set dummy_rset;
+  fd_set dummy_wset;
 
+  if (!rset)
+    {
+      FD_ZERO (&dummy_rset);
+      rset = &dummy_rset;
+    }
+  if (!wset)
+    {
+      FD_ZERO (&dummy_wset);
+      wset = &dummy_wset;
+    }
+  for (int i = 0; i < maxfds; i++)
+    {
+      short flag = 0;
+      if (FD_ISSET (i, rset))
+	flag |= POLLIN;
+      if (FD_ISSET (i, wset))
+	flag |= POLLOUT;
+      if (flag != 0)
+	{
+	  pollfds[poll_idx].fd = i;
+	  pollfds[poll_idx].events = flag;
+	  poll_idx++;
+	}
+    }
+  return poll_idx;
+}
+
+/* Convert an array of struct pollfd to the corresponding read and
+   write fd_sets.  poll_count is the number of file descriptors set in
+   the array.  rset and wset can be NULL, in which case they're
+   treated as if they were empty.  */
+
+void
+pollfds_to_fd_sets (fd_set *rset, fd_set *wset, int poll_count)
+{
+  fd_set dummy_rset;
+  fd_set dummy_wset;
+
+  if (!rset)
+    rset = &dummy_rset;
+  FD_ZERO (rset);
+  if (!wset)
+    wset = &dummy_wset;
+  FD_ZERO (wset);
+  for (int i = 0; i < poll_count; i++)
+    {
+      if (pollfds[i].revents & (POLLIN|POLLHUP))
+	FD_SET (pollfds[i].fd, rset);
+      if (pollfds[i].revents & POLLOUT)
+	FD_SET (pollfds[i].fd, wset);
+    }
+}
+
+/* Convert a struct timespec to the corresponding timeout in
+   milliseconds.  A NULL timespec is treated as infinity.  */
+
+int
+timespec_to_timeout (const struct timespec *ts)
+{
+  if (!ts)
+    return -1;
+  return (ts->tv_sec * 1000 + ts->tv_nsec / 1000000);
+}
+
+/* Wrapper around poll() with the calling convention of pselect.
+   Converts arguments as appropriate.  */
+int
+emacs_pselect (int nfds, fd_set *readfds, fd_set *writefds,
+	       fd_set *errorfds, const struct timespec *timeout,
+	       const sigset_t *sigmask)
+{
+  int ret;
+  int poll_count;
+
+  poll_count = fd_sets_to_pollfds (readfds, writefds, nfds);
+  ret = poll (pollfds, poll_count, timespec_to_timeout (timeout));
+  if (ret > 0)
+    pollfds_to_fd_sets(readfds, writefds, poll_count);
+  else
+    {
+      if (readfds)
+	FD_ZERO (readfds);
+      if (writefds)
+	FD_ZERO (writefds);
+    }
+  return ret;
+}
+#endif	/* USE_POLL */
 
 /* Add a file descriptor FD to be monitored for when read is possible.
    When read is possible, call FUNC with argument DATA.  */
@@ -471,7 +578,7 @@ add_read_fd (int fd, fd_callback func, void *data)
 {
   add_keyboard_wait_descriptor (fd);
 
-  eassert (0 <= fd && fd < FD_SETSIZE);
+  eassert (0 <= fd && fd < EMACS_MAX_FD);
   fd_callback_info[fd].func = func;
   fd_callback_info[fd].data = data;
 }
@@ -486,14 +593,14 @@ add_non_keyboard_read_fd (int fd, fd_callback func, void *data)
 static void
 add_process_read_fd (int fd)
 {
-  eassert (fd >= 0 && fd < FD_SETSIZE);
+  eassert (fd >= 0 && fd < EMACS_MAX_FD);
   eassert (fd_callback_info[fd].func == NULL);
 
   fd_callback_info[fd].flags &= ~KEYBOARD_FD;
   fd_callback_info[fd].flags |= FOR_READ;
   if (fd > max_desc)
     max_desc = fd;
-  eassert (0 <= fd && fd < FD_SETSIZE);
+  eassert (0 <= fd && fd < EMACS_MAX_FD);
   fd_callback_info[fd].flags |= PROCESS_FD;
 }
 
@@ -504,7 +611,7 @@ delete_read_fd (int fd)
 {
   delete_keyboard_wait_descriptor (fd);
 
-  eassert (0 <= fd && fd < FD_SETSIZE);
+  eassert (0 <= fd && fd < EMACS_MAX_FD);
   if (fd_callback_info[fd].flags == 0)
     {
       fd_callback_info[fd].func = 0;
@@ -518,7 +625,7 @@ delete_read_fd (int fd)
 void
 add_write_fd (int fd, fd_callback func, void *data)
 {
-  eassert (fd >= 0 && fd < FD_SETSIZE);
+  eassert (fd >= 0 && fd < EMACS_MAX_FD);
 
   fd_callback_info[fd].func = func;
   fd_callback_info[fd].data = data;
@@ -530,7 +637,7 @@ add_write_fd (int fd, fd_callback func, void *data)
 static void
 add_non_blocking_write_fd (int fd)
 {
-  eassert (fd >= 0 && fd < FD_SETSIZE);
+  eassert (fd >= 0 && fd < EMACS_MAX_FD);
   eassert (fd_callback_info[fd].func == NULL);
 
   fd_callback_info[fd].flags |= FOR_WRITE | NON_BLOCKING_CONNECT_FD;
@@ -544,7 +651,7 @@ recompute_max_desc (void)
 {
   int fd;
 
-  eassert (max_desc < FD_SETSIZE);
+  eassert (max_desc < EMACS_MAX_FD);
   for (fd = max_desc; fd >= 0; --fd)
     {
       if (fd_callback_info[fd].flags != 0)
@@ -553,7 +660,7 @@ recompute_max_desc (void)
 	  break;
 	}
     }
-  eassert (max_desc < FD_SETSIZE);
+  eassert (max_desc < EMACS_MAX_FD);
 }
 
 /* Stop monitoring file descriptor FD for when write is possible.  */
@@ -561,7 +668,7 @@ recompute_max_desc (void)
 void
 delete_write_fd (int fd)
 {
-  eassert (0 <= fd && fd < FD_SETSIZE);
+  eassert (0 <= fd && fd < EMACS_MAX_FD);
   if ((fd_callback_info[fd].flags & NON_BLOCKING_CONNECT_FD) != 0)
     {
       if (--num_pending_connects < 0)
@@ -584,7 +691,7 @@ compute_input_wait_mask (fd_set *mask)
   int fd;
 
   FD_ZERO (mask);
-  eassert (max_desc < FD_SETSIZE);
+  eassert (max_desc < EMACS_MAX_FD);
   for (fd = 0; fd <= max_desc; ++fd)
     {
       if (fd_callback_info[fd].thread != NULL
@@ -607,7 +714,7 @@ compute_non_process_wait_mask (fd_set *mask)
   int fd;
 
   FD_ZERO (mask);
-  eassert (max_desc < FD_SETSIZE);
+  eassert (max_desc < EMACS_MAX_FD);
   for (fd = 0; fd <= max_desc; ++fd)
     {
       if (fd_callback_info[fd].thread != NULL
@@ -631,7 +738,7 @@ compute_non_keyboard_wait_mask (fd_set *mask)
   int fd;
 
   FD_ZERO (mask);
-  eassert (max_desc < FD_SETSIZE);
+  eassert (max_desc < EMACS_MAX_FD);
   for (fd = 0; fd <= max_desc; ++fd)
     {
       if (fd_callback_info[fd].thread != NULL
@@ -655,7 +762,7 @@ compute_write_mask (fd_set *mask)
   int fd;
 
   FD_ZERO (mask);
-  eassert (max_desc < FD_SETSIZE);
+  eassert (max_desc < EMACS_MAX_FD);
   for (fd = 0; fd <= max_desc; ++fd)
     {
       if (fd_callback_info[fd].thread != NULL
@@ -677,7 +784,7 @@ clear_waiting_thread_info (void)
 {
   int fd;
 
-  eassert (max_desc < FD_SETSIZE);
+  eassert (max_desc < EMACS_MAX_FD);
   for (fd = 0; fd <= max_desc; ++fd)
     {
       if (fd_callback_info[fd].waiting_thread == current_thread)
@@ -969,10 +1076,10 @@ update_processes_for_thread_death (Lisp_Object dying_thread)
 	  struct Lisp_Process *proc = XPROCESS (process);
 
 	  pset_thread (proc, Qnil);
-	  eassert (proc->infd < FD_SETSIZE);
+	  eassert (proc->infd < EMACS_MAX_FD);
 	  if (proc->infd >= 0)
 	    fd_callback_info[proc->infd].thread = NULL;
-	  eassert (proc->outfd < FD_SETSIZE);
+	  eassert (proc->outfd < EMACS_MAX_FD);
 	  if (proc->outfd >= 0)
 	    fd_callback_info[proc->outfd].thread = NULL;
 	}
@@ -1413,10 +1520,10 @@ DEFUN ("set-process-thread", Fset_process_thread, Sset_process_thread,
 
   proc = XPROCESS (process);
   pset_thread (proc, thread);
-  eassert (proc->infd < FD_SETSIZE);
+  eassert (proc->infd < EMACS_MAX_FD);
   if (proc->infd >= 0)
     fd_callback_info[proc->infd].thread = tstate;
-  eassert (proc->outfd < FD_SETSIZE);
+  eassert (proc->outfd < EMACS_MAX_FD);
   if (proc->outfd >= 0)
     fd_callback_info[proc->outfd].thread = tstate;
 
@@ -2144,7 +2251,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
 	}
     }
 
-  if (FD_SETSIZE <= inchannel || FD_SETSIZE <= outchannel)
+  if (EMACS_MAX_FD <= inchannel || EMACS_MAX_FD <= outchannel)
     report_file_errno ("Creating pipe", Qnil, EMFILE);
 
 #ifndef WINDOWSNT
@@ -2156,7 +2263,7 @@ create_process (Lisp_Object process, char **new_argv, Lisp_Object current_dir)
   fcntl (outchannel, F_SETFL, O_NONBLOCK);
 
   /* Record this as an active process, with its channels.  */
-  eassert (0 <= inchannel && inchannel < FD_SETSIZE);
+  eassert (0 <= inchannel && inchannel < EMACS_MAX_FD);
   chan_process[inchannel] = process;
   p->infd = inchannel;
   p->outfd = outchannel;
@@ -2251,7 +2358,7 @@ create_pty (Lisp_Object process)
   if (pty_fd >= 0)
     {
       p->open_fd[SUBPROCESS_STDIN] = pty_fd;
-      if (FD_SETSIZE <= pty_fd)
+      if (EMACS_MAX_FD <= pty_fd)
 	report_file_errno ("Opening pty", Qnil, EMFILE);
 #if ! defined (USG) || defined (USG_SUBTTY_WORKS)
       /* On most USG systems it does not work to open the pty's tty here,
@@ -2274,7 +2381,7 @@ create_pty (Lisp_Object process)
 
       /* Record this as an active process, with its channels.
 	 As a result, child_setup will close Emacs's side of the pipes.  */
-      eassert (0 <= pty_fd && pty_fd < FD_SETSIZE);
+      eassert (0 <= pty_fd && pty_fd < EMACS_MAX_FD);
       chan_process[pty_fd] = process;
       p->infd = pty_fd;
       p->outfd = pty_fd;
@@ -2360,7 +2467,7 @@ DEFUN ("make-pipe-process", Fmake_pipe_process, Smake_pipe_process,
   outchannel = p->open_fd[WRITE_TO_SUBPROCESS];
   inchannel = p->open_fd[READ_FROM_SUBPROCESS];
 
-  if (FD_SETSIZE <= inchannel || FD_SETSIZE <= outchannel)
+  if (EMACS_MAX_FD <= inchannel || EMACS_MAX_FD <= outchannel)
     report_file_errno ("Creating pipe", Qnil, EMFILE);
 
   fcntl (inchannel, F_SETFL, O_NONBLOCK);
@@ -2371,7 +2478,7 @@ DEFUN ("make-pipe-process", Fmake_pipe_process, Smake_pipe_process,
 #endif
 
   /* Record this as an active process, with its channels.  */
-  eassert (0 <= inchannel && inchannel < FD_SETSIZE);
+  eassert (0 <= inchannel && inchannel < EMACS_MAX_FD);
   chan_process[inchannel] = proc;
   p->infd = inchannel;
   p->outfd = outchannel;
@@ -2702,7 +2809,7 @@ DEFUN ("process-datagram-address", Fprocess_datagram_address, Sprocess_datagram_
     return Qnil;
 
   channel = XPROCESS (process)->infd;
-  eassert (0 <= channel && channel < FD_SETSIZE);
+  eassert (0 <= channel && channel < EMACS_MAX_FD);
   return conv_sockaddr_to_lisp (datagram_address[channel].sa,
 				datagram_address[channel].len);
 }
@@ -2731,7 +2838,7 @@ DEFUN ("set-process-datagram-address", Fset_process_datagram_address, Sset_proce
   channel = XPROCESS (process)->infd;
 
   len = get_lisp_to_sockaddr_size (address, &family);
-  eassert (0 <= channel && channel < FD_SETSIZE);
+  eassert (0 <= channel && channel < EMACS_MAX_FD);
   if (len == 0 || datagram_address[channel].len != len)
     return Qnil;
   conv_lisp_to_sockaddr (family, address, datagram_address[channel].sa, len);
@@ -3105,13 +3212,13 @@ DEFUN ("make-serial-process", Fmake_serial_process, Smake_serial_process,
 
   fd = serial_open (port);
   p->open_fd[SUBPROCESS_STDIN] = fd;
-  if (FD_SETSIZE <= fd)
+  if (EMACS_MAX_FD <= fd)
     report_file_errno ("Opening serial port", port, EMFILE);
   p->infd = fd;
   p->outfd = fd;
   if (fd > max_desc)
     max_desc = fd;
-  eassert (0 <= fd && fd < FD_SETSIZE);
+  eassert (0 <= fd && fd < EMACS_MAX_FD);
   chan_process[fd] = proc;
 
   buffer = Fplist_get (contact, QCbuffer);
@@ -3283,7 +3390,7 @@ finish_after_tls_connection (Lisp_Object proc)
 		    Fplist_get (contact, QChost),
 		    Fplist_get (contact, QCservice));
 
-  eassert (p->outfd < FD_SETSIZE);
+  eassert (p->outfd < EMACS_MAX_FD);
   if (NILP (result))
     {
       pset_status (p, list2 (Qfailed,
@@ -3329,7 +3436,7 @@ connect_network_socket (Lisp_Object proc, Lisp_Object addrinfos,
   if (!NILP (use_external_socket_p))
     {
       socket_to_use = external_sock_fd;
-      eassert (socket_to_use < FD_SETSIZE);
+      eassert (socket_to_use < EMACS_MAX_FD);
 
       /* Ensure we don't consume the external socket twice.  */
       external_sock_fd = -1;
@@ -3372,7 +3479,7 @@ connect_network_socket (Lisp_Object proc, Lisp_Object addrinfos,
 	      continue;
 	    }
 	  /* Reject file descriptors that would be too large.  */
-	  if (FD_SETSIZE <= s)
+	  if (EMACS_MAX_FD <= s)
 	    {
 	      emacs_close (s);
 	      s = -1;
@@ -3510,7 +3617,7 @@ connect_network_socket (Lisp_Object proc, Lisp_Object addrinfos,
 	      if (errno == EINTR)
 		goto retry_select;
 	      else
-		report_file_error ("Failed select", Qnil);
+		report_file_error ("Failed select/poll", Qnil);
 	    }
 	  eassert (sc > 0);
 
@@ -3543,7 +3650,7 @@ connect_network_socket (Lisp_Object proc, Lisp_Object addrinfos,
 #ifdef DATAGRAM_SOCKETS
       if (p->socktype == SOCK_DGRAM)
 	{
-	  eassert (0 <= s && s < FD_SETSIZE);
+	  eassert (0 <= s && s < EMACS_MAX_FD);
 	  if (datagram_address[s].sa)
 	    emacs_abort ();
 
@@ -3608,7 +3715,7 @@ connect_network_socket (Lisp_Object proc, Lisp_Object addrinfos,
   inch = s;
   outch = s;
 
-  eassert (0 <= inch && inch < FD_SETSIZE);
+  eassert (0 <= inch && inch < EMACS_MAX_FD);
   chan_process[inch] = proc;
 
   fcntl (inch, F_SETFL, O_NONBLOCK);
@@ -3635,7 +3742,7 @@ connect_network_socket (Lisp_Object proc, Lisp_Object addrinfos,
       if (! (connecting_status (p->status)
 	     && EQ (XCDR (p->status), addrinfos)))
 	pset_status (p, Fcons (Qconnect, addrinfos));
-      eassert (0 <= inch && inch < FD_SETSIZE);
+      eassert (0 <= inch && inch < EMACS_MAX_FD);
       if ((fd_callback_info[inch].flags & NON_BLOCKING_CONNECT_FD) == 0)
 	add_non_blocking_write_fd (inch);
     }
@@ -4703,7 +4810,7 @@ deactivate_process (Lisp_Object proc)
     close_process_fd (&p->open_fd[i]);
 
   inchannel = p->infd;
-  eassert (inchannel < FD_SETSIZE);
+  eassert (inchannel < EMACS_MAX_FD);
   if (inchannel >= 0)
     {
       p->infd  = -1;
@@ -4839,7 +4946,7 @@ server_accept_connection (Lisp_Object server, int channel)
 
   s = accept4 (channel, &saddr.sa, &len, SOCK_CLOEXEC);
 
-  if (FD_SETSIZE <= s)
+  if (EMACS_MAX_FD <= s)
     {
       emacs_close (s);
       s = -1;
@@ -4943,7 +5050,7 @@ server_accept_connection (Lisp_Object server, int channel)
   Lisp_Object name = Fformat (nargs, args);
   Lisp_Object proc = make_process (name);
 
-  eassert (0 <= s && s < FD_SETSIZE);
+  eassert (0 <= s && s < EMACS_MAX_FD);
   chan_process[s] = proc;
 
   fcntl (s, F_SETFL, O_NONBLOCK);
@@ -5226,7 +5333,7 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
       if (! NILP (wait_for_cell) && ! NILP (XCAR (wait_for_cell)))
 	break;
 
-      eassert (max_desc < FD_SETSIZE);
+      eassert (max_desc < EMACS_MAX_FD);
 
 #if defined HAVE_GETADDRINFO_A || defined HAVE_GNUTLS
       {
@@ -5359,7 +5466,7 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 	     because otherwise we wouldn't run into a timeout
 	     below.  */
 	  int fd = child_signal_read_fd;
-	  eassert (fd < FD_SETSIZE);
+	  eassert (fd < EMACS_MAX_FD);
 	  if (0 <= fd)
 	    FD_CLR (fd, &Atemp);
 
@@ -5453,7 +5560,7 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 	 an asynchronous process.  Otherwise this might deadlock if we
 	 receive a SIGCHLD during `pselect'.  */
       int child_fd = child_signal_read_fd;
-      eassert (child_fd < FD_SETSIZE);
+      eassert (child_fd < EMACS_MAX_FD);
       if (0 <= child_fd)
         FD_SET (child_fd, &Available);
 
@@ -5563,7 +5670,7 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 	     And if so, we need to skip the select which could block. */
 	  FD_ZERO (&tls_available);
 	  tls_nfds = 0;
-	  for (channel = 0; channel < FD_SETSIZE; ++channel)
+	  for (channel = 0; channel < EMACS_MAX_FD; ++channel)
 	    if (! NILP (chan_process[channel])
 		&& FD_ISSET (channel, &Available))
 	      {
@@ -5625,7 +5732,7 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 	      else if (nfds > 0)
 		/* Slow path, merge one by one.  Note: nfds does not need
 		   to be accurate, just positive is enough. */
-		for (channel = 0; channel < FD_SETSIZE; ++channel)
+		for (channel = 0; channel < EMACS_MAX_FD; ++channel)
 		  if (FD_ISSET(channel, &tls_available))
 		    FD_SET(channel, &Available);
 	    }
@@ -6019,7 +6126,7 @@ read_process_output (Lisp_Object proc, int channel)
 {
   ssize_t nbytes;
   struct Lisp_Process *p = XPROCESS (proc);
-  eassert (0 <= channel && channel < FD_SETSIZE);
+  eassert (0 <= channel && channel < EMACS_MAX_FD);
   struct coding_system *coding = proc_decode_coding_system[channel];
   int carryover = p->decoding_carryover;
   ptrdiff_t readmax = clip_to_bounds (1, read_process_output_max, PTRDIFF_MAX);
@@ -6184,7 +6291,7 @@ read_and_dispose_of_process_output (struct Lisp_Process *p, char *chars,
 	 proc_encode_coding_system[p->outfd] surely points to a
 	 valid memory because p->outfd will be changed once EOF is
 	 sent to the process.  */
-      eassert (p->outfd < FD_SETSIZE);
+      eassert (p->outfd < EMACS_MAX_FD);
       if (NILP (p->encode_coding_system) && p->outfd >= 0
 	  && proc_encode_coding_system[p->outfd])
 	{
@@ -6415,7 +6522,7 @@ send_process (Lisp_Object proc, const char *buf, ptrdiff_t len,
   if (p->outfd < 0)
     error ("Output file descriptor of %s is closed", SDATA (p->name));
 
-  eassert (p->outfd < FD_SETSIZE);
+  eassert (p->outfd < EMACS_MAX_FD);
   coding = proc_encode_coding_system[p->outfd];
   Vlast_coding_system_used = CODING_ID_NAME (coding->id);
 
@@ -6528,7 +6635,7 @@ send_process (Lisp_Object proc, const char *buf, ptrdiff_t len,
           if (outfd < 0)
             error ("Output file descriptor of %s is closed",
                    SDATA (p->name));
-	  eassert (0 <= outfd && outfd < FD_SETSIZE);
+	  eassert (0 <= outfd && outfd < EMACS_MAX_FD);
 #ifdef DATAGRAM_SOCKETS
 	  if (DATAGRAM_CHAN_P (outfd))
 	    {
@@ -6980,7 +7087,7 @@ DEFUN ("continue-process", Fcontinue_process, Scontinue_process, 0, 2, 0,
       struct Lisp_Process *p;
 
       p = XPROCESS (process);
-      eassert (p->infd < FD_SETSIZE);
+      eassert (p->infd < EMACS_MAX_FD);
       if (EQ (p->command, Qt)
 	  && p->infd >= 0
 	  && (!EQ (p->filter, Qt) || EQ (p->status, Qlisten)))
@@ -7124,7 +7231,7 @@ DEFUN ("process-send-eof", Fprocess_send_eof, Sprocess_send_eof, 0, 1, 0,
 
 
   outfd = XPROCESS (proc)->outfd;
-  eassert (outfd < FD_SETSIZE);
+  eassert (outfd < EMACS_MAX_FD);
   if (outfd >= 0)
     coding = proc_encode_coding_system[outfd];
 
@@ -7172,13 +7279,13 @@ DEFUN ("process-send-eof", Fprocess_send_eof, Sprocess_send_eof, 0, 1, 0,
       p->open_fd[WRITE_TO_SUBPROCESS] = new_outfd;
       p->outfd = new_outfd;
 
-      eassert (0 <= new_outfd && new_outfd < FD_SETSIZE);
+      eassert (0 <= new_outfd && new_outfd < EMACS_MAX_FD);
       if (!proc_encode_coding_system[new_outfd])
 	proc_encode_coding_system[new_outfd]
 	  = xmalloc (sizeof (struct coding_system));
       if (old_outfd >= 0)
 	{
-	  eassert (old_outfd < FD_SETSIZE);
+	  eassert (old_outfd < EMACS_MAX_FD);
 	  *proc_encode_coding_system[new_outfd]
 	    = *proc_encode_coding_system[old_outfd];
 	  memset (proc_encode_coding_system[old_outfd], 0,
@@ -7251,7 +7358,7 @@ child_signal_init (void)
   int fds[2];
   if (emacs_pipe (fds) < 0)
     report_file_error ("Creating pipe for child signal", Qnil);
-  if (FD_SETSIZE <= fds[0])
+  if (EMACS_MAX_FD <= fds[0])
     {
       /* Since we need to `pselect' on the read end, it has to fit
 	 into an `fd_set'.  */
@@ -7723,7 +7830,7 @@ DEFUN ("process-filter-multibyte-p", Fprocess_filter_multibyte_p,
   struct Lisp_Process *p = XPROCESS (process);
   if (p->infd < 0)
     return Qnil;
-  eassert (p->infd < FD_SETSIZE);
+  eassert (p->infd < EMACS_MAX_FD);
   struct coding_system *coding = proc_decode_coding_system[p->infd];
   return (CODING_FOR_UNIBYTE (coding) ? Qnil : Qt);
 }
@@ -7757,7 +7864,7 @@ keyboard_bit_set (fd_set *mask)
 {
   int fd;
 
-  eassert (max_desc < FD_SETSIZE);
+  eassert (max_desc < EMACS_MAX_FD);
   for (fd = 0; fd <= max_desc; fd++)
     if (FD_ISSET (fd, mask)
 	&& ((fd_callback_info[fd].flags & (FOR_READ | KEYBOARD_FD))
@@ -8005,7 +8112,7 @@ wait_reading_process_output (intmax_t time_limit, int nsecs, int read_kbd,
 void
 add_timer_wait_descriptor (int fd)
 {
-  eassert (0 <= fd && fd < FD_SETSIZE);
+  eassert (0 <= fd && fd < EMACS_MAX_FD);
   add_read_fd (fd, timerfd_callback, NULL);
   fd_callback_info[fd].flags &= ~KEYBOARD_FD;
 }
@@ -8031,7 +8138,7 @@ remove_slash_colon (Lisp_Object name)
 add_keyboard_wait_descriptor (int desc)
 {
 #ifdef subprocesses /* Actually means "not MSDOS".  */
-  eassert (desc >= 0 && desc < FD_SETSIZE);
+  eassert (desc >= 0 && desc < EMACS_MAX_FD);
   fd_callback_info[desc].flags &= ~PROCESS_FD;
   fd_callback_info[desc].flags |= (FOR_READ | KEYBOARD_FD);
   if (desc > max_desc)
@@ -8045,7 +8152,7 @@ add_keyboard_wait_descriptor (int desc)
 delete_keyboard_wait_descriptor (int desc)
 {
 #ifdef subprocesses
-  eassert (desc >= 0 && desc < FD_SETSIZE);
+  eassert (desc >= 0 && desc < EMACS_MAX_FD);
 
   fd_callback_info[desc].flags &= ~(FOR_READ | KEYBOARD_FD | PROCESS_FD);
 
@@ -8068,7 +8175,7 @@ setup_process_coding_systems (Lisp_Object process)
   if (inch < 0 || outch < 0)
     return;
 
-  eassert (0 <= inch && inch < FD_SETSIZE);
+  eassert (0 <= inch && inch < EMACS_MAX_FD);
   if (!proc_decode_coding_system[inch])
     proc_decode_coding_system[inch] = xmalloc (sizeof (struct coding_system));
   coding_system = p->decode_coding_system;
@@ -8080,7 +8187,7 @@ setup_process_coding_systems (Lisp_Object process)
     }
   setup_coding_system (coding_system, proc_decode_coding_system[inch]);
 
-  eassert (0 <= outch && outch < FD_SETSIZE);
+  eassert (0 <= outch && outch < EMACS_MAX_FD);
   if (!proc_encode_coding_system[outch])
     proc_encode_coding_system[outch] = xmalloc (sizeof (struct coding_system));
   setup_coding_system (p->encode_coding_system,
@@ -8322,7 +8429,7 @@ catch_child_signal (void)
 restore_nofile_limit (void)
 {
 #ifdef HAVE_SETRLIMIT
-  if (FD_SETSIZE < nofile_limit.rlim_cur)
+  if (EMACS_MAX_FD < nofile_limit.rlim_cur)
     setrlimit (RLIMIT_NOFILE, &nofile_limit);
 #endif
 }
@@ -8383,13 +8490,13 @@ init_process_emacs (int sockfd)
     }
 
 #ifdef HAVE_SETRLIMIT
-  /* Don't allocate more than FD_SETSIZE file descriptors for Emacs itself.  */
+  /* Don't allocate more than EMACS_MAX_FD file descriptors for Emacs itself.  */
   if (getrlimit (RLIMIT_NOFILE, &nofile_limit) != 0)
     nofile_limit.rlim_cur = 0;
-  else if (FD_SETSIZE < nofile_limit.rlim_cur)
+  else if (EMACS_MAX_FD < nofile_limit.rlim_cur)
     {
       struct rlimit rlim = nofile_limit;
-      rlim.rlim_cur = FD_SETSIZE;
+      rlim.rlim_cur = EMACS_MAX_FD;
       if (setrlimit (RLIMIT_NOFILE, &rlim) != 0)
 	nofile_limit.rlim_cur = 0;
     }
@@ -8425,7 +8532,7 @@ init_process_emacs (int sockfd)
 
   Vprocess_alist = Qnil;
   deleted_pid_list = Qnil;
-  for (i = 0; i < FD_SETSIZE; i++)
+  for (i = 0; i < EMACS_MAX_FD; i++)
     {
       chan_process[i] = Qnil;
       proc_buffered_char[i] = -1;
diff --git a/src/sysdep.c b/src/sysdep.c
index 95295e7e67..5988b35927 100644
--- a/src/sysdep.c
+++ b/src/sysdep.c
@@ -777,7 +777,7 @@ restore_signal_handlers (struct save_signal *saved_handlers)
 }
 \f
 #ifdef USABLE_SIGIO
-static int old_fcntl_flags[FD_SETSIZE];
+static int old_fcntl_flags[EMACS_MAX_FD];
 #endif
 
 void
@@ -1079,7 +1079,7 @@ emacs_set_tty (int fd, struct emacs_tty *settings, bool flushp)
 \f
 
 #ifdef F_SETOWN
-static int old_fcntl_owner[FD_SETSIZE];
+static int old_fcntl_owner[EMACS_MAX_FD];
 #endif /* F_SETOWN */
 
 /* Initialize the terminal mode on all tty devices that are currently
diff --git a/src/syspoll.h b/src/syspoll.h
new file mode 100644
index 0000000000..22f8a47b93
--- /dev/null
+++ b/src/syspoll.h
@@ -0,0 +1,34 @@
+/* syspoll.h - System-dependent definitions for the poll function.
+   Copyright (C) 2022 Free Software Foundation, Inc.
+
+This file is part of GNU Emacs.
+
+GNU Emacs is free software: you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation, either version 3 of the License, or (at
+your option) any later version.
+
+GNU Emacs is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with GNU Emacs.  If not, see <https://www.gnu.org/licenses/>.  */
+
+#ifndef SYSPOLL_H
+#define SYSPOLL_H 1
+
+#if !defined (DOS_NT) && !defined (WINDOWSNT)
+#include <sys/poll.h>
+
+extern int fd_sets_to_pollfds (fd_set *, fd_set *, int);
+extern void pollfds_to_fd_sets (fd_set *, fd_set *, int);
+extern int timespec_to_timeout (const struct timespec *);
+extern int emacs_pselect (int, fd_set *, fd_set *,
+			  fd_set *, const struct timespec *,
+			  const sigset_t *);
+extern struct pollfd pollfds[];
+#define pselect emacs_pselect
+#endif
+#endif
diff --git a/src/sysselect.h b/src/sysselect.h
index 45cc22bc4c..5c1175a4ff 100644
--- a/src/sysselect.h
+++ b/src/sysselect.h
@@ -29,6 +29,7 @@ #define SYSSELECT_H 1
    where w32 needs it, but not where sysselect.h is included.  The w32
    definitions in w32.h are incompatible with the below.  */
 #ifndef WINDOWSNT
+#ifndef USE_POLL
 #ifdef FD_SET
 #ifndef FD_SETSIZE
 #define FD_SETSIZE 64
@@ -43,6 +44,35 @@ #define FD_CLR(n, p) (*(p) &= ~(1 << (n)))
 #define FD_ISSET(n, p) (*(p) & (1 << (n)))
 #define FD_ZERO(p) (*(p) = 0)
 #endif /* no FD_SET */
+#define EMACS_MAX_FD FD_SETSIZE
+#else /* no USE_POLL */
+#define EMACS_MAX_FD (10 * FD_SETSIZE)
+#define fd_set emacs_fd_set
+#undef FD_CLR
+#undef FD_ISSET
+#undef FD_SET
+#undef FD_ZERO
+
+typedef struct {
+  EMACS_UINT bits[EMACS_MAX_FD / EMACS_UINT_WIDTH];
+} emacs_fd_set;
+
+/* standard access macros */
+#define FD_SET(n, p) \
+  do { \
+    if ((n) < EMACS_MAX_FD) { \
+      (p)->bits[(n)/EMACS_UINT_WIDTH] |= (1 << (n)%EMACS_UINT_WIDTH); \
+    } \
+  } while (0)
+#define FD_CLR(n, p) \
+  do { \
+    if ((n) < EMACS_MAX_FD) { \
+      (p)->bits[(n)/EMACS_UINT_WIDTH] &= ~(1 << (n)%EMACS_UINT_WIDTH); \
+    } \
+  } while (0)
+#define FD_ISSET(n, p) ((n) < EMACS_MAX_FD ? ((p)->bits[(n)/EMACS_UINT_WIDTH] & (1 << (n)%EMACS_UINT_WIDTH)) : 0)
+#define FD_ZERO(p) memset((p), 0, sizeof(emacs_fd_set))
+#endif /* no USE_POLL */
 #endif /* not WINDOWSNT */
 
 #if !defined (HAVE_SELECT)
@@ -66,21 +96,21 @@ #define pselect sys_select
 INLINE void
 fd_CLR (int fd, fd_set *set)
 {
-  eassume (0 <= fd && fd < FD_SETSIZE);
+  eassume (0 <= fd && fd < EMACS_MAX_FD);
   FD_CLR (fd, set);
 }
 
 INLINE bool
 fd_ISSET (int fd, fd_set *set)
 {
-  eassume (0 <= fd && fd < FD_SETSIZE);
+  eassume (0 <= fd && fd < EMACS_MAX_FD);
   return FD_ISSET (fd, set) != 0;
 }
 
 INLINE void
 fd_SET (int fd, fd_set *set)
 {
-  eassume (0 <= fd && fd < FD_SETSIZE);
+  eassume (0 <= fd && fd < EMACS_MAX_FD);
   FD_SET (fd, set);
 }
 
diff --git a/src/thread.c b/src/thread.c
index 626d14aad0..b5c90e20a3 100644
--- a/src/thread.c
+++ b/src/thread.c
@@ -27,6 +27,9 @@ Copyright (C) 2012-2022 Free Software Foundation, Inc.
 #include "syssignal.h"
 #include "pdumper.h"
 #include "keyboard.h"
+#ifdef USE_POLL
+#include "syspoll.h"
+#endif
 
 #ifdef HAVE_NS
 #include "nsterm.h"
diff --git a/src/xgselect.c b/src/xgselect.c
index 7252210c68..cc64ae04bc 100644
--- a/src/xgselect.c
+++ b/src/xgselect.c
@@ -21,6 +21,10 @@ Copyright (C) 2009-2022 Free Software Foundation, Inc.
 
 #include "xgselect.h"
 
+#ifdef USE_POLL
+#include "syspoll.h"
+#endif
+
 #ifdef HAVE_GLIB
 
 #include <glib.h>
diff --git a/src/xmenu.c b/src/xmenu.c
index 418628d491..31bc3eaeb9 100644
--- a/src/xmenu.c
+++ b/src/xmenu.c
@@ -45,6 +45,9 @@ Copyright (C) 1986, 1988, 1993-1994, 1996, 1999-2022 Free Software
 #include "buffer.h"
 #include "coding.h"
 #include "sysselect.h"
+#ifdef USE_POLL
+#include "syspoll.h"
+#endif
 #include "pdumper.h"
 
 #ifdef MSDOS
diff --git a/src/xterm.c b/src/xterm.c
index 01832d60c9..3576267626 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -637,6 +637,9 @@ Copyright (C) 1989, 1993-2022 Free Software Foundation, Inc.
 #include "font.h"
 #include "xsettings.h"
 #include "sysselect.h"
+#ifdef USE_POLL
+#include "syspoll.h"
+#endif
 #include "menu.h"
 #include "pdumper.h"
 
-- 
2.35.1.607.gf01e51a7cf

Re: 1024 file descriptors should be enough for anyone

[Po Lu]

Robert Pluim <rpluim@gmail.com> writes:

> Attached an initial attempt at using poll() instead of select(), and
> bumping the open filed descriptor limit to 10240. It passes 'make
> check' on GNU/Linux, FreeBSD 13, and macOS, and M-x eww
> works. MS-Windows should still build with no change in functionality.

Great, thanks!  Though would prefer `syspoll.h' to be part of
`sysselect.h' instead, since it just defines a wrapper that pretends to
be `pselect'.  It also means we can get rid of the following
incantation:

> +#ifdef USE_POLL
> +#include "syspoll.h"
> +#endif

Some more comments below:

> +  Does Emacs use poll()?                                  ${USE_POLL}

> +** Emacs can now use poll() and larger file descriptor sets.
> +There is experimental support for using poll() instead of select() to
> +check file descriptor statuses, which can be requested by using the
> +'--with-poll' option to the 'configure' script.  As part of this
> +feature, the maximum number of open files supported has been increased
> +to 10xFD_SETSIZE (which is typically 1024).  Note that there may be
> +ulimit or kernel limits that still restrict the number of simultaneous
> +open files.  This feature is not supported on MS-Windows.

In here and various other places, "poll()" should be written `poll',
since it is not a function call without arguments.

> diff --git a/src/xterm.c b/src/xterm.c
> index 01832d60c9..3576267626 100644
> --- a/src/xterm.c
> +++ b/src/xterm.c
> @@ -637,6 +637,9 @@ Copyright (C) 1989, 1993-2022 Free Software Foundation, Inc.
>  #include "font.h"
>  #include "xsettings.h"
>  #include "sysselect.h"
> +#ifdef USE_POLL
> +#include "syspoll.h"
> +#endif
>  #include "menu.h"
>  #include "pdumper.h"

init_xterm must reject displays whose fds are larger than FD_MAXSIZE on
Lucid and Motif builds, since Xt uses pselect internally.  (I don't
remember if XtOpenDisplay does that for you anymore.)

I would prefer that x_wait_for_event, XTflash and
x_next_event_from_any_display be rewritten to utilize poll directly
instead of going through the wrapper as well.

Once again, thanks for making this happen!

Surprisingly high use of file descriptors

[Stefan Monnier]

Robert Pluim [2022-05-04 17:22:27] wrote:
> But for those people who insist on using LSP with large codebases,
> Emacs's use of 'select' is somewhat limiting.

[ Thanks for working on this, I'll let other comment on the patch, I'm
  just curious here:  ]

Why would ``lsp-mode` or `eglot` consume anywhere near 1024 file descriptors?
Are they launching a separate LSP server process per buffer?


        Stefan

Re: 1024 file descriptors should be enough for anyone

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

> init_xterm

I meant x_term_init, sorry.

Re: 1024 file descriptors should be enough for anyone

[Robert Pluim]

>>>>> On Thu, 05 May 2022 08:55:10 +0800, Po Lu <luangruo@yahoo.com> said:

    Po> Robert Pluim <rpluim@gmail.com> writes:
    >> Attached an initial attempt at using poll() instead of select(), and
    >> bumping the open filed descriptor limit to 10240. It passes 'make
    >> check' on GNU/Linux, FreeBSD 13, and macOS, and M-x eww
    >> works. MS-Windows should still build with no change in functionality.

    Po> Great, thanks!  Though would prefer `syspoll.h' to be part of
    Po> `sysselect.h' instead, since it just defines a wrapper that pretends to
    Po> be `pselect'.  It also means we can get rid of the following
    Po> incantation:

    >> +#ifdef USE_POLL
    >> +#include "syspoll.h"
    >> +#endif

I guess thatʼs possible, although the pattern for other similar
includes is sys<name of feature/syscall>/h.

    Po> Some more comments below:

    >> +  Does Emacs use poll()?                                  ${USE_POLL}

    >> +** Emacs can now use poll() and larger file descriptor sets.
    >> +There is experimental support for using poll() instead of select() to
    >> +check file descriptor statuses, which can be requested by using the
    >> +'--with-poll' option to the 'configure' script.  As part of this
    >> +feature, the maximum number of open files supported has been increased
    >> +to 10xFD_SETSIZE (which is typically 1024).  Note that there may be
    >> +ulimit or kernel limits that still restrict the number of simultaneous
    >> +open files.  This feature is not supported on MS-Windows.

    Po> In here and various other places, "poll()" should be written `poll',
    Po> since it is not a function call without arguments.

Fixed

    >> diff --git a/src/xterm.c b/src/xterm.c
    >> index 01832d60c9..3576267626 100644
    >> --- a/src/xterm.c
    >> +++ b/src/xterm.c
    >> @@ -637,6 +637,9 @@ Copyright (C) 1989, 1993-2022 Free Software Foundation, Inc.
    >> #include "font.h"
    >> #include "xsettings.h"
    >> #include "sysselect.h"
    >> +#ifdef USE_POLL
    >> +#include "syspoll.h"
    >> +#endif
    >> #include "menu.h"
    >> #include "pdumper.h"

    Po> init_xterm must reject displays whose fds are larger than FD_MAXSIZE on
    Po> Lucid and Motif builds, since Xt uses pselect internally.  (I don't
    Po> remember if XtOpenDisplay does that for you anymore.)

OK. Iʼll look at that. Of course, who opens a second display connection
when they have 1024 file descriptors open? 😈

    Po> I would prefer that x_wait_for_event, XTflash and
    Po> x_next_event_from_any_display be rewritten to utilize poll directly
    Po> instead of going through the wrapper as well.

Iʼm not a great fan of flag days, especially in something as
fundamental as file descriptor handling.

    Po> Once again, thanks for making this happen!

What can I say? It all just seemed like fun 😉

Robert
-- 

Re: Surprisingly high use of file descriptors

[Robert Pluim]

>>>>> On Wed, 04 May 2022 22:11:59 -0400, Stefan Monnier <monnier@iro.umontreal.ca> said:

    Stefan> Robert Pluim [2022-05-04 17:22:27] wrote:
    >> But for those people who insist on using LSP with large codebases,
    >> Emacs's use of 'select' is somewhat limiting.

    Stefan> [ Thanks for working on this, I'll let other comment on the patch, I'm
    Stefan>   just curious here:  ]

    Stefan> Why would ``lsp-mode` or `eglot` consume anywhere near 1024 file descriptors?
    Stefan> Are they launching a separate LSP server process per buffer?

They use filenotify.el to watch for the changes to all the files in
their project, which on some platforms consumes a file descriptor per
file (at least macOS, and anything kqueue based like FreeBSD). On
other platforms it uses one per directory, and Iʼm sure people have
codebases with >1024 directories.

Robert
-- 

Re: Surprisingly high use of file descriptors

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: emacs-devel@gnu.org
> Date: Thu, 05 May 2022 09:13:27 +0200
> 
>     Stefan> Why would ``lsp-mode` or `eglot` consume anywhere near 1024 file descriptors?
>     Stefan> Are they launching a separate LSP server process per buffer?
> 
> They use filenotify.el to watch for the changes to all the files in
> their project, which on some platforms consumes a file descriptor per
> file (at least macOS, and anything kqueue based like FreeBSD). On
> other platforms it uses one per directory, and Iʼm sure people have
> codebases with >1024 directories.

FTR, on MS-Windows the implementation uses a file handle per watch
(one watch for each directory), but that handle is not monitored by
the emulated pselect.  It also creates one thread per each watch.  The
maximum number of file handles per process on MS-Windows is AFAIK very
large (in the millions), and the maximum number of threads per process
is on the order of tens of thousands.

Re: Surprisingly high use of file descriptors

[Robert Pluim]

>>>>> On Thu, 05 May 2022 10:47:26 +0300, Eli Zaretskii <eliz@gnu.org> said:

    Eli> FTR, on MS-Windows the implementation uses a file handle per watch
    Eli> (one watch for each directory), but that handle is not monitored by
    Eli> the emulated pselect.  It also creates one thread per each watch.  The
    Eli> maximum number of file handles per process on MS-Windows is AFAIK very
    Eli> large (in the millions), and the maximum number of threads per process
    Eli> is on the order of tens of thousands.

That sounds like something Someone™ could use to write a proper 'poll'
emulation ☺️ (I think gnulib has one, but I haven't looked at it).

Robert
-- 

Re: Surprisingly high use of file descriptors

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Thu, 05 May 2022 09:51:03 +0200
> 
> >>>>> On Thu, 05 May 2022 10:47:26 +0300, Eli Zaretskii <eliz@gnu.org> said:
> 
>     Eli> FTR, on MS-Windows the implementation uses a file handle per watch
>     Eli> (one watch for each directory), but that handle is not monitored by
>     Eli> the emulated pselect.  It also creates one thread per each watch.  The
>     Eli> maximum number of file handles per process on MS-Windows is AFAIK very
>     Eli> large (in the millions), and the maximum number of threads per process
>     Eli> is on the order of tens of thousands.
> 
> That sounds like something Someone™ could use to write a proper 'poll'
> emulation ☺️

What for? for file notifications we don't use any poll-like interface
anyway.

> (I think gnulib has one, but I haven't looked at it).

Isn't is still limited to 64 handles?

Re: Surprisingly high use of file descriptors

[Robert Pluim]

>>>>> On Thu, 05 May 2022 11:20:49 +0300, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
    >> Date: Thu, 05 May 2022 09:51:03 +0200
    >> 
    >> >>>>> On Thu, 05 May 2022 10:47:26 +0300, Eli Zaretskii <eliz@gnu.org> said:
    >> 
    Eli> FTR, on MS-Windows the implementation uses a file handle per watch
    Eli> (one watch for each directory), but that handle is not monitored by
    Eli> the emulated pselect.  It also creates one thread per each watch.  The
    Eli> maximum number of file handles per process on MS-Windows is AFAIK very
    Eli> large (in the millions), and the maximum number of threads per process
    Eli> is on the order of tens of thousands.
    >> 
    >> That sounds like something Someone™ could use to write a proper 'poll'
    >> emulation ☺️

    Eli> What for? for file notifications we don't use any poll-like interface
    Eli> anyway.

I meant for subprocess support. Although if it behaved like 'select' I
wouldnʼt mind either.

    >> (I think gnulib has one, but I haven't looked at it).

    Eli> Isn't is still limited to 64 handles?

<peeks> It uses MsgWaitForMultipleObjects, so I think so.

Robert
-- 

Re: 1024 file descriptors should be enough for anyone

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

> Attached an initial attempt at using poll() instead of select(), and
> bumping the open filed descriptor limit to 10240. 

That's wonderful.

> Comments welcome (I can stick it on a branch if needed). Things that
> could be done:
>
> - make the upper limit either statically or dynamically configurable
>   (on GNU/Linux in theory you could have 1e6 open descriptors)

I think that would be nice.

> - a feature flag
> - dynamic switching between poll and select

I guess those could be handy when debugging, but I don't really envision
any users would want to twiddle this stuff (famous last words).

Anyway, I think sticking this on a feature branch so that a few more
people can give it a go (before merging) would be good.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: 1024 file descriptors should be enough for anyone

[Robert Pluim]

>>>>> On Thu, 05 May 2022 13:35:51 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:
    >> Comments welcome (I can stick it on a branch if needed). Things that
    >> could be done:
    >> 
    >> - make the upper limit either statically or dynamically configurable
    >> (on GNU/Linux in theory you could have 1e6 open descriptors)

    Lars> I think that would be nice.

OK. I think it increases the coupling between sysdep.c and process.c,
but nothing too bad.

    >> - a feature flag
    >> - dynamic switching between poll and select

    Lars> I guess those could be handy when debugging, but I don't really envision
    Lars> any users would want to twiddle this stuff (famous last words).

Itʼs in system-configuration-features, but perhaps packages will want
to change their behaviour based on a feature flag.

    Lars> Anyway, I think sticking this on a feature branch so that a few more
    Lars> people can give it a go (before merging) would be good.

Done (and yes, the commit message is incomplete).

Robert
-- 

Re: 1024 file descriptors should be enough for anyone

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

>     Lars> Anyway, I think sticking this on a feature branch so that a few more
>     Lars> people can give it a go (before merging) would be good.
>
> Done (and yes, the commit message is incomplete).

I've now tried the branch on Debian/bookworm with --with-poll:

  Does Emacs use 'poll'?                                  yes

And everything seems to work just fine for me.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: 1024 file descriptors should be enough for anyone

[Robert Pluim]

>>>>> On Thu, 05 May 2022 15:12:47 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

    Lars> Robert Pluim <rpluim@gmail.com> writes:
    Lars> Anyway, I think sticking this on a feature branch so that a few more
    Lars> people can give it a go (before merging) would be good.
    >> 
    >> Done (and yes, the commit message is incomplete).

    Lars> I've now tried the branch on Debian/bookworm with --with-poll:

    Lars>   Does Emacs use 'poll'?                                  yes

    Lars> And everything seems to work just fine for me.  :-)

Of course it does: thatʼs exactly where I do most of my testing :-)

Robert
-- 

Re: 1024 file descriptors should be enough for anyone

[Robert Pluim]

>>>>> On Thu, 05 May 2022 08:55:10 +0800, Po Lu <luangruo@yahoo.com> said:

    Po> Great, thanks!  Though would prefer `syspoll.h' to be part of
    Po> `sysselect.h' instead, since it just defines a wrapper that pretends to
    Po> be `pselect'.  It also means we can get rid of the following
    Po> incantation:

    >>> +#ifdef USE_POLL
    >>> +#include "syspoll.h"
    >>> +#endif

I kind of did this: syspoll.h still exists, but itʼs now included via
sysselect.h

    Po> init_xterm must reject displays whose fds are larger than FD_MAXSIZE on
    Po> Lucid and Motif builds, since Xt uses pselect internally.  (I don't
    Po> remember if XtOpenDisplay does that for you anymore.)

xcb will check for exceeding FD_SETSIZE, but only when using 'select',
and Xt will use 'poll' if itʼs available, so I donʼt think itʼs an
issue.

Robert
-- 

Re: 1024 file descriptors should be enough for anyone

[Po Lu]

Robert Pluim <rpluim@gmail.com> writes:

>     Po> init_xterm must reject displays whose fds are larger than FD_MAXSIZE on
>     Po> Lucid and Motif builds, since Xt uses pselect internally.  (I don't
>     Po> remember if XtOpenDisplay does that for you anymore.)

> xcb will check for exceeding FD_SETSIZE, but only when using 'select',
> and Xt will use 'poll' if itʼs available, so I donʼt think itʼs an
> issue.

Xlib may or may not be built with xcb, that's hardly a given.  (We only
know that's true when USE_XCB is defined.)  Likewise for Xt using
`poll', which is optional, and may or may not be how the user's
installation is built.

Re: 1024 file descriptors should be enough for anyone

[Robert Pluim]

>>>>> On Fri, 06 May 2022 16:54:08 +0800, Po Lu <luangruo@yahoo.com> said:

    Po> Robert Pluim <rpluim@gmail.com> writes:
    Po> init_xterm must reject displays whose fds are larger than FD_MAXSIZE on
    Po> Lucid and Motif builds, since Xt uses pselect internally.  (I don't
    Po> remember if XtOpenDisplay does that for you anymore.)

    >> xcb will check for exceeding FD_SETSIZE, but only when using 'select',
    >> and Xt will use 'poll' if itʼs available, so I donʼt think itʼs an
    >> issue.

    Po> Xlib may or may not be built with xcb, that's hardly a given.  (We only
    Po> know that's true when USE_XCB is defined.)  Likewise for Xt using
    Po> `poll', which is optional, and may or may not be how the user's
    Po> installation is built.

Itʼs not really optional: the configure script for Xt will use 'poll'
if itʼs found. But I guess I can add a check in the Lucid and Motif
builds.

Robert
-- 

Re: 1024 file descriptors should be enough for anyone

[Po Lu]

Robert Pluim <rpluim@gmail.com> writes:

> Itʼs not really optional: the configure script for Xt will use 'poll'
> if itʼs found. But I guess I can add a check in the Lucid and Motif
> builds.

If the Imake.cf used to build the X11 libraries doesn't define
"HavePoll", it'll use (p)select.  That is common (even on systems with
poll), so we cannot rely on Xt using poll all of the time.

Re: 1024 file descriptors should be enough for anyone

[Robert Pluim]

>>>>> On Fri, 06 May 2022 17:28:41 +0800, Po Lu <luangruo@yahoo.com> said:

    Po> Robert Pluim <rpluim@gmail.com> writes:
    >> Itʼs not really optional: the configure script for Xt will use 'poll'
    >> if itʼs found. But I guess I can add a check in the Lucid and Motif
    >> builds.

    Po> If the Imake.cf used to build the X11 libraries doesn't define
    Po> "HavePoll", it'll use (p)select.  That is common (even on systems with
    Po> poll), so we cannot rely on Xt using poll all of the time.

TIL that Imake.cf is still used, and now Iʼm having flashbacks ☺️

Robert
-- 

Re: 1024 file descriptors should be enough for anyone

[Po Lu]

Robert Pluim <rpluim@gmail.com> writes:

> TIL that Imake.cf is still used, and now Iʼm having flashbacks ☺️

Actually, you were right.  It seems that Xt migrated to Autoconf as
well.

I thought only the X server switched to autoconf.

Re: 1024 file descriptors should be enough for anyone

[Robert Pluim]

>>>>> On Fri, 06 May 2022 18:41:52 +0800, Po Lu <luangruo@yahoo.com> said:

    Po> Robert Pluim <rpluim@gmail.com> writes:
    >> TIL that Imake.cf is still used, and now Iʼm having flashbacks ☺️

    Po> Actually, you were right.  It seems that Xt migrated to Autoconf as
    Po> well.

    Po> I thought only the X server switched to autoconf.

Is this the kind of change you were thinking of? Or do I need to call
'XtCloseDisplay' as well?

diff --git a/src/xterm.c b/src/xterm.c
index 285b1d625e..28bf81c0bd 100644
--- a/src/xterm.c
+++ b/src/xterm.c
@@ -23232,13 +23232,18 @@ #define NUM_ARGV 10
 			 resource_name, EMACS_CLASS,
 			 emacs_options, XtNumber (emacs_options),
 			 &argc, argv);
-    request_sigio ();
-    turn_on_atimers (true);
+    if (ConnectionNumber (dpy) >= FD_SETSIZE)
+      dpy = 0;
+    else
+      {
+	request_sigio ();
+	turn_on_atimers (true);
 
 #ifdef HAVE_X11XTR6
     /* I think this is to compensate for XtSetLanguageProc.  */
-    fixup_locale ();
+	fixup_locale ();
 #endif
+      }
   }
 
 #else /* not USE_X_TOOLKIT */

Robert
-- 

Re: 1024 file descriptors should be enough for anyone

[Po Lu]

Robert Pluim <rpluim@gmail.com> writes:

> Is this the kind of change you were thinking of? Or do I need to call
> 'XtCloseDisplay' as well?

You have to call `XtCloseDisplay', yes.  Otherwise, that LGTM.

> diff --git a/src/xterm.c b/src/xterm.c
> index 285b1d625e..28bf81c0bd 100644
> --- a/src/xterm.c
> +++ b/src/xterm.c
> @@ -23232,13 +23232,18 @@ #define NUM_ARGV 10
>  			 resource_name, EMACS_CLASS,
>  			 emacs_options, XtNumber (emacs_options),
>  			 &argc, argv);
> -    request_sigio ();
> -    turn_on_atimers (true);
> +    if (ConnectionNumber (dpy) >= FD_SETSIZE)
> +      dpy = 0;
> +    else
> +      {
> +	request_sigio ();
> +	turn_on_atimers (true);
>  
>  #ifdef HAVE_X11XTR6
>      /* I think this is to compensate for XtSetLanguageProc.  */
> -    fixup_locale ();
> +	fixup_locale ();
>  #endif
> +      }
>    }
>  
>  #else /* not USE_X_TOOLKIT */
>
> Robert

Re: 1024 file descriptors should be enough for anyone

[Po Lu]

Po Lu <luangruo@yahoo.com> writes:

>> +    if (ConnectionNumber (dpy) >= FD_SETSIZE)
>> +      dpy = 0;

Speaking of which, DPY could be NULL here, so this should be:

  if (dpy && ConnectionNumber (dpy) >= FD_SETSIZE)
    dpy = 0;

instead.

Re: 1024 file descriptors should be enough for anyone

[Robert Pluim]

>>>>> On Fri, 06 May 2022 21:20:25 +0800, Po Lu <luangruo@yahoo.com> said:

    Po> Po Lu <luangruo@yahoo.com> writes:
    >>> +    if (ConnectionNumber (dpy) >= FD_SETSIZE)
    >>> +      dpy = 0;

    Po> Speaking of which, DPY could be NULL here, so this should be:

    Po>   if (dpy && ConnectionNumber (dpy) >= FD_SETSIZE)
    Po>     dpy = 0;

Thanks, Iʼll stick that somewhere on the branch.

Robert
-- 

Re: 1024 file descriptors should be enough for anyone

[Lars Ingebrigtsen]

Robert Pluim <rpluim@gmail.com> writes:

>     Lars> I've now tried the branch on Debian/bookworm with --with-poll:
>
>     Lars>   Does Emacs use 'poll'?                                  yes
>
>     Lars> And everything seems to work just fine for me.  :-)
>
> Of course it does: thatʼs exactly where I do most of my testing :-)

It'd be nice if people on other platforms could test this, too, before
we merge it.

The branch is "feature/more-fds", and "./configure --with-poll", and
then see whether networking stuff still works.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: 1024 file descriptors should be enough for anyone

[Robert Pluim]

>>>>> On Wed, 25 May 2022 01:50:43 +0200, Lars Ingebrigtsen <larsi@gnus.org> said:

    Lars> Robert Pluim <rpluim@gmail.com> writes:
    Lars> I've now tried the branch on Debian/bookworm with --with-poll:
    >> 
    Lars> Does Emacs use 'poll'?                                  yes
    >> 
    Lars> And everything seems to work just fine for me.  :-)
    >> 
    >> Of course it does: thatʼs exactly where I do most of my testing :-)

    Lars> It'd be nice if people on other platforms could test this, too, before
    Lars> we merge it.

I have a few more commits to push before we can merge it. Hopefully
this long weekend.

    Lars> The branch is "feature/more-fds", and "./configure --with-poll", and
    Lars> then see whether networking stuff still works.

Networking and anything involving sub-processes, since they share a
lot of code. File notifications can rely on file descriptors as well,
depending on the platform.

Robert
--