unofficial mirror of bug-guile@gnu.org 
 help / color / mirror / Atom feed
From: Andrew Whatson <whatson@gmail.com>
To: hylophile@posteo.de
Cc: 59321@debbugs.gnu.org
Subject: bug#59321: ice-9's open-input-pipe is unexpectedly slow on some systems
Date: Fri, 18 Nov 2022 14:49:05 +1000	[thread overview]
Message-ID: <CAPE069fJ0DRHv7b+mwwqOM22LJtWtUTMAJcDEAUCTbhSmO-abQ@mail.gmail.com> (raw)
In-Reply-To: <8d55cf7d1e5382c874cfcaee1f4cddd3@posteo.de>

Hi Nate,

Thanks for reporting this!  I'm able to reproduce on my systems, and
have identified the culprit:

 > strace -c -f guile -c '(begin (use-modules (ice-9 popen) (ice-9
rdelim)) (display (read-delimited "" (open-input-pipe "ls"))))'

 % time     seconds  usecs/call     calls    errors syscall
 ------ ----------- ----------- --------- --------- ------------------
  93.38    0.309261           1    250107    249994 close
   4.61    0.015268         135       113        21 futex
   0.52    0.001730           4       430       270 newfstatat
   0.38    0.001263           6       183           mmap
   0.29    0.000953           4       197        92 openat
   0.16    0.000542         180         3           execve
 <...snip...>

These close calls are from the start_child routine in libguile/posix.c:

 static pid_t
 start_child (const char *exec_file, char **exec_argv,
          int reading, int c2p[2], int writing, int p2c[2],
              int in, int out, int err)
 {
   int pid;
   int max_fd = 1024;

 #if defined (HAVE_GETRLIMIT) && defined (RLIMIT_NOFILE)
   {
     struct rlimit lim = { 0, 0 };
     if (getrlimit (RLIMIT_NOFILE, &lim) == 0)
       max_fd = lim.rlim_cur;
   }
 #endif

 <...snip...>
   /* Close all file descriptors in ports inherited from the parent
      except for in, out, and err.  Heavy-handed, but robust.  */
   while (max_fd--)
     if (max_fd != in && max_fd != out && max_fd != err)
       close (max_fd);
 <...snip...>

On my system, `ulimit -n` shows the maximum number of open files as
250000, explaining the 250000 calls to close shown by strace.

Testing a build of guile with the max_fd loop commented out shows the
expected performance:

 # unmodified guile
 > time guile -c '(begin (use-modules (ice-9 popen) (ice-9 rdelim))
(display (read-delimited "" (open-input-pipe "ls"))))'
 real 0m0.202s

 # guile patched to remove max_fd loop
 > time guile -c '(begin (use-modules (ice-9 popen) (ice-9 rdelim))
(display (read-delimited "" (open-input-pipe "ls"))))'
 real 0m0.056s

Forcibly closing file descriptors like this shouldn't be necessary if
the application has properly opened descriptors with the FD_CLOEXEC
flag.  It would be good to get input from some more experienced Guile
hackers on the potential consequences of this change.

Cheers,
Andrew





  reply	other threads:[~2022-11-18  4:49 UTC|newest]

Thread overview: 10+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-11-16 18:06 bug#59321: ice-9's open-input-pipe is unexpectedly slow on some systems hylophile
2022-11-18  4:49 ` Andrew Whatson [this message]
2022-11-18  5:52   ` tomas
2022-11-20 17:24   ` Ludovic Courtès
2022-11-20 19:00     ` tomas
2022-11-21  4:22     ` Andrew Whatson
2022-11-26 14:39       ` Ludovic Courtès
2022-12-08 12:02         ` Andrew Whatson
2022-12-08 14:34           ` Ludovic Courtès
2023-01-13 15:37           ` Ludovic Courtès

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

  List information: https://www.gnu.org/software/guile/

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAPE069fJ0DRHv7b+mwwqOM22LJtWtUTMAJcDEAUCTbhSmO-abQ@mail.gmail.com \
    --to=whatson@gmail.com \
    --cc=59321@debbugs.gnu.org \
    --cc=hylophile@posteo.de \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).