From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Andrew Whatson Newsgroups: gmane.lisp.guile.bugs Subject: bug#59321: ice-9's open-input-pipe is unexpectedly slow on some systems Date: Thu, 8 Dec 2022 22:02:04 +1000 Message-ID: References: <8d55cf7d1e5382c874cfcaee1f4cddd3@posteo.de> <87r0xx5yja.fsf@gnu.org> <871qppaigl.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="000000000000dcd55e05ef4fce9c" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="26650"; mail-complaints-to="usenet@ciao.gmane.io" Cc: hylophile@posteo.de, 59321@debbugs.gnu.org To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-X-From: bug-guile-bounces+guile-bugs=m.gmane-mx.org@gnu.org Thu Dec 08 13:03:44 2022 Return-path: Envelope-to: guile-bugs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1p3Fcu-0006kC-Ki for guile-bugs@m.gmane-mx.org; Thu, 08 Dec 2022 13:03:44 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1p3FcV-0003GU-Si; Thu, 08 Dec 2022 07:03:19 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1p3FcE-0003Er-D2 for bug-guile@gnu.org; Thu, 08 Dec 2022 07:03:09 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1p3FcE-00011g-4o for bug-guile@gnu.org; Thu, 08 Dec 2022 07:03:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1p3FcD-0008Lw-W6 for bug-guile@gnu.org; Thu, 08 Dec 2022 07:03:02 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Andrew Whatson Original-Sender: "Debbugs-submit" Resent-CC: bug-guile@gnu.org Resent-Date: Thu, 08 Dec 2022 12:03:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 59321 X-GNU-PR-Package: guile Original-Received: via spool by 59321-submit@debbugs.gnu.org id=B59321.167050094432098 (code B ref 59321); Thu, 08 Dec 2022 12:03:01 +0000 Original-Received: (at 59321) by debbugs.gnu.org; 8 Dec 2022 12:02:24 +0000 Original-Received: from localhost ([127.0.0.1]:56375 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1p3Fbc-0008Le-0j for submit@debbugs.gnu.org; Thu, 08 Dec 2022 07:02:24 -0500 Original-Received: from mail-yb1-f170.google.com ([209.85.219.170]:43563) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1p3FbZ-0008LW-BL for 59321@debbugs.gnu.org; Thu, 08 Dec 2022 07:02:22 -0500 Original-Received: by mail-yb1-f170.google.com with SMTP id d128so1343083ybf.10 for <59321@debbugs.gnu.org>; Thu, 08 Dec 2022 04:02:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=QTBvVFb9XJDAEYE/LNp4ZbMEgNV0h1utU7vA9UQViTU=; b=VEQDF7302cRKMU+DrpcKXEAbck+xYtb8tkDxQFErv/qX+8QegRDCduIVo418F2323P h/BGkLk6ytmay8QTXCLtK6GQRO4DbQSBecQ0gDykw0hXrtwFaMunC6Ubp7Qq9R7VVARH rNDkCUcAeoh29ccU3gFAFiKZtHAujYznf75CmSX7F7gScah6YZikd863wF7WovDQU/ru MwM0rdIyA9AD1gmnJ8xvzTxpdiKWBhGgpTUKIZUGO3nr8fPI+F7CD63j5CUMPXDNWih4 tOboo9ZOdcmARhXEekzoag1cshvwtGbexQWDGGTEVMBaPgOhIoQEYDlW86nQae/ENuj2 mcLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=QTBvVFb9XJDAEYE/LNp4ZbMEgNV0h1utU7vA9UQViTU=; b=SpA9cN2OJ2fBxnH89dVR+ZWPa2z3CL4orfo9ZRw9xP0liHiPGgF8OJlFYk0OZAWrG8 8J6jFKtGFQjsdaNxoJeU6yANCHUPa2qoT46RI1f7Cvs2Tn6cA8p7rvr3tMrNm8yTs2h+ TjJStCwO5D3x3sIBk6aRg+Vm8No/QB5pM8PmJ5K/O9GoERAKbOCQAxQRm/kTS2+cW5/W HizUUM9Iz73oseRRNmwkJrd9b914Q+Nu5ICdvtk0mmQuVf6IvkFYBaBSxMYYtniwVkYm GJI3iBzK5fH4bCmn0JWJRh4qUoaaHIS8HMVaaGB83ABwi+WnS1YKhIYa9adchaz94hRb YQ7w== X-Gm-Message-State: ANoB5pk/OHPZLPF3g+zdDIQFPYiYszusKwTIXRaXXJiI3JwnER0WCedZ oVhRTUt4dA0xeR+AAYrDlG5vtGw5PRQU4rOmplQ= X-Google-Smtp-Source: AA0mqf7ej2mCLK/YJ8o5VHE18xK1f+loofrn7cmpX8jGDl/7mC5RZA3+l4c91Ws+/iPuBR3NZGblXrRwbrm1ALL29dk= X-Received: by 2002:a25:880e:0:b0:6e6:e31e:3dc5 with SMTP id c14-20020a25880e000000b006e6e31e3dc5mr82802095ybl.534.1670500935546; Thu, 08 Dec 2022 04:02:15 -0800 (PST) In-Reply-To: <871qppaigl.fsf@gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guile@gnu.org List-Id: "Bug reports for GUILE, GNU's Ubiquitous Extension Language" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guile-bounces+guile-bugs=m.gmane-mx.org@gnu.org Original-Sender: bug-guile-bounces+guile-bugs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.lisp.guile.bugs:10459 Archived-At: --000000000000dcd55e05ef4fce9c Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s wrote: > > Andrew Whatson skribis: > > > We also need equivalent functionality around SOCK_CLOEXEC. It seems > > this is implemented for =E2=80=98accept=E2=80=99, but not =E2=80=98sock= et=E2=80=99 or =E2=80=98socketpair=E2=80=99. > > It=E2=80=99s possible to use SOCK_CLOEXEC with =E2=80=98socket=E2=80=99 a= nd =E2=80=98socketpair=E2=80=99 > already, as in: > > (socket AF_INET (logior SOCK_CLOEXEC SOCK_STREAM) 0) > > With commit 1d313bf5f0d296d766bd3a0e6d030df37c71711b, =E2=80=98pipe=E2=80= =99 is also > covered. > > So I think we have pretty much everything we need, at least starting > with 3.0.9. Ah, nice! In that case it might be possible to deprecate this auto-closing behaviour in a future version. > > Python's PEP 433 contains a good explanation of the issues which can > > arise from leaked file descriptors: > > https://peps.python.org/pep-0433/#inherited-file-descriptors-issues > > > > Given the risks, I'm convinced that Guile's conservative approach is > > actually quite sensible. It seems like the best path forward would be > > to implement platform-specific optimizations where possible. > > > > I've attached a draft patch which implements a fast-path on systems > > where "/proc/self/fd" is available. > > The patch LGTM; it=E2=80=99s certainly an improvement on systems configur= ed with > a high per-process FD limit. > > Now, I believe use of =E2=80=98posix_spawn=E2=80=99 as proposed in > makes that unnecessary. Let=E2=80=99= s take > a closer look at that other patch and so we can see=E2=80=A6 Playing with the wip-posix-spawn branch, it has the same slowdown (actually a bit worse). I've updated the "/proc/self/fd" fast-path patch for posix_spawn, please find attached. > Thanks, > Ludo=E2=80=99. Thank you! --000000000000dcd55e05ef4fce9c Content-Type: text/x-patch; charset="US-ASCII"; name="close-inherited-with-proc-self-fd-psawn.patch" Content-Disposition: attachment; filename="close-inherited-with-proc-self-fd-psawn.patch" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_lbf12l8l0 Y29tbWl0IGMwMTJkN2IwZDUyNDhhOTlhM2E5Mjc4MDY4N2E2NzZjNWQ0MjBmNWYKQXV0aG9yOiBB bmRyZXcgV2hhdHNvbiA8d2hhdHNvbkBnbWFpbC5jb20+CkRhdGU6ICAgVGh1IERlYyA4IDIxOjQz OjI4IDIwMjIgKzEwMDAKCiAgICBSZWR1Y2UgcmVkdW5kYW50IGNsb3NlKCkgY2FsbHMgd2hlbiBm b3JraW5nIG9uIHNvbWUgc3lzdGVtcy4KICAgIAogICAgU29tZSBzeXN0ZW1zIHByb3ZpZGUgIi9w cm9jL3NlbGYvZmQiIHdoaWNoIGlzIGEgZGlyZWN0b3J5IGNvbnRhaW5pbmcgYW4KICAgIGVudHJ5 IGZvciBlYWNoIG9wZW4gZmlsZSBkZXNjcmlwdG9yIGluIHRoZSBjdXJyZW50IHByb2Nlc3MuICBX ZSB1c2UgdGhpcwogICAgdG8gbGltaXQgdGhlIG51bWJlciBvZiBjbG9zZSgpIGNhbGxzIG5lZWRl ZCB0byBlbnN1cmUgZmlsZSBkZXNjcmlwdG9ycwogICAgYXJlbid0IGxlYWtlZCB0byB0aGUgY2hp bGQgcHJvY2VzcyB3aGVuIGZvcmtpbmcuCiAgICAKICAgICogbGliZ3VpbGUvcG9zaXguYyAoY2xv c2VfaW5oZXJpdGVkX2Zkc19zbG93KToKICAgIChjbG9zZV9pbmhlcml0ZWRfZmRzKTogTmV3IHN0 YXRpYyBoZWxwZXIgZnVuY3Rpb25zLgogICAgKHNjbV9zcGF3bl9wcm9jZXNzKTogQXR0ZW1wdCB0 byBjbG9zZSBpbmhlcml0ZWQgZmlsZSBkZXNjcmlwdG9ycwogICAgZWZmaWNpZW50bHkgdXNpbmcg J2Nsb3NlX2luaGVyaXRlZF9mZHMnLCBmYWxsaW5nIGJhY2sgdG8gdGhlIGJydXRlLWZvcmNlCiAg ICBhcHByb2FjaCBpbiAnY2xvc2VfaW5oZXJpdGVkX2Zkc19zbG93Jy4KCmRpZmYgLS1naXQgYS9s aWJndWlsZS9wb3NpeC5jIGIvbGliZ3VpbGUvcG9zaXguYwppbmRleCA4N2IzMjlkYTkuLjhjOTAy MjExNiAxMDA2NDQKLS0tIGEvbGliZ3VpbGUvcG9zaXguYworKysgYi9saWJndWlsZS9wb3NpeC5j CkBAIC0yNCw2ICsyNCw3IEBACiAjICBpbmNsdWRlIDxjb25maWcuaD4KICNlbmRpZgogCisjaW5j bHVkZSA8ZGlyZW50Lmg+CiAjaW5jbHVkZSA8ZXJybm8uaD4KICNpbmNsdWRlIDxmY250bC5oPgog I2luY2x1ZGUgPHN0ZGlvLmg+CkBAIC0xMzA5LDYgKzEzMTAsNDEgQEAgU0NNX0RFRklORSAoc2Nt X2ZvcmssICJwcmltaXRpdmUtZm9yayIsIDAsIDAsIDAsCiAjdW5kZWYgRlVOQ19OQU1FCiAjZW5k aWYgLyogSEFWRV9GT1JLICovCiAKK3N0YXRpYyB2b2lkCitjbG9zZV9pbmhlcml0ZWRfZmRzX3Ns b3cgKHBvc2l4X3NwYXduX2ZpbGVfYWN0aW9uc190ICphY3Rpb25zLCBpbnQgbWF4X2ZkKQorewor ICB3aGlsZSAoLS1tYXhfZmQgPiAyKQorICAgIHBvc2l4X3NwYXduX2ZpbGVfYWN0aW9uc19hZGRj bG9zZSAoYWN0aW9ucywgbWF4X2ZkKTsKK30KKworc3RhdGljIHZvaWQKK2Nsb3NlX2luaGVyaXRl ZF9mZHMgKHBvc2l4X3NwYXduX2ZpbGVfYWN0aW9uc190ICphY3Rpb25zLCBpbnQgbWF4X2ZkKQor eworICBESVIgKmRpcnA7CisgIHN0cnVjdCBkaXJlbnQgKmQ7CisgIGludCBmZDsKKworICAvKiBU cnkgdG8gdXNlIHRoZSBwbGF0Zm9ybS1zcGVjaWZpYyBsaXN0IG9mIG9wZW4gZmlsZSBkZXNjcmlw dG9ycywgc28KKyAgICAgd2UgZG9uJ3QgbmVlZCB0byB1c2UgdGhlIGJydXRlIGZvcmNlIGFwcHJv YWNoLiAqLworICBkaXJwID0gb3BlbmRpciAoIi9wcm9jL3NlbGYvZmQiKTsKKworICBpZiAoZGly cCA9PSBOVUxMKQorICAgIHJldHVybiBjbG9zZV9pbmhlcml0ZWRfZmRzX3Nsb3cgKGFjdGlvbnMs IG1heF9mZCk7CisKKyAgd2hpbGUgKChkID0gcmVhZGRpciAoZGlycCkpICE9IE5VTEwpCisgICAg eworICAgICAgZmQgPSBhdG9pIChkLT5kX25hbWUpOworCisgICAgICAvKiBTa2lwICIuIiBhbmQg Ii4uIiwgZ2FyYmFnZSBlbnRyaWVzLCBzdGRpbi9zdGRvdXQvc3RkZXJyLiAqLworICAgICAgaWYg KGZkIDw9IDIpCisgICAgICAgIGNvbnRpbnVlOworCisgICAgICBwb3NpeF9zcGF3bl9maWxlX2Fj dGlvbnNfYWRkY2xvc2UgKGFjdGlvbnMsIGZkKTsKKyAgICB9CisKKyAgY2xvc2VkaXIgKGRpcnAp OworfQorCiBzdGF0aWMgU0NNCiBzY21fc3Bhd25fcHJvY2VzcyAoU0NNIHByb2csIFNDTSBhcmdz LCBTQ00gc2NtX2luLCBTQ00gc2NtX291dCwgU0NNIHNjbV9lcnIpCiAjZGVmaW5lIEZVTkNfTkFN RSAic3Bhd24qIgpAQCAtMTM2Myw4ICsxMzk5LDcgQEAgc2NtX3NwYXduX3Byb2Nlc3MgKFNDTSBw cm9nLCBTQ00gYXJncywgU0NNIHNjbV9pbiwgU0NNIHNjbV9vdXQsIFNDTSBzY21fZXJyKQogICBw b3NpeF9zcGF3bl9maWxlX2FjdGlvbnNfYWRkZHVwMiAoJmFjdGlvbnMsIGZkX3Nsb3RbMV0sIDEp OwogICBwb3NpeF9zcGF3bl9maWxlX2FjdGlvbnNfYWRkZHVwMiAoJmFjdGlvbnMsIGZkX3Nsb3Rb Ml0sIDIpOwogCi0gIHdoaWxlICgtLW1heF9mZCA+IDIpCi0gICAgcG9zaXhfc3Bhd25fZmlsZV9h Y3Rpb25zX2FkZGNsb3NlICgmYWN0aW9ucywgbWF4X2ZkKTsKKyAgY2xvc2VfaW5oZXJpdGVkX2Zk cyAoJmFjdGlvbnMsIG1heF9mZCk7CiAKICAgaWYgKHBvc2l4X3NwYXducCAoJnBpZCwgZXhlY19m aWxlLCAmYWN0aW9ucywgYXR0cnAsIGV4ZWNfYXJndiwgZW52aXJvbikgIT0gMCkKICAgICB7Cg== --000000000000dcd55e05ef4fce9c--