From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.blaine.gmane.org!not-for-mail From: Andrew Whatson Newsgroups: gmane.lisp.guile.bugs Subject: bug#59321: ice-9's open-input-pipe is unexpectedly slow on some systems Date: Mon, 21 Nov 2022 14:22:52 +1000 Message-ID: References: <8d55cf7d1e5382c874cfcaee1f4cddd3@posteo.de> <87r0xx5yja.fsf@gnu.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="000000000000558d8305edf369cc" Injection-Info: ciao.gmane.io; posting-host="blaine.gmane.org:116.202.254.214"; logging-data="30362"; mail-complaints-to="usenet@ciao.gmane.io" Cc: hylophile@posteo.de, 59321@debbugs.gnu.org To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Original-X-From: bug-guile-bounces+guile-bugs=m.gmane-mx.org@gnu.org Mon Nov 21 05:24:47 2022 Return-path: Envelope-to: guile-bugs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1owyMR-0007ly-Ft for guile-bugs@m.gmane-mx.org; Mon, 21 Nov 2022 05:24:47 +0100 Original-Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1owyLl-0006yn-3N; Sun, 20 Nov 2022 23:24:05 -0500 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1owyLi-0006yW-MB for bug-guile@gnu.org; Sun, 20 Nov 2022 23:24:02 -0500 Original-Received: from debbugs.gnu.org ([209.51.188.43]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1owyLi-0006yM-Dw for bug-guile@gnu.org; Sun, 20 Nov 2022 23:24:02 -0500 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1owyLh-0006bS-Uh for bug-guile@gnu.org; Sun, 20 Nov 2022 23:24:01 -0500 X-Loop: help-debbugs@gnu.org Resent-From: Andrew Whatson Original-Sender: "Debbugs-submit" Resent-CC: bug-guile@gnu.org Resent-Date: Mon, 21 Nov 2022 04:24:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 59321 X-GNU-PR-Package: guile Original-Received: via spool by 59321-submit@debbugs.gnu.org id=B59321.166900459425325 (code B ref 59321); Mon, 21 Nov 2022 04:24:01 +0000 Original-Received: (at 59321) by debbugs.gnu.org; 21 Nov 2022 04:23:14 +0000 Original-Received: from localhost ([127.0.0.1]:45013 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1owyKv-0006aO-Jv for submit@debbugs.gnu.org; Sun, 20 Nov 2022 23:23:14 -0500 Original-Received: from mail-yw1-f178.google.com ([209.85.128.178]:43785) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1owyKr-0006a9-8k for 59321@debbugs.gnu.org; Sun, 20 Nov 2022 23:23:11 -0500 Original-Received: by mail-yw1-f178.google.com with SMTP id 00721157ae682-39451671bdfso77289697b3.10 for <59321@debbugs.gnu.org>; Sun, 20 Nov 2022 20:23:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=zQ+NYjrXwu4DhvYqqjBfUY0BCM9MgnIc428sFv5gVPg=; b=Ie75V/5J6LHL8uTh2pzj413MY532n4LLtrohHIDhD2n0aL3sxTLnY8S2LkxQheNw3m 8JqhO1ReJmseIYbrkO5bV18ZHSWHKGbnUMRP0gw9/qI06xlI7S90LtkPutl38NGrAKOF 5B3yu9HItjVpwKxbEKWROIj3LyxtmMRbdKmSj6WgqNSykvW62qPgxMzCOMxDWPePagbq j/NLNq4y9FAAiPBIIeWs9gh+VWtBGEi1FsajTvaAZs6La2f7YcyHhElsP76IzU/BmGca AC6YmNwbJ5+v14/c63VQTfsiEHevl1s9x+hA6CJJtjUfFRWtsHeb/k/SvHnM0szlvNin qKJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=zQ+NYjrXwu4DhvYqqjBfUY0BCM9MgnIc428sFv5gVPg=; b=msgmayxMEsdtXTp98FGLIz88KxX3ktHhNDSqWS6N++xZKrcX/sy3ztGH8Ro62eAxyE 9wsrbCR08QsN5zRYTykdxn1Cq8SJ6tss72hvM8a8RNHHVVr+N9PcRnhcdg8S14WYQDmz tDJDAPY4FyU1tpkQqRLMEt49h4uqWod+NBzjzgfrD9EUXbBhdKvJ5do66fvVPTtwKyjY sDwkwsvaubX+Wr7+VfCE0QvWrzKDXvpxfsH5qTxpHcA5gkkpi1NIviRU0xkweHfQE4Hn 4b1XVkq9goAmMjEnqjkMXr8870e2wYRSIdSU11pw51jbnI2BNgB+8j6M24PE5glUKhXA bVFw== X-Gm-Message-State: ANoB5pk4uf65/kVHnfnTf60CwisK6WWbEqJ/Wh7bXcYLwup5I1Sv12Fo bTyAe7fC5D3WljkQQWImVB8+atT4u/9l5ig0GkU= X-Google-Smtp-Source: AA0mqf7fXLeWWTvTc2SjohFJsZ4YkP4tAaP1lRM6yQOvTQYU/pouiz08qsVqhQAQRKcNHKxdEjqkhxUXP64iX6o1SHw= X-Received: by 2002:a81:b04f:0:b0:357:94cb:7a9 with SMTP id x15-20020a81b04f000000b0035794cb07a9mr15114990ywk.328.1669004583522; Sun, 20 Nov 2022 20:23:03 -0800 (PST) In-Reply-To: <87r0xx5yja.fsf@gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guile@gnu.org List-Id: "Bug reports for GUILE, GNU's Ubiquitous Extension Language" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guile-bounces+guile-bugs=m.gmane-mx.org@gnu.org Original-Sender: bug-guile-bounces+guile-bugs=m.gmane-mx.org@gnu.org Xref: news.gmane.io gmane.lisp.guile.bugs:10443 Archived-At: --000000000000558d8305edf369cc Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Ludovic Court=C3=A8s wrote: > > Andrew Whatson skribis: > > > Forcibly closing file descriptors like this shouldn't be necessary if > > the application has properly opened descriptors with the FD_CLOEXEC > > flag. It would be good to get input from some more experienced Guile > > hackers on the potential consequences of this change. > > Libguile opens all its own file descriptors at O_CLOEXEC (one omission > was recently fixed in 0aa1a9976fc3c6af4d1087e59d728cb8fe7d369a) so it > may be possible to remove that FD-closing loop. There=E2=80=99s still th= e > possibility that application bug unwillingly leaks FDs, but we could > consider it=E2=80=99s none of our business. > > Thoughts? I agree with this approach in principle, but from what Tomas is saying it seems like it's not currently possible for applications to do the right thing in all cases. > Similarly, with commit a356ceebee000efe91a2a16dbcaa64d6c6a3a922, it=E2=80= =99s > possible to pass =E2=80=98open-file=E2=80=99 a flag that corresponds to O= _CLOEXEC, > which wasn=E2=80=99t possible before. Nice! > I=E2=80=99ve also been thinking that files opened with =E2=80=98call-with= -*=E2=80=99 should be > O_CLOEXEC. That=E2=80=99d be an incompatible change though, so maybe not > something for 3.0.x. This sounds reasonable to me. We also need equivalent functionality around SOCK_CLOEXEC. It seems this is implemented for =E2=80=98accept=E2=80=99, but not =E2=80=98socket= =E2=80=99 or =E2=80=98socketpair=E2=80=99. Python's PEP 433 contains a good explanation of the issues which can arise from leaked file descriptors: https://peps.python.org/pep-0433/#inherited-file-descriptors-issues Given the risks, I'm convinced that Guile's conservative approach is actually quite sensible. It seems like the best path forward would be to implement platform-specific optimizations where possible. I've attached a draft patch which implements a fast-path on systems where "/proc/self/fd" is available. --000000000000558d8305edf369cc Content-Type: text/x-patch; charset="US-ASCII"; name="close-inherited-with-proc-self-fd.patch" Content-Disposition: attachment; filename="close-inherited-with-proc-self-fd.patch" Content-Transfer-Encoding: base64 Content-ID: X-Attachment-Id: f_laqa7xar0 Y29tbWl0IDA4OTQzY2FlOTA1NDVkZGRlYTQ0Y2E1NWVhYjY4MDQ3ZTVhZTJmOWQKQXV0aG9yOiBB bmRyZXcgV2hhdHNvbiA8d2hhdHNvbkBnbWFpbC5jb20+CkRhdGU6ICAgTW9uIE5vdiAyMSAxMzo0 MDozMyAyMDIyICsxMDAwCgogICAgUmVkdWNlIHJlZHVuZGFudCBjbG9zZSgpIGNhbGxzIHdoZW4g Zm9ya2luZyBvbiBzb21lIHN5c3RlbXMuCiAgICAKICAgIFNvbWUgc3lzdGVtcyBwcm92aWRlICIv cHJvYy9zZWxmL2ZkIiB3aGljaCBpcyBhIGRpcmVjdG9yeSBjb250YWluaW5nIGFuCiAgICBlbnRy eSBmb3IgZWFjaCBvcGVuIGZpbGUgZGVzY3JpcHRvciBpbiB0aGUgY3VycmVudCBwcm9jZXNzLiAg V2UgdXNlIHRoaXMKICAgIHRvIGxpbWl0IHRoZSBudW1iZXIgb2YgY2xvc2UoKSBjYWxscyBuZWVk ZWQgdG8gZW5zdXJlIGZpbGUgZGVzY3JpcHRvcnMKICAgIGFyZW4ndCBsZWFrZWQgdG8gdGhlIGNo aWxkIHByb2Nlc3Mgd2hlbiBmb3JraW5nLgogICAgCiAgICAqIGxpYmd1aWxlL3Bvc2l4LmMgKGNs b3NlX2luaGVyaXRlZF9mZHNfc2xvdyk6CiAgICAoY2xvc2VfaW5oZXJpdGVkX2Zkcyk6IE5ldyBz dGF0aWMgaGVscGVyIGZ1bmN0aW9ucy4KICAgIChzdGFydF9jaGlsZCk6IEF0dGVtcHQgdG8gY2xv c2UgaW5oZXJpdGVkIGZpbGUgZGVzY3JpcHRvcnMgZWZmaWNpZW50bHkKICAgIHVzaW5nICdjbG9z ZV9pbmhlcml0ZWRfZmRzJywgZmFsbGluZyBiYWNrIHRvIHRoZSBicnV0ZS1mb3JjZSBhcHByb2Fj aCBpbgogICAgJ2Nsb3NlX2luaGVyaXRlZF9mZHNfc2xvdycuCgpkaWZmIC0tZ2l0IGEvbGliZ3Vp bGUvcG9zaXguYyBiL2xpYmd1aWxlL3Bvc2l4LmMKaW5kZXggYjUzNTJjMmM0Li5mYzM1MTIwNTQg MTAwNjQ0Ci0tLSBhL2xpYmd1aWxlL3Bvc2l4LmMKKysrIGIvbGliZ3VpbGUvcG9zaXguYwpAQCAt MjQsNiArMjQsNyBAQAogIyAgaW5jbHVkZSA8Y29uZmlnLmg+CiAjZW5kaWYKIAorI2luY2x1ZGUg PGRpcmVudC5oPgogI2luY2x1ZGUgPGVycm5vLmg+CiAjaW5jbHVkZSA8ZmNudGwuaD4KICNpbmNs dWRlIDxzdGRpby5oPgpAQCAtMTMzNyw2ICsxMzM4LDQ2IEBAIHJlbnVtYmVyX2ZpbGVfZGVzY3Jp cHRvciAoaW50IGZkLCBpbnQgZXJyKQogfQogI2VuZGlmIC8qIEhBVkVfRk9SSyAqLwogCitzdGF0 aWMgdm9pZAorY2xvc2VfaW5oZXJpdGVkX2Zkc19zbG93KGludCBtYXhfZmQsIGludCBpbiwgaW50 IG91dCwgaW50IGVycikKK3sKKyAgd2hpbGUgKG1heF9mZC0tKQorICAgIGlmIChtYXhfZmQgIT0g aW4gJiYgbWF4X2ZkICE9IG91dCAmJiBtYXhfZmQgIT0gZXJyKQorICAgICAgY2xvc2UgKG1heF9m ZCk7Cit9CisKK3N0YXRpYyB2b2lkCitjbG9zZV9pbmhlcml0ZWRfZmRzKGludCBtYXhfZmQsIGlu dCBpbiwgaW50IG91dCwgaW50IGVycikKK3sKKyAgRElSICpkaXJwOworICBzdHJ1Y3QgZGlyZW50 ICpkOworICBpbnQgZmQ7CisKKyAgLyogVHJ5IHRvIHVzZSB0aGUgcGxhdGZvcm0tc3BlY2lmaWMg bGlzdCBvZiBvcGVuIGZpbGUgZGVzY3JpcHRvcnMsIHNvCisgICAgIHdlIGRvbid0IG5lZWQgdG8g dXNlIHRoZSBicnV0ZSBmb3JjZSBhcHByb2FjaC4gKi8KKyAgZGlycCA9IG9wZW5kaXIgKCIvcHJv Yy9zZWxmL2ZkIik7CisKKyAgaWYgKGRpcnAgPT0gTlVMTCkKKyAgICByZXR1cm4gY2xvc2VfaW5o ZXJpdGVkX2Zkc19zbG93IChtYXhfZmQsIGluLCBvdXQsIGVycik7CisKKyAgd2hpbGUgKChkID0g cmVhZGRpciAoZGlycCkpICE9IE5VTEwpCisgICAgeworICAgICAgZmQgPSBhdG9pIChkLT5kX25h bWUpOworCisgICAgICAvKiBTa2lwICIuIiBhbmQgIi4uIiwgYW5kIGFueSBnYXJiYWdlIGVudHJp ZXMuICovCisgICAgICBpZiAoZmQgPD0gMCkKKyAgICAgICAgY29udGludWU7CisKKyAgICAgIC8q IEtlZXAgaW4vb3V0L2VyciBvcGVuLiAqLworICAgICAgaWYgKGZkID09IGluIHx8IGZkID09IG91 dCB8fCBmZCA9PSBlcnIpCisgICAgICAgIGNvbnRpbnVlOworCisgICAgICBjbG9zZSAoZmQpOwor ICAgIH0KKworICBjbG9zZWRpciAoZGlycCk7Cit9CisKICNpZmRlZiBIQVZFX0ZPUksKICNkZWZp bmUgSEFWRV9TVEFSVF9DSElMRCAxCiAvKiBTaW5jZSBHdWlsZSB1c2VzIHRocmVhZHMsIHdlIGhh dmUgdG8gYmUgdmVyeSBjYXJlZnVsIHRvIGF2b2lkIGNhbGxpbmcKQEAgLTEzNzMsOSArMTQxNCw3 IEBAIHN0YXJ0X2NoaWxkIChjb25zdCBjaGFyICpleGVjX2ZpbGUsIGNoYXIgKipleGVjX2FyZ3Ys CiAKICAgLyogQ2xvc2UgYWxsIGZpbGUgZGVzY3JpcHRvcnMgaW4gcG9ydHMgaW5oZXJpdGVkIGZy b20gdGhlIHBhcmVudAogICAgICBleGNlcHQgZm9yIGluLCBvdXQsIGFuZCBlcnIuICBIZWF2eS1o YW5kZWQsIGJ1dCByb2J1c3QuICAqLwotICB3aGlsZSAobWF4X2ZkLS0pCi0gICAgaWYgKG1heF9m ZCAhPSBpbiAmJiBtYXhfZmQgIT0gb3V0ICYmIG1heF9mZCAhPSBlcnIpCi0gICAgICBjbG9zZSAo bWF4X2ZkKTsKKyAgY2xvc2VfaW5oZXJpdGVkX2ZkcyAobWF4X2ZkLCBpbiwgb3V0LCBlcnIpOwog CiAgIC8qIElnbm9yZSBlcnJvcnMgb24gdGhlc2Ugb3BlbigpIGNhbGxzLiAgKi8KICAgaWYgKGlu ID09IC0xKQo= --000000000000558d8305edf369cc--