From: Eric Wong <e@80x24.org>
To: meta@public-inbox.org
Subject: [PATCH 09/13] ipc: simplify partial sendmsg fallback
Date: Thu, 9 Nov 2023 10:09:42 +0000 [thread overview]
Message-ID: <20231109100946.1440611-10-e@80x24.org> (raw)
In-Reply-To: <20231109100946.1440611-1-e@80x24.org>
In the rare case sendmsg(2) isn't able to send the full amount
(due to buffers >=2GB on Linux), use print + (autodie)close
to send the remainder and retry on EINTR. `substr' should be
able to avoid a large malloc via offsets and CoW on modern Perl.
---
lib/PublicInbox/IPC.pm | 13 +++----------
t/ipc.t | 7 +++++++
2 files changed, 10 insertions(+), 10 deletions(-)
diff --git a/lib/PublicInbox/IPC.pm b/lib/PublicInbox/IPC.pm
index 3292d960..a5cae6f2 100644
--- a/lib/PublicInbox/IPC.pm
+++ b/lib/PublicInbox/IPC.pm
@@ -10,7 +10,7 @@
package PublicInbox::IPC;
use v5.12;
use parent qw(Exporter);
-use autodie qw(fork pipe read socketpair sysread);
+use autodie qw(close fork pipe read socketpair sysread);
use Carp qw(croak);
use PublicInbox::DS qw(awaitpid);
use PublicInbox::Spawn;
@@ -266,15 +266,8 @@ sub stream_in_full ($$$) {
0) // croak "sendmsg: $!";
undef $r;
$n = $send_cmd->($w, $fds, $buf, 0) // croak "sendmsg: $!";
- while ($n < length($buf)) {
- my $x = syswrite($w, $buf, length($buf) - $n, $n);
- if (!defined($n)) {
- next if $!{EINTR};
- croak "syswrite: $!";
- }
- $x or croak "syswrite wrote 0 bytes";
- $n += $x;
- }
+ print $w substr($buf, $n) if $n < length($buf); # need > 2G on Linux
+ close $w; # autodies
}
sub wq_io_do { # always async
diff --git a/t/ipc.t b/t/ipc.t
index 519ef089..23ae2e7b 100644
--- a/t/ipc.t
+++ b/t/ipc.t
@@ -132,6 +132,13 @@ for my $t ('worker', 'worker again') {
$exp = sha1_hex($bigger)."\n";
is(readline($rb), $exp, "SHA WQWorker limit ($t)");
}
+ SKIP: {
+ $ENV{TEST_EXPENSIVE} or skip 'TEST_EXPENSIVE not set', 1;
+ my $bigger = $big x 75000; # over 2G to trigger partial sendmsg
+ $ipc->wq_io_do('test_sha', [ $wa, $wb ], $bigger);
+ my $exp = sha1_hex($bigger)."\n";
+ is(readline($rb), $exp, "SHA WQWorker sendmsg limit ($t)");
+ }
}
# wq_io_do works across fork (siblings can feed)
next prev parent reply other threads:[~2023-11-09 10:09 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-11-09 10:09 [PATCH 00/13] misc error handling stuff and simplifications Eric Wong
2023-11-09 10:09 ` [PATCH 01/13] lei_xsearch: put query in process title for debugging Eric Wong
2023-11-09 10:09 ` [PATCH 02/13] lei: use cached $daemon_pid when possible Eric Wong
2023-11-09 10:09 ` [PATCH 03/13] lei: reuse FDs atfork and close explicitly Eric Wong
2023-11-09 10:09 ` [PATCH 04/13] lei_up: use v5.12 Eric Wong
2023-11-09 10:09 ` [PATCH 05/13] net_nntp_socks: more comments around how it works Eric Wong
2023-11-09 10:09 ` [PATCH 06/13] lei ls-mail-source: gracefully handle network failures Eric Wong
2023-11-09 10:09 ` [PATCH 07/13] net: retry on EINTR and check for {quit} flag Eric Wong
2023-11-09 10:09 ` [PATCH 08/13] lei_mirror: note missing local manifests are non-fatal Eric Wong
2023-11-09 10:09 ` Eric Wong [this message]
2023-11-09 10:09 ` [PATCH 10/13] lei_input: always close single `eml' inputs Eric Wong
2023-11-09 10:09 ` [PATCH 11/13] xapcmd: get rid of scalar wantarray popen_rd Eric Wong
2023-11-09 10:09 ` [PATCH 12/13] lei: get rid of autoreap usage Eric Wong
2023-11-09 10:09 ` [PATCH 13/13] spawn: get rid of wantarray popen_rd/popen_wr Eric Wong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://public-inbox.org/README
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20231109100946.1440611-10-e@80x24.org \
--to=e@80x24.org \
--cc=meta@public-inbox.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).