From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on dcvr.yhbt.net X-Spam-Level: X-Spam-ASN: X-Spam-Status: No, score=-4.0 required=3.0 tests=ALL_TRUSTED,BAYES_00 shortcircuit=no autolearn=ham autolearn_force=no version=3.4.2 Received: from localhost (dcvr.yhbt.net [127.0.0.1]) by dcvr.yhbt.net (Postfix) with ESMTP id B6FDD1F461 for ; Mon, 24 Jun 2019 02:52:58 +0000 (UTC) From: Eric Wong To: meta@public-inbox.org Subject: [PATCH 00/57] ds: shrink, TLS support, buffer writes to FS Date: Mon, 24 Jun 2019 02:52:01 +0000 Message-Id: <20190624025258.25592-1-e@80x24.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit List-Id: I finally took the step of making changes to DS after wanting to do something along these lines to Danga::Socket for the past decade or so And down the rabitt-hole I went. Write buffering now goes to the filesystem (which is quite fast on Linux and FreeBSD), so memory usage with giant messages is slightly reduced compared to before. It could be better if we replace Email::(Simple|MIME) with something which doesn't require slurping (but that's a big task). Fields for read (for NNTP) and all write buffers are lazily allocated, now, so there's some memory savings with 10K clients Further memory savings were achieved by passing $self to DS->write(sub {...}), eliminiating the need for most anonymous subs. NNTPS and NNTP+STARTTLS are now supported via public-inbox-nntpd using the --key and --cert parameters (HTTPS coming). I'm very happy with how I was able to reuse the write-buffering code for TLS negotiation and not have to add additional fields or code in hot paths. I'm pretty happy with this, so far; but there's still plenty left to be done. I'm not too impressed with the per-client memory cost of IO::Socket::SSL, even with SSL_MODE_RELEASE_BUFFERS, and will need to do further analysis to see what memory reductions are possible. Eric Wong (57): ds: get rid of {closed} field ds: get rid of more unused debug instance methods ds: use and export monotonic now() AddTimer: avoid clock_gettime for the '0' case ds: get rid of on_incomplete_write wrapper ds: lazy initialize wbuf_off ds: split out from ->flush_write and ->write ds: lazy-initialize wbuf ds: don't pass `events' arg to EPOLL_CTL_DEL ds: remove support for DS->write(undef) http: favor DS->write(strref) when reasonable ds: share send(..., MSG_MORE) logic ds: switch write buffering to use a tempfile ds: get rid of redundant and unnecessary POLL* constants syscall: get rid of unused EPOLL* constants syscall: get rid of unnecessary uname local vars ds: set event flags directly at initialization ds: import IO::KQueue namespace ds: share watch_chg between watch_read/watch_write ds: remove IO::Poll support (for now) ds: get rid of event_watch field httpd/async: remove EINTR check spawn: remove `Blocking' flag handling qspawn: describe where `$rpipe' come from http|nntp: favor "$! == EFOO" over $!{EFOO} checks ds: favor `delete' over assigning fields to `undef' http: don't pass extra args to PublicInbox::DS::close ds: pass $self to code references evcleanup: replace _run_asap with `event_step' callback ds: remove pointless exit calls http|nntp: be explicit about bytes::length on rbuf ds: hoist out do_read from NNTP and HTTP nntp: simplify re-arming/requeue logic allow use of PerlIO layers for filesystem writes ds: deal better with FS-related errors IO buffers nntp: wait for writability before sending greeting nntp: NNTPS and NNTP+STARTTLS working certs/create-certs.perl: fix cert validity on 32-bit daemon: map inherited sockets to well-known schemes ds|nntp: use CORE::close on socket nntp: call SSL_shutdown in normal cases t/nntpd-tls: slow client connection test daemon: use SSL_MODE_RELEASE_BUFFERS ds: allow ->write callbacks to syswrite directly nntp: reduce allocations for greeting ds: always use EV_ADD with EV_SET nntp: simplify long response logic and fix nesting ds: flush_write runs ->write callbacks even if closed nntp: lazily allocate and stash rbuf ci: require IO::KQueue on FreeBSD, for now nntp: send greeting immediately for plain sockets daemon: set TCP_DEFER_ACCEPT on everything but NNTP daemon: use FreeBSD accept filters on non-NNTP ds: split out IO::KQueue-specific code ds: reimplement IO::Poll support to look like epoll Revert "ci: require IO::KQueue on FreeBSD, for now" ds: reduce overhead of tempfile creation MANIFEST | 7 + certs/.gitignore | 4 + certs/create-certs.perl | 132 +++++++ lib/PublicInbox/DS.pm | 635 ++++++++++++------------------ lib/PublicInbox/DSKQXS.pm | 73 ++++ lib/PublicInbox/DSPoll.pm | 58 +++ lib/PublicInbox/Daemon.pm | 152 ++++++- lib/PublicInbox/EvCleanup.pm | 20 +- lib/PublicInbox/GitHTTPBackend.pm | 18 +- lib/PublicInbox/HTTP.pm | 154 +++----- lib/PublicInbox/HTTPD/Async.pm | 44 ++- lib/PublicInbox/Listener.pm | 4 +- lib/PublicInbox/NNTP.pm | 243 +++++------- lib/PublicInbox/NNTPD.pm | 2 + lib/PublicInbox/ParentPipe.pm | 3 +- lib/PublicInbox/Qspawn.pm | 11 +- lib/PublicInbox/Spawn.pm | 2 - lib/PublicInbox/Syscall.pm | 27 +- lib/PublicInbox/TLS.pm | 24 ++ script/public-inbox-nntpd | 3 +- t/ds-poll.t | 58 +++ t/httpd-corner.t | 38 +- t/httpd.t | 18 + t/nntpd-tls.t | 224 +++++++++++ t/nntpd.t | 2 + t/spawn.t | 11 - 26 files changed, 1251 insertions(+), 716 deletions(-) create mode 100644 certs/.gitignore create mode 100755 certs/create-certs.perl create mode 100644 lib/PublicInbox/DSKQXS.pm create mode 100644 lib/PublicInbox/DSPoll.pm create mode 100644 lib/PublicInbox/TLS.pm create mode 100644 t/ds-poll.t create mode 100644 t/nntpd-tls.t -- EW