From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Mark H Weaver Newsgroups: gmane.lisp.guile.devel Subject: Re: Unexpectedly low read/write performance of open-pipe Date: Tue, 09 Apr 2019 04:35:38 -0400 Message-ID: <87k1g3y3re.fsf@netris.org> References: <87d0lxmzyv.fsf@trouble.defaultvalue.org> <878swlmz6q.fsf@trouble.defaultvalue.org> <874l79mwbw.fsf@trouble.defaultvalue.org> <87zhp1ld31.fsf@trouble.defaultvalue.org> <87sgus6a65.fsf@netris.org> <87k1g3ll96.fsf@trouble.defaultvalue.org> Mime-Version: 1.0 Content-Type: text/plain Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="265672"; mail-complaints-to="usenet@blaine.gmane.org" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.1 (gnu/linux) Cc: guile-devel@gnu.org To: Rob Browning Original-X-From: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Tue Apr 09 10:37:49 2019 Return-path: Envelope-to: guile-devel@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1hDmGL-00170I-13 for guile-devel@m.gmane.org; Tue, 09 Apr 2019 10:37:49 +0200 Original-Received: from localhost ([127.0.0.1]:37436 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hDmGK-0008Bb-1Y for guile-devel@m.gmane.org; Tue, 09 Apr 2019 04:37:48 -0400 Original-Received: from eggs.gnu.org ([209.51.188.92]:45605) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1hDmFm-000819-W0 for guile-devel@gnu.org; Tue, 09 Apr 2019 04:37:20 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1hDmFl-0003c7-R4 for guile-devel@gnu.org; Tue, 09 Apr 2019 04:37:14 -0400 Original-Received: from world.peace.net ([64.112.178.59]:60252) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1hDmFl-0003aM-Kp for guile-devel@gnu.org; Tue, 09 Apr 2019 04:37:13 -0400 Original-Received: from mhw by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hDmFh-0003yT-E6; Tue, 09 Apr 2019 04:37:09 -0400 In-Reply-To: <87k1g3ll96.fsf@trouble.defaultvalue.org> (Rob Browning's message of "Tue, 09 Apr 2019 01:56:37 -0500") X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 64.112.178.59 X-BeenThere: guile-devel@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: "Developers list for Guile, the GNU extensibility library" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-devel-bounces+guile-devel=m.gmane.org@gnu.org Original-Sender: "guile-devel" Xref: news.gmane.org gmane.lisp.guile.devel:19881 Archived-At: Hi Rob, Rob Browning writes: > Mark H Weaver writes: > >> See below for a draft reimplementation of the OPEN_BOTH mode of >> open-pipe* based on R6RS custom binary input/output. On my machine it >> increases the speed of your test by a factor of ~1k. > > Hah, I was about to report that I'd tested something along similar lines > (though much more a quick hack to just replace make-rw-port and see what > happened), and that I had seen substantial improvements: > > (define (make-rw-bin-port read-port write-port) > (define (read! dest offset count) > (let ((result (get-bytevector-n! read-port dest offset count))) > (if (eof-object? result) 0 result))) > (define (write! src offset count) > (put-bytevector write-port src offset count) > count) > (define (close x) > (close-port read-port) > (close-port write-port)) > (make-custom-binary-input/output-port "open-bin-pipe-port" > read! write! #f #f > close)) Hah, we had the same idea! :-) FYI, the reason I didn't use 'get-bytevector-n!', although it leads to the much simpler code above, is that it has the wrong semantics w.r.t. blocking behavior. 'get-bytevector-n!' blocks until the entire requested count is read, returning less than the requested amount only if EOF is reached. In this case, 'read!' is free to return less than 'count' bytes, and should block _only_ as needed to ensure that at least one byte is returned (or EOF). Of course, an effort should be made to return additional bytes, and preferably a sizeable buffer, but only if it can be done without blocking unnecessarily. Guile has only one I/O primitive capable of doing this job efficiently: 'get-bytevector-some'. Internally, it works by simply returning the entire existing port read buffer if it's non-empty. If the read buffer is empty, then read(2) is used to refill the read buffer, which is then returned to the user. The reason for the extra complexity in my reimplementation is that there's no way to specify an upper bound on the size of the bytevector returned by 'get-bytevector-some', so in general we may need to preserve the remaining bytes more future 'read!' calls. >> Let me know how it works for you. > > For a first quick test of your patch using the original program I was > working on, I see about ~1.4MiB/s without the patch, and about 150MiB/s > with it, measured by pv. It's interesting that I saw a much larger improvement than you're seeing. In my case, the numbers reported by your test program went from ~0.35 mb/s to ~333 mb/s, on a Thinkpad X200. > (If the patch holds up, it'd be nice to have in 2.2, but I suppose that > might not be appropriate.) I think it's probably fine for 2.2, although a more careful check should be made for differences in behavior between the old and new implementations, and tests should be added. I'll try to get to it soon. Regards, Mark