From: Rob Browning <rlb@defaultvalue.org>
To: guile-devel@gnu.org
Subject: Unexpectedly low read/write performance of open-pipe
Date: Sun, 07 Apr 2019 13:28:56 -0500 [thread overview]
Message-ID: <87d0lxmzyv.fsf@trouble.defaultvalue.org> (raw)
While evaluating guile as a possibility to replace some python code,
assuming I'm not just doing something wrong, I noticed that open-pipe
appears to transfer data *much* more slowly than python when OPEN_BOTH is
specified as opposed to OPEN_READ:
discarding dev-zero as file: 12500.00 mb/s
discarding dev-zero via OPEN_READ: 4132.23 mb/s
discarding dev-zero via OPEN_WRITE: 1.42 mb/s
For something similar to the original python code and roughly similar to
open-pipe OPEN_BOTH (see code below) I see:
mb/s: 1713.26754296
In the end, what I'd need is an OPEN_BOTH (or equivalent) that could
support block-reads and ideally read-delimited operations
on the subproces output pipe at speeds much closer to python's.
Here's the trivial guile test program:
#!/usr/bin/env guile -s
!#
(use-modules
((ice-9 binary-ports) :select (get-bytevector-n! put-bytevector))
((ice-9 format) :select (format))
((ice-9 popen) :select (open-pipe open-pipe*))
((rnrs bytevectors) :select (bytevector-length make-bytevector)))
(define (cat-bytes src dest len)
;; Discard bytes if dest is #f
(let* ((buf (make-bytevector 65536))
(buf-len (bytevector-length buf)))
(let loop ((remaining len))
(unless (zero? remaining)
(let ((n-or-eof (get-bytevector-n! src buf 0 (min remaining buf-len))))
(unless (eof-object? n-or-eof)
(when dest (put-bytevector dest buf 0 n-or-eof))
(loop (- remaining n-or-eof))))))))
(define *discard* #f)
(define *dev-zero* (with-fluids ((%default-port-encoding #f))
(open-input-file "/dev/zero")))
(define (cat-zero mode)
(with-fluids ((%default-port-encoding #f))
(open-pipe* mode "cat" "/dev/zero")))
(define (time-cat-mb mb src dest)
(let ((start (tms:clock (times))))
(cat-bytes src dest (* mb 1024 1024))
(when dest (force-output dest))
(let ((end (tms:clock (times))))
(format (current-error-port)
"~,2f mb/s\n" (/ mb
(/ (- (tms:clock (times)) start)
internal-time-units-per-second))))))
(display "discarding dev-zero as file: " (current-error-port))
(time-cat-mb 10000 *dev-zero* *discard*)
(display "discarding dev-zero via OPEN_READ: " (current-error-port))
(time-cat-mb 5000 (cat-zero OPEN_READ) *discard*)
(display "discarding dev-zero via OPEN_WRITE: " (current-error-port))
(time-cat-mb 10 (cat-zero OPEN_BOTH) *discard*)
And here's the python code:
#!/usr/bin/env python
import os, subprocess
proc = subprocess.Popen(['cat', '/dev/zero'],
stdin=subprocess.PIPE, stdout=subprocess.PIPE,
close_fds = True, bufsize = 4096)
n_kb = 1000000
start = os.times()[4]
written = 0
for i in range(n_kb):
assert(len(proc.stdout.read(1024)) == 1024)
written += 1024
end = os.times()[4]
proc.terminate()
proc.wait()
print "mb/s:", (n_kb / 1024.0) / (end - start)
Thanks
--
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4
next reply other threads:[~2019-04-07 18:28 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-04-07 18:28 Rob Browning [this message]
2019-04-07 18:45 ` Unexpectedly low read/write performance of open-pipe Rob Browning
2019-04-07 19:47 ` Rob Browning
2019-04-07 21:28 ` Rob Browning
2019-04-08 10:52 ` Mark H Weaver
2019-04-09 6:56 ` Rob Browning
2019-04-09 8:35 ` Mark H Weaver
2019-04-09 9:21 ` Chris Vine
2019-04-09 18:24 ` Mark H Weaver
2019-04-09 21:36 ` Chris Vine
2019-04-10 0:07 ` Mark H Weaver
2019-04-16 21:42 ` Mark H Weaver
2019-04-09 18:33 ` Mark H Weaver
2019-04-17 4:02 ` Mark H Weaver
2019-04-21 16:22 ` Rob Browning
2019-04-22 18:39 ` Arne Babenhauserheide
2019-04-23 7:32 ` tomas
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
List information: https://www.gnu.org/software/guile/
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87d0lxmzyv.fsf@trouble.defaultvalue.org \
--to=rlb@defaultvalue.org \
--cc=guile-devel@gnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).