From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED!not-for-mail From: Thomas Danckaert Newsgroups: gmane.lisp.guile.user Subject: reading from input pipe is slow? Date: Fri, 22 Jun 2018 18:40:30 +0200 (CEST) Message-ID: <20180622.184030.2037412309735944431.post@thomasdanckaert.be> NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Trace: blaine.gmane.org 1529685542 29331 195.159.176.226 (22 Jun 2018 16:39:02 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Fri, 22 Jun 2018 16:39:02 +0000 (UTC) To: guile-user@gnu.org Original-X-From: guile-user-bounces+guile-user=m.gmane.org@gnu.org Fri Jun 22 18:38:58 2018 Return-path: Envelope-to: guile-user@m.gmane.org Original-Received: from lists.gnu.org ([208.118.235.17]) by blaine.gmane.org with esmtp (Exim 4.84_2) (envelope-from ) id 1fWP5N-0007VQ-Oc for guile-user@m.gmane.org; Fri, 22 Jun 2018 18:38:57 +0200 Original-Received: from localhost ([::1]:35256 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWP7U-0007Rl-Oa for guile-user@m.gmane.org; Fri, 22 Jun 2018 12:41:08 -0400 Original-Received: from eggs.gnu.org ([2001:4830:134:3::10]:46566) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fWP79-0007Ql-25 for guile-user@gnu.org; Fri, 22 Jun 2018 12:40:48 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fWP74-0006lp-62 for guile-user@gnu.org; Fri, 22 Jun 2018 12:40:47 -0400 Original-Received: from s02.spamexperts.axc.nl ([185.182.56.112]:48053) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fWP73-0006kz-UW for guile-user@gnu.org; Fri, 22 Jun 2018 12:40:42 -0400 Original-Received: from vserver42.axc.nl ([185.182.56.92]) by s02.spamexperts.axc.nl with esmtps (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.89) (envelope-from ) id 1fWP6w-00078K-FA for guile-user@gnu.org; Fri, 22 Jun 2018 18:40:35 +0200 Original-Received: from mail.axc.nl ([185.182.56.42]:43906) by vserver42.axc.nl with esmtp (Exim 4.91) (envelope-from ) id 1fWP6v-00DfEy-Aw for guile-user@gnu.org; Fri, 22 Jun 2018 18:40:34 +0200 X-Mailer: Mew version 6.7 on Emacs 25.3 / Mule 6.0 (HANACHIRUSATO) PrimaryMX: Accepted email from trusted host. Hint: This skips spam scanning so make sure other host is not vulnerable X-Relay-Host: 185.182.56.42 SpamTally: Final spam score: 0 X-Originating-IP: 185.182.56.92 X-SpamExperts-Domain: vserver42.axc.nl X-SpamExperts-Username: 185.182.56.92 Authentication-Results: spamexperts.axc.nl; auth=pass smtp.auth=185.182.56.92@vserver42.axc.nl X-SpamExperts-Outgoing-Class: unsure X-SpamExperts-Outgoing-Evidence: Combined (0.46) X-Recommended-Action: accept X-Filter-ID: EX5BVjFpneJeBchSMxfU5iCLBqVuPluKEPghRiGQYLZ602E9L7XzfQH6nu9C/Fh9KJzpNe6xgvOx q3u0UDjvO587IEwkeqVIekXgxeoWskaUFzM/DcqXjylVQRFCEtTnfS4gJT85sdTFWWxzMNtBcQTJ sjWw5+0eZIpQIcK9yrTOdWG4cwy4eb7cCNwhEPxIE/bGKkWjX0vBrTNL7qKCVSRh5bRscFlNvvbM elN97t2xkzc3MaPwhyyXNpaCoQ64z42+kaWZdt6N+q59XgE2+hrDr0cJJpOv+LTkhDH6DXqBaQIi fdaGzMoXcgXnOXfsRAwX31WVY5lWjWxuGSRuxdTzQjfwuJwXKuGhnkXMT2vCKpehLm7iOaWCp3Uj S/lklOFLt8X6SDUJLm68sooWCjjvhxRy8c+nflOBK9anR8p0FiGstWhrTik/0w3zGs+GbpavmAnN SX/SNW9gGWBK3KoneqzY9b0HkI9Ue1WahAWkywmZtiDkyd4mEBjJGGEJgawbllbHk+xyUKopM6rc KCaQX/lIXcRWtobViGg9fpWIU697qeeLvIg/H72Ok3ykF8FcHzzV3acxBudgYcInb4oAoyWe4E6y p+nauF72PBSLAAcYM8UqElwso7Dm8OQeUxnKPf5PKm3obfMmem7XdughnDT7wacUNP1hUSvuKpzz PL1HhdB2qPJVzcaLCGVLD5YOVi5LzatSCVd0PYrfQRco/1Dlmycalm305xB8ZPA3FPjD/1YV5+2C y8JU7qWbPAlbDjazCbhs7qBpykynMj/zgiDXVYK2wRvTTX2DHOimLYe4iKFfjtTVW27ip1AODq1f JdHHPURIpZqMBlBKWcBG/JtTS9161 X-Report-Abuse-To: spam@s01.spamexperts.axc.nl X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 185.182.56.112 X-BeenThere: guile-user@gnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: General Guile related discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guile-user-bounces+guile-user=m.gmane.org@gnu.org Original-Sender: "guile-user" Xref: news.gmane.org gmane.lisp.guile.user:14648 Archived-At: Dear guile users, I've notice that reading a child process' stdout from a port opened with (open-input-pipe "command") is very slow. In fact, it's faster to redirect the child process' output to a temporary file, and then read this temporary file from Guile, than to directly read the child process' output using a port (I've added some example code below where you can see this). Is there an explanation for this? Or even better: a solution? Thank you for any information! Thomas You can execute the following at the REPL. Here, I "cat" a big file to simulate a process generating a largish amount of text. (use-modules (ice-9 textual-ports) (ice-9 popen)) (define big-file "/tmp/big-file.txt") ;; write 5MB to big-file (with-output-to-file big-file (lambda () (let put-line ((n 0)) (simple-format (current-output-port) "~a~%" (make-string 1024 #\a)) (unless (> n (* 1024 5)) (put-line (1+ n)))) (simple-format (current-output-port) "This is the end"))) (define my-command (format #f "cat ~a" big-file)) ;; 15 seconds on my machine: ,time (format #t "end of file: '~a'~%" (string-take-right (get-string-all (open-input-pipe my-command)) 15)) ;; 1 second on my machine: ,time (let ((tmpfile (tmpnam))) ;; redirect my-command into a temporary file and read the file: (system (format #f "~a > ~a" my-command tmpfile)) (format #t "end of file: '~a'~%" (string-take-right (call-with-input-file tmpfile get-string-all) 15)) (delete-file tmpfile))