From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Lars Ingebrigtsen Newsgroups: gmane.emacs.bugs Subject: bug#32729: Xemacs 23 times as fast as GNU Emacs Date: Sat, 12 Oct 2019 05:57:09 +0200 Message-ID: <871rviobu2.fsf@gnus.org> References: Mime-Version: 1.0 Content-Type: text/plain Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="58609"; mail-complaints-to="usenet@blaine.gmane.org" User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) Cc: Kevin Layer , 32729@debbugs.gnu.org, 32728@debbugs.gnu.org To: "Benninghofen\, Benjamin Dr." Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Sat Oct 12 05:58:12 2019 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane.org Original-Received: from lists.gnu.org ([209.51.188.17]) by blaine.gmane.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1iJ8Xk-000F7P-Ak for geb-bug-gnu-emacs@m.gmane.org; Sat, 12 Oct 2019 05:58:12 +0200 Original-Received: from localhost ([::1]:58654 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iJ8Xj-0007RO-08 for geb-bug-gnu-emacs@m.gmane.org; Fri, 11 Oct 2019 23:58:11 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:60533) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1iJ8Xb-0007Qp-O2 for bug-gnu-emacs@gnu.org; Fri, 11 Oct 2019 23:58:05 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1iJ8Xa-00027j-Bn for bug-gnu-emacs@gnu.org; Fri, 11 Oct 2019 23:58:03 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:51733) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1iJ8Xa-00027d-9H for bug-gnu-emacs@gnu.org; Fri, 11 Oct 2019 23:58:02 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1iJ8Xa-0006mc-7x for bug-gnu-emacs@gnu.org; Fri, 11 Oct 2019 23:58:02 -0400 X-Loop: help-debbugs@gnu.org In-Reply-To: Resent-From: Lars Ingebrigtsen Original-Sender: "Debbugs-submit" Resent-CC: bug-gnu-emacs@gnu.org Resent-Date: Sat, 12 Oct 2019 03:58:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 32729 X-GNU-PR-Package: emacs Original-Received: via spool by 32729-submit@debbugs.gnu.org id=B32729.157085264026017 (code B ref 32729); Sat, 12 Oct 2019 03:58:02 +0000 Original-Received: (at 32729) by debbugs.gnu.org; 12 Oct 2019 03:57:20 +0000 Original-Received: from localhost ([127.0.0.1]:60553 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iJ8Wt-0006lT-Je for submit@debbugs.gnu.org; Fri, 11 Oct 2019 23:57:19 -0400 Original-Received: from quimby.gnus.org ([80.91.231.51]:50562) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1iJ8Wo-0006lB-1F; Fri, 11 Oct 2019 23:57:14 -0400 Original-Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=marnie) by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1iJ8Wj-0006U3-Bq; Sat, 12 Oct 2019 05:57:11 +0200 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.org gmane.emacs.bugs:169021 Archived-At: The example is a bit convoluted, but it's an interesting question -- is Emacs' handling of process output fast or not? As a baseline, let's read a GB of zeroes and output to /dev/null: larsi@marnie:~/src/emacs/trunk$ time dd if=/dev/zero bs=1000 count=1000000 > /dev/null 1000000+0 records in 1000000+0 records out 1000000000 bytes (1.0 GB, 954 MiB) copied, 2.04672 s, 489 MB/s real 0m2.064s user 0m0.173s sys 0m1.684s (benchmark-run 1 (call-process "dd" nil nil nil "if=/dev/zero" "bs=1000" "count=1000000")) => (0.665899839 0 0.0) So that's better in Emacs than in the shell, but we're just setting stdout to /dev/zero here, so we're not actually seeing the data at all. But let's insert the data somewhere: (benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil "if=/dev/zero" "bs=1000" "count=1000000")) => (4.703641145 0 0.0) (Note: Don't visit the " *zeroes*" buffer after this, because that will hang Emacs totally. I guess the long-line display problem hasn't been fixed after all?) 4.7s isn't horrible, but it's not good, either. But most of that time is taken up in coding system conversion, so: (let ((coding-system-for-read 'binary)) (benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil "if=/dev/zero" "bs=1000" "count=1000000"))) => (1.750549617 0 0.0) Which is faster than writing to a file: larsi@marnie:~/src/emacs/trunk$ time dd if=/dev/zero bs=1000 count=1000000 of=/tmp/foo 1000000+0 records in 1000000+0 records out 1000000000 bytes (1.0 GB, 954 MiB) copied, 2.21987 s, 450 MB/s real 0m2.325s user 0m0.168s sys 0m1.957s So that's OK. But what happens when we add a process filter? (let ((coding-system-for-read 'binary)) (kill-buffer (get-buffer-create " *zeroes*")) (benchmark-run 1 (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd" "if=/dev/zero" "bs=1000" "count=1000000"))) (set-process-filter proc (lambda (proc string) )) (while (and (process-live-p proc) (accept-process-output proc 0.01)))))) => (16.878995199 993 12.469541476) That's slow, and we're just discarding the data. If we output the data, it's even slower, but not a surprising amount: (let ((coding-system-for-read 'binary)) (kill-buffer (get-buffer-create " *zeroes*")) (benchmark-run 1 (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd" "if=/dev/zero" "bs=1000" "count=1000000"))) (set-process-filter proc (lambda (proc string) (with-current-buffer (get-buffer-create " *zeroes*") (goto-char (point-max)) (insert string)))) (while (and (process-live-p proc) (accept-process-output proc 0.01)))))) => (19.801399562 1000 12.700370797000001) Byte-compiling the function makes no difference. So it would seem that the Emacs filter method is just unnecessarily slow, which I've long suspected. Creating the strings before calling the filter is probably what's taking quite a bit of this time, but the rest is taken up by garbage collecting as it spends 13 of these 20 seconds doing that. And it's a real world problem: When reading data from any network source, you have to use filters because the protocol is usually based on parsing the output to find out when it's over, so you can't use sentinels. So for Emacs 28 I want to explore adding a new type of filter to processes: One that doesn't take a string argument, but which just inserts the data into the buffer, and then calls the filter with the region positions of what was inserted, which is just as powerful, but should allow streams to be 10x more efficient. -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no