bug#32729: Xemacs 23 times as fast as GNU Emacs

bug#32729: Xemacs 23 times as fast as GNU Emacs

[Benninghofen, Benjamin Dr.]

[-- Attachment #1: Type: text/plain, Size: 3069 bytes --]

I did send an email with the same subject earlier, but its attachment was probably lost.

Now I inserted the contents of the attachments in the email text.


The versions of Xemacs and GNU Emacs are those that come with RHEL 7.5.

To reproduce the benchmark:

Evaluate this in Emacs-Lisp


(defun demo ()
    (interactive)
    (let ((buffer (get-buffer-create "*demo*"))
          proc)
      (switch-to-buffer buffer)
      (erase-buffer)
      (setq proc (start-process "demo" buffer "~/prog.sh"))
      (set-process-filter proc 'demo-process-filter)))

(defun demo-process-filter (process output)
    (let* ((xmarker (process-mark process))
           (marker (if (marker-position xmarker)
                       xmarker
                     (set-marker (make-marker) 0 buffer)))
           (marker-point (marker-position marker)))
      (goto-char marker-point)
      (insert output)
      (set-marker marker (point))))

Place a file "prog.sh" with the contents below in the home directory:

#! /bin/bash
  start=$(date +%s)
  echo sleeping 5 to wait for process filter to get installed...
  sleep 5
  while read line; do
      echo $line
  done <<< "$(cat input.txt)"
  end=$(date +%s)
  echo duration: $(( end - start )) seconds




Furthermore a large text file "input.txt"  is needed in the home directory. The file should have 1000000 lines, each line longer than 80 characters.


The file I used was created with the following ANSI-COMMON-LISP function:

(defun print-nums (&key (first 1) (last 1000000))
  (check-type first fixnum)
  (check-type last fixnum)
  (loop for k of-type fixnum from first to last do (format t "~%~B ^3 = ~B" k (expt k 3)))
  t)

Alternatively the "input.txt" file can be created as follows:

#! /bin/bash
  last=1000000
  function base2 {
      echo "obase=2;$1" | bc
  }
  for k in $(seq 1 $last); do
      x=$(( k * k * k ))
      echo $(base2 $k) '^3 =' $(base2 $x)
  done

generate the input like this:

  $ ./gen.sh > input.txt



The benchmark is executed by
(M-x) demo

At the end the time is printed and I received the following results:

Xemacs    : 51 seconds
GNU Emacs : 1205 seconds

So the Xemacs is more than 23 times as fast as the GNU Emacs.

---
Benjamin Benninghofen


The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised.
If you are not the intended recipient, please notify Airbus immediately and delete this e-mail.
Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately.
All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

[-- Attachment #2: Type: text/html, Size: 5541 bytes --]

bug#32729: Xemacs 23 times as fast as GNU Emacs

[Lars Ingebrigtsen]

The example is a bit convoluted, but it's an interesting question -- is
Emacs' handling of process output fast or not?

As a baseline, let's read a GB of zeroes and output to /dev/null:

larsi@marnie:~/src/emacs/trunk$ time dd if=/dev/zero bs=1000 count=1000000 > /dev/null
1000000+0 records in
1000000+0 records out
1000000000 bytes (1.0 GB, 954 MiB) copied, 2.04672 s, 489 MB/s

real	0m2.064s
user	0m0.173s
sys	0m1.684s

(benchmark-run 1 (call-process "dd" nil nil nil "if=/dev/zero" "bs=1000" "count=1000000"))
=> (0.665899839 0 0.0)

So that's better in Emacs than in the shell, but we're just setting
stdout to /dev/zero here, so we're not actually seeing the data at all.

But let's insert the data somewhere:

(benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil "if=/dev/zero" "bs=1000" "count=1000000"))
=> (4.703641145 0 0.0)

(Note: Don't visit the " *zeroes*" buffer after this, because that will
hang Emacs totally.  I guess the long-line display problem hasn't been
fixed after all?)

4.7s isn't horrible, but it's not good, either.  But most of that time
is taken up in coding system conversion, so:

(let ((coding-system-for-read 'binary))
  (benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil "if=/dev/zero" "bs=1000" "count=1000000")))
=> (1.750549617 0 0.0)

Which is faster than writing to a file:

larsi@marnie:~/src/emacs/trunk$ time dd if=/dev/zero bs=1000 count=1000000 of=/tmp/foo
1000000+0 records in
1000000+0 records out
1000000000 bytes (1.0 GB, 954 MiB) copied, 2.21987 s, 450 MB/s

real	0m2.325s
user	0m0.168s
sys	0m1.957s

So that's OK.  But what happens when we add a process filter?

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=/dev/zero" "bs=1000" "count=1000000")))
      (set-process-filter proc (lambda (proc string)
				 ))
      (while (and (process-live-p proc)
		  (accept-process-output proc 0.01))))))
=> (16.878995199 993 12.469541476)

That's slow, and we're just discarding the data.  If we output the data,
it's even slower, but not a surprising amount:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=/dev/zero" "bs=1000" "count=1000000")))
      (set-process-filter proc (lambda (proc string)
				 (with-current-buffer
				     (get-buffer-create " *zeroes*")
				   (goto-char (point-max))
				   (insert string))))
      (while (and (process-live-p proc)
		  (accept-process-output proc 0.01))))))
=> (19.801399562 1000 12.700370797000001)

Byte-compiling the function makes no difference.

So it would seem that the Emacs filter method is just unnecessarily
slow, which I've long suspected.  Creating the strings before calling
the filter is probably what's taking quite a bit of this time, but the
rest is taken up by garbage collecting as it spends 13 of these 20
seconds doing that.

And it's a real world problem: When reading data from any network
source, you have to use filters because the protocol is usually based on
parsing the output to find out when it's over, so you can't use
sentinels.

So for Emacs 28 I want to explore adding a new type of filter to
processes: One that doesn't take a string argument, but which just
inserts the data into the buffer, and then calls the filter with the
region positions of what was inserted, which is just as powerful, but
should allow streams to be 10x more efficient.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#32728: bug#32729: Xemacs 23 times as fast as GNU Emacs

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sat, 12 Oct 2019 05:57:09 +0200
> Cc: Kevin Layer <layer@franz.com>, 32729@debbugs.gnu.org, 32728@debbugs.gnu.org
> 
> (benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil "if=/dev/zero" "bs=1000" "count=1000000"))
> => (4.703641145 0 0.0)
> 
> (Note: Don't visit the " *zeroes*" buffer after this, because that will
> hang Emacs totally.  I guess the long-line display problem hasn't been
> fixed after all?)

It's unrelated.  Did you try to insert that into a unibyte buffer
instead?

> So it would seem that the Emacs filter method is just unnecessarily
> slow, which I've long suspected.  Creating the strings before calling
> the filter is probably what's taking quite a bit of this time, but the
> rest is taken up by garbage collecting as it spends 13 of these 20
> seconds doing that.
> 
> And it's a real world problem: When reading data from any network
> source, you have to use filters because the protocol is usually based on
> parsing the output to find out when it's over, so you can't use
> sentinels.

Why can't you use the default filter, and in the sentinel work on the
buffer with the complete response?

Please also note that, GC aside, inserting stuff of size that is
unknown in advance is not free, either: we need to enlarge the buffer
text each time more stuff arrives.

> So for Emacs 28 I want to explore adding a new type of filter to
> processes: One that doesn't take a string argument, but which just
> inserts the data into the buffer, and then calls the filter with the
> region positions of what was inserted, which is just as powerful, but
> should allow streams to be 10x more efficient.

Doesn't the default filter already do that?

bug#32729: Xemacs 23 times as fast as GNU Emacs

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> (benchmark-run 1 (call-process "dd" nil (get-buffer-create "
>> *zeroes*") nil "if=/dev/zero" "bs=1000" "count=1000000"))
>> => (4.703641145 0 0.0)
>> 
>> (Note: Don't visit the " *zeroes*" buffer after this, because that will
>> hang Emacs totally.  I guess the long-line display problem hasn't been
>> fixed after all?)
>
> It's unrelated.  Did you try to insert that into a unibyte buffer
> instead?

Nope.  Does Emacs need to do a lot of recomputing when going to
multibyte buffers?

>> And it's a real world problem: When reading data from any network
>> source, you have to use filters because the protocol is usually based on
>> parsing the output to find out when it's over, so you can't use
>> sentinels.
>
> Why can't you use the default filter, and in the sentinel work on the
> buffer with the complete response?

No sentinel is called, because the process status doesn't change,
typically.  All the relevant network protocols do not close after having
"done something" (IMAP, HTTP, etc), but instead use in-protocol markers
to say that an operation is done.  So a filter has to be used.

> Please also note that, GC aside, inserting stuff of size that is
> unknown in advance is not free, either: we need to enlarge the buffer
> text each time more stuff arrives.

Yes, I was wondering about that -- how slow is resizing buffers, really?
Does Emacs rely on OS-level realloc that doesn't necessitate actually
copying all that data all the time?

Also -- some of the networking operations know in advance how much data
is going to be received.  For instance, when using non-chunked HTTP
(quite common when fetching "data", i.e., images, PDFs etc), we get the
content size in a header.  Would it make sense to have a way to tell
Emacs about this in advance so that it could pre-size the buffer?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#32728: bug#32729: Xemacs 23 times as fast as GNU Emacs

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: benjamin.benninghofen@airbus.com,  layer@franz.com,
>   32729@debbugs.gnu.org,  32728@debbugs.gnu.org
> Date: Sat, 12 Oct 2019 19:55:54 +0200
> 
> >> (Note: Don't visit the " *zeroes*" buffer after this, because that will
> >> hang Emacs totally.  I guess the long-line display problem hasn't been
> >> fixed after all?)
> >
> > It's unrelated.  Did you try to insert that into a unibyte buffer
> > instead?
> 
> Nope.  Does Emacs need to do a lot of recomputing when going to
> multibyte buffers?

Of course: we try to display the multibyte text as characters, search
for fonts, invoke bidi reordering, etc.

> >> And it's a real world problem: When reading data from any network
> >> source, you have to use filters because the protocol is usually based on
> >> parsing the output to find out when it's over, so you can't use
> >> sentinels.
> >
> > Why can't you use the default filter, and in the sentinel work on the
> > buffer with the complete response?
> 
> No sentinel is called, because the process status doesn't change,
> typically.  All the relevant network protocols do not close after having
> "done something" (IMAP, HTTP, etc), but instead use in-protocol markers
> to say that an operation is done.  So a filter has to be used.

OK, so not in sentinel, but in a timer or somesuch.

> > Please also note that, GC aside, inserting stuff of size that is
> > unknown in advance is not free, either: we need to enlarge the buffer
> > text each time more stuff arrives.
> 
> Yes, I was wondering about that -- how slow is resizing buffers, really?
> Does Emacs rely on OS-level realloc that doesn't necessitate actually
> copying all that data all the time?

Not necessarily realloc, sometimes mmap or similar.  And yes, this
requires copying the data when the OS cannot grow the previous
allocation, but instead gives us a memory block at another address.
In most systems this copying is under the hood, but it does happen.

> Also -- some of the networking operations know in advance how much data
> is going to be received.  For instance, when using non-chunked HTTP
> (quite common when fetching "data", i.e., images, PDFs etc), we get the
> content size in a header.  Would it make sense to have a way to tell
> Emacs about this in advance so that it could pre-size the buffer?

We could provide a Lisp interface for such cases.  The C
implementation is in enlarge_buffer_text (but we need to remember the
decoding of human-readable text, when applicable).

However, I would begin by measuring the effect of this resizing on the
time it takes to receive large amounts of data.  Maybe other factors
make this part negligible.

bug#32728: bug#32729: Xemacs 23 times as fast as GNU Emacs

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> No sentinel is called, because the process status doesn't change,
>> typically.  All the relevant network protocols do not close after having
>> "done something" (IMAP, HTTP, etc), but instead use in-protocol markers
>> to say that an operation is done.  So a filter has to be used.
>
> OK, so not in sentinel, but in a timer or somesuch.

A timer wastes resources by triggering work when there's nothing to do,
and conversely, makes network connections sluggish by not having the
code happen immediately when there's data ready.

After thinking about this a bit more, I'm reminded of the problems I had
in this area when doing the NSM, because I really wanted to put a filter
on (some of) the processes, but I couldn't, because we only allow one
filter per process.

So this is the design I want to do:

process-add-callback PROCESS FUNCTION
process-remove-callback PROCESS FUNCTION

FUNCTION takes three parameters: The PROCESS and the start/end of the
region inserted.  Perhaps it would make sense to do something with the
return values -- if the function returns non-nil, then further callbacks
are inhibited?

Anyway, with this, the current filters can trivially be implemented as a
callback instead -- the filters would just be wrapped in a function that
does (funcall filter-function (buffer-substring start end)) so that
we don't need duplicated code for these mechanisms on the C level.  So
set-process-filter and related moves to the Lisp level...

> We could provide a Lisp interface for such cases.  The C
> implementation is in enlarge_buffer_text (but we need to remember the
> decoding of human-readable text, when applicable).
>
> However, I would begin by measuring the effect of this resizing on the
> time it takes to receive large amounts of data.  Maybe other factors
> make this part negligible.

Sure.  My simple dd test (without a filter) surprised me by being as
fast as it was, so Emacs was able to grow that buffer quicker than I
expected.  But it's also a pretty simple test case -- I can try to see
what happens if I call enlarge_buffer_text to 1GB first and see what the
effects are.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#32729: Xemacs 23 times as fast as GNU Emacs

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: benjamin.benninghofen@airbus.com,  layer@franz.com,
>   32729@debbugs.gnu.org,  32728@debbugs.gnu.org
> Date: Sun, 13 Oct 2019 19:36:47 +0200
> 
> So this is the design I want to do:
> 
> process-add-callback PROCESS FUNCTION
> process-remove-callback PROCESS FUNCTION
> 
> FUNCTION takes three parameters: The PROCESS and the start/end of the
> region inserted.  Perhaps it would make sense to do something with the
> return values -- if the function returns non-nil, then further callbacks
> are inhibited?

I don't understand what would trigger these callbacks, and how do you
specify the region in advance, without knowing what will be inserted.

Without understanding this, I don't think I see the utility, and most
important: why this would be faster.

> > However, I would begin by measuring the effect of this resizing on the
> > time it takes to receive large amounts of data.  Maybe other factors
> > make this part negligible.
> 
> Sure.  My simple dd test (without a filter) surprised me by being as
> fast as it was, so Emacs was able to grow that buffer quicker than I
> expected.  But it's also a pretty simple test case -- I can try to see
> what happens if I call enlarge_buffer_text to 1GB first and see what the
> effects are.

Btw, unlike what I originally implied, the default filter also
receives a Lisp string, so the question why by default reading dd
output is so much faster than when you define a non-default filter
function still stands.

bug#32728: bug#32729: Xemacs 23 times as fast as GNU Emacs

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't understand what would trigger these callbacks, and how do you
> specify the region in advance, without knowing what will be inserted.

accept_process_output inserts the data into the buffer and then calls
the callback with the region in question.  Well,
read_and_dispose_of_process_output, I guess...

> Without understanding this, I don't think I see the utility, and most
> important: why this would be faster.

It would avoid creating (and garbaging) the strings.

> Btw, unlike what I originally implied, the default filter also
> receives a Lisp string, so the question why by default reading dd
> output is so much faster than when you define a non-default filter
> function still stands.

Oh!  That is curious indeed.  Are the Lisp_Object strings somehow
... special here when they never leave C land?  The speed differential
is completely repeatable...  hm...  Is the only difference that gc isn't
given a chance to run in the non-filter case?  Even if you subtract the
gc time, that doesn't explain the difference fully.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#32728: bug#32729: Xemacs 23 times as fast as GNU Emacs

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: benjamin.benninghofen@airbus.com,  layer@franz.com,
>   32729@debbugs.gnu.org,  32728@debbugs.gnu.org
> Date: Mon, 14 Oct 2019 10:36:11 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I don't understand what would trigger these callbacks, and how do you
> > specify the region in advance, without knowing what will be inserted.
> 
> accept_process_output inserts the data into the buffer and then calls
> the callback with the region in question.  Well,
> read_and_dispose_of_process_output, I guess...

Filter functions are called even if the Lisp program never calls
accept-process-output, so your proposal doesn't seem to be equivalent
to what we have now, right?

OTOH, if one has to call accept-process-output, then why do we need
callbacks?  Just extend accept-process-output to call a function with
the received output.  No?

> > Without understanding this, I don't think I see the utility, and most
> > important: why this would be faster.
> 
> It would avoid creating (and garbaging) the strings.

I'm not sure I see how.

The way it works now is that we get the process output as a C string;
we then decode it and make a Lisp string from the result of decoding;
and then we invoke the filter with that Lisp string.  (If the filter
is nil, we invoke internal-default-process-filter instead, but it
still gets the text as a string.)

Which part(s) of this will be avoided under your proposal?

> > Btw, unlike what I originally implied, the default filter also
> > receives a Lisp string, so the question why by default reading dd
> > output is so much faster than when you define a non-default filter
> > function still stands.
> 
> Oh!  That is curious indeed.  Are the Lisp_Object strings somehow
> ... special here when they never leave C land?

No, I don't think so.

> The speed differential is completely repeatable...  hm...  Is the
> only difference that gc isn't given a chance to run in the
> non-filter case?

You could test that hypothesis by setting gc-cons-threshold to a very
high value.

Bottom line: I think we must understand better what takes the time in
your last test case, before we discuss solutions.  I'd start by
profiling that with "M-x profiler-start".

bug#32728: bug#32729: Xemacs 23 times as fast as GNU Emacs

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> (Note: Don't visit the " *zeroes*" buffer after this, because that will
>> >> hang Emacs totally.  I guess the long-line display problem hasn't been
>> >> fixed after all?)
>> >
>> > It's unrelated.  Did you try to insert that into a unibyte buffer
>> > instead?
>> 
>> Nope.  Does Emacs need to do a lot of recomputing when going to
>> multibyte buffers?
>
> Of course: we try to display the multibyte text as characters, search
> for fonts, invoke bidi reordering, etc.

I tried:

(benchmark-run
    1
  (call-process "dd" nil (with-current-buffer
			     (get-buffer-create " *zeroes*")
			   (set-buffer-multibyte nil)
			   (current-buffer))
		nil "if=/dev/zero" "bs=4096" "count=250000"))

and then jumped to the buffer, and Emacs hung anyway.  (That is, I
killed Emacs after a minute, so I don't know whether it would have
recovered after a while...)

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#32728: bug#32729: Xemacs 23 times as fast as GNU Emacs

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: benjamin.benninghofen@airbus.com,  layer@franz.com,
>   32729@debbugs.gnu.org,  32728@debbugs.gnu.org
> Date: Sun, 13 Oct 2019 19:47:16 +0200
> 
> >> Nope.  Does Emacs need to do a lot of recomputing when going to
> >> multibyte buffers?
> >
> > Of course: we try to display the multibyte text as characters, search
> > for fonts, invoke bidi reordering, etc.
> 
> I tried:
> 
> (benchmark-run
>     1
>   (call-process "dd" nil (with-current-buffer
> 			     (get-buffer-create " *zeroes*")
> 			   (set-buffer-multibyte nil)
> 			   (current-buffer))
> 		nil "if=/dev/zero" "bs=4096" "count=250000"))
> 
> and then jumped to the buffer, and Emacs hung anyway.  (That is, I
> killed Emacs after a minute, so I don't know whether it would have
> recovered after a while...)

Do that, _and_ move point to the beginning of the buffer.

bug#32729: Xemacs 23 times as fast as GNU Emacs

[Lars Ingebrigtsen]

Actually, my benchmarking is somewhat wrong.

start-process with a filter, but discard output:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=/dev/zero" "bs=4096" "count=250000")))
      (set-process-filter proc (lambda (proc string)))
      (while (and (process-live-p proc)
		  (accept-process-output proc 1))))))
=> (18.828236636 59 13.315468088000017)

filter, but insert the output:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=/dev/zero" "bs=4096" "count=250000")))
      (set-process-filter proc (lambda (proc string)
				 (with-current-buffer (get-buffer " *zeroes*")
				   (goto-char (point-max))
				   (insert string))))
      (while (and (process-live-p proc)
		  (accept-process-output proc 1))))))
=> (21.120281346 59 13.250166416000013)

With the default filter:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=/dev/zero" "bs=4096" "count=250000")))
      (while (and (process-live-p proc)
		  (accept-process-output proc 1))))))
=> (34.046986424 116 26.025843717999976)

(!)

So the default filter is really slow?

Anyway, compare with call-process:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil "if=/dev/zero" "bs=4096" "count=250000")))
=> (1.694743653 0 0.0)

So what makes start-process 10x slower than call-process?  If it is all
the string creation before calling the filters, default or not, then my
point stands, but this obviously requires a more in-depth dive into
process.c.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#32728: bug#32729: Xemacs 23 times as fast as GNU Emacs

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: layer@franz.com,  32729@debbugs.gnu.org,
>   benjamin.benninghofen@airbus.com,  32728@debbugs.gnu.org
> Date: Mon, 14 Oct 2019 10:54:25 +0200
> 
> So what makes start-process 10x slower than call-process?

The fact that it goes through pselect, and only accepts the process
output when Emacs is idle?

> this obviously requires a more in-depth dive into process.c.

Agreed.

bug#32728: bug#32729: Xemacs 23 times as fast as GNU Emacs

[Benninghofen, Benjamin Dr.]

For me your discussion about the performance problem appears very theoretical.
Why do you not just compare the relevant GNU Emacs Source Code with the Xemacs Source Code?

Benjamin Benninghofen

-----Original Message-----
From: Lars Ingebrigtsen [mailto:larsi@gnus.org] 
Sent: Monday, October 14, 2019 10:54 AM
To: Eli Zaretskii
Cc: layer@franz.com; 32729@debbugs.gnu.org; Benninghofen, Benjamin Dr.; 32728@debbugs.gnu.org
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs

Actually, my benchmarking is somewhat wrong.

start-process with a filter, but discard output:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=/dev/zero" "bs=4096" "count=250000")))
      (set-process-filter proc (lambda (proc string)))
      (while (and (process-live-p proc)
		  (accept-process-output proc 1))))))
=> (18.828236636 59 13.315468088000017)

filter, but insert the output:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=/dev/zero" "bs=4096" "count=250000")))
      (set-process-filter proc (lambda (proc string)
				 (with-current-buffer (get-buffer " *zeroes*")
				   (goto-char (point-max))
				   (insert string))))
      (while (and (process-live-p proc)
		  (accept-process-output proc 1))))))
=> (21.120281346 59 13.250166416000013)

With the default filter:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run
      1
    (let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
			       "if=/dev/zero" "bs=4096" "count=250000")))
      (while (and (process-live-p proc)
		  (accept-process-output proc 1))))))
=> (34.046986424 116 26.025843717999976)

(!)

So the default filter is really slow?

Anyway, compare with call-process:

(let ((coding-system-for-read 'binary))
  (kill-buffer (get-buffer-create " *zeroes*"))
  (benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil "if=/dev/zero" "bs=4096" "count=250000")))
=> (1.694743653 0 0.0)

So what makes start-process 10x slower than call-process?  If it is all
the string creation before calling the filters, default or not, then my
point stands, but this obviously requires a more in-depth dive into
process.c.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

The information in this e-mail is confidential. The contents may not be disclosed or used by anyone other than the addressee. Access to this e-mail by anyone else is unauthorised.
If you are not the intended recipient, please notify Airbus immediately and delete this e-mail.
Airbus cannot accept any responsibility for the accuracy or completeness of this e-mail as it has been sent over public networks. If you have any concerns over the content of this message or its Accuracy or Integrity, please contact Airbus immediately.
All outgoing e-mails from Airbus are checked using regularly updated virus scanning software but you should take whatever measures you deem to be appropriate to ensure that this message and any attachments are virus free.

bug#32728: bug#32729: Xemacs 23 times as fast as GNU Emacs

[Eli Zaretskii]

> From: "Benninghofen, Benjamin Dr." <benjamin.benninghofen@airbus.com>
> CC: "layer@franz.com" <layer@franz.com>, "32729@debbugs.gnu.org"
> 	<32729@debbugs.gnu.org>, "32728@debbugs.gnu.org" <32728@debbugs.gnu.org>
> Date: Fri, 25 Oct 2019 06:38:16 +0000
> 
> For me your discussion about the performance problem appears very theoretical.

I don't think it was theoretical.  It was an attempt to identify which
factor(s) affect(s) this issue, so that further investigation could be
more focused.

> Why do you not just compare the relevant GNU Emacs Source Code with the Xemacs Source Code?

Did you try doing that?  Every time I did I found that the code is so
different that any comparison is very hard at best, and usually simply
meaningless.

bug#32729: Xemacs 23 times as fast as GNU Emacs

[Phil Sainty]

On 12/10/19 4:57 PM, Lars Ingebrigtsen wrote:
> (Note: Don't visit the " *zeroes*" buffer after this, because that
> will hang Emacs totally.  I guess the long-line display problem
> hasn't been fixed after all?)

I imagine you're thinking of so-long.el here, in which case there may
be a misconception.  so-long helps tremendously in certain situations,
but it doesn't (and cannot) eliminate all sources of potential
slowness in the face of extremely long lines -- ultimately the C code
still needs time to process the lines being displayed, and so-long.el
can't make that code work any faster.

The biggest problem in this case (by far) is that point has been left
at the end of the line following the insertion, and in order for the
redisplay to *display* the end of the line, it's having to process a
gigabyte of data.  My largest test case for so-long.el was a couple of
orders of magnitude smaller than this, and Emacs wasn't remotely happy
showing the end of that line, let alone this one.

(What so-long is really good at is preventing situations whereby
merely visiting a file -- with point still at the beginning of the
buffer -- caused Emacs to hang badly; which can easily happen when
various modes and options are active.  It can work wonders in those
cases.  If you try to display the end of a really gargantuan line of
data like this, though, you're simply at the mercy of other factors.)

If you added a (goto-char (point-min)) before switching to the
*zeroes* buffer then, given that there's not much else happening in
that buffer which would create issues, Emacs shouldn't hang when you
switch to it.

The further you move into that line, though, the worse the performance
is going to get (so the key take-away here would be to avoid doing that
when dealing with this kind of data, if at all possible).

Also note that `so-long' doesn't actually trigger automatically if you
simply create a buffer and throw a ton of data into it.  The automatic
behaviour is tied to `set-auto-mode' and intended to deal with users
visiting files.  Auto-detecting the sudden insertion of large lines
into an existing buffer is a different kettle of fish, and the current
library does not attempt to cater for it.  So if you wanted to trigger
it in this example you would need to call it explicitly.


-Phil

bug#32728: bug#32729: Xemacs 23 times as fast as GNU Emacs

[Lars Ingebrigtsen]

Phil Sainty <psainty@orcon.net.nz> writes:

> On 12/10/19 4:57 PM, Lars Ingebrigtsen wrote:
>> (Note: Don't visit the " *zeroes*" buffer after this, because that
>> will hang Emacs totally.  I guess the long-line display problem
>> hasn't been fixed after all?)
>
> I imagine you're thinking of so-long.el here, in which case there may
> be a misconception.

Yes, I hadn't followed the development closely, but just registered that
... things ... had gotten better.  :-)

> The biggest problem in this case (by far) is that point has been left
> at the end of the line following the insertion, and in order for the
> redisplay to *display* the end of the line, it's having to process a
> gigabyte of data.  My largest test case for so-long.el was a couple of
> orders of magnitude smaller than this, and Emacs wasn't remotely happy
> showing the end of that line, let alone this one.

Ah, I see.  Thanks for the explanation.

It is unfortunate that stuff like this makes Emacs hang, though.  It'd
be nice if Emacs had a "OK, we just give up" mode if the buffer is too
intractable, where "too intractable" may be, for instance, if it finds a
line that's longer than a few megabytes, perhaps?

I don't know what "just give up" would entail, though.  Just put point
at the start of the buffer and refuse to scroll or do anything?  Just
about anything would be better than the current situation where you have
to kill Emacs if you're playing with (some) binary files and try to
display the buffer.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#32729: Xemacs 23 times as fast as GNU Emacs

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Sun, 13 Oct 2019 19:24:58 +0200
> Cc: Kevin Layer <layer@franz.com>, "Benninghofen,
>  Benjamin Dr." <benjamin.benninghofen@airbus.com>, 32729@debbugs.gnu.org,
>  32728@debbugs.gnu.org
> 
> It is unfortunate that stuff like this makes Emacs hang, though.  It'd
> be nice if Emacs had a "OK, we just give up" mode if the buffer is too
> intractable, where "too intractable" may be, for instance, if it finds a
> line that's longer than a few megabytes, perhaps?
> 
> I don't know what "just give up" would entail, though.  Just put point
> at the start of the buffer and refuse to scroll or do anything?  Just
> about anything would be better than the current situation where you have
> to kill Emacs if you're playing with (some) binary files and try to
> display the buffer.

The problem is exactly to decide what to do when you "give up" in a
way that will still get you a functional editor that can display
something reasonable.