Speed of keyboard macro execution?

Speed of keyboard macro execution?

[Perry E. Metzger]

This video:

https://www.youtube.com/watch?v=sWy8zzdbRIY

discussed on this reddit thread:

https://www.reddit.com/r/emacs/comments/3vp583/macros_in_gnu_emacs_and_mg_speed_comparison/

discusses the fact that in some cases, keyboard macros take an awful
long time to execute. (In particular, the case in question executes in
a blink of an eye in the mg emacs-like editor and takes tens of
minutes to execute in Gnu Emacs.)

Perhaps it would be good to dig in to what is causing the issues in
question...

-- 
Perry E. Metzger		perry@piermont.com

Re: Speed of keyboard macro execution?

[David Engster]

Perry E. Metzger writes:
> Perhaps it would be good to dig in to what is causing the issues in
> question...

Emacs does not cope very well with very long lines.

-David

Re: Speed of keyboard macro execution?

[Perry E. Metzger]

On Wed, 09 Dec 2015 23:13:17 +0100 David Engster
<deng@randomsample.de> wrote:
> Perry E. Metzger writes:
> > Perhaps it would be good to dig in to what is causing the issues
> > in question...
> 
> Emacs does not cope very well with very long lines.

How might that be addressed? (Note that I've personally noted that
keyboard macros often take unexpectedly long times even when they do
not operate over long lines...)

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Speed of keyboard macro execution?

[Eli Zaretskii]

> Date: Wed, 9 Dec 2015 18:03:43 -0500
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: emacs-devel@gnu.org
> 
> On Wed, 09 Dec 2015 23:13:17 +0100 David Engster
> <deng@randomsample.de> wrote:
> > Perry E. Metzger writes:
> > > Perhaps it would be good to dig in to what is causing the issues
> > > in question...
> > 
> > Emacs does not cope very well with very long lines.
> 
> How might that be addressed?

This is simply bug#13675.

It's a bit disingenuous on the part of that person to compare Emacs
with another editor _solely_ where there's a known inefficiency in
Emacs, and pretend that this is somehow representative of the general
differences in performance between these two editors.  After all, how
frequently do you need to edit files with lines in excess of 11K
characters?

Re: Speed of keyboard macro execution?

[Perry E. Metzger]

On Thu, 10 Dec 2015 18:43:06 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > Date: Wed, 9 Dec 2015 18:03:43 -0500
> > From: "Perry E. Metzger" <perry@piermont.com>
> > Cc: emacs-devel@gnu.org
> > 
> > On Wed, 09 Dec 2015 23:13:17 +0100 David Engster
> > <deng@randomsample.de> wrote:
> > > Perry E. Metzger writes:
> > > > Perhaps it would be good to dig in to what is causing the
> > > > issues in question...
> > > 
> > > Emacs does not cope very well with very long lines.
> > 
> > How might that be addressed?
> 
> This is simply bug#13675.
> 
> It's a bit disingenuous on the part of that person to compare Emacs
> with another editor _solely_ where there's a known inefficiency in
> Emacs, and pretend that this is somehow representative of the
> general differences in performance between these two editors.
> After all, how frequently do you need to edit files with lines in
> excess of 11K characters?

I don't know if it was disingenuous or not, but I think that being
able to improve performance would not be a bad thing. I've noticed
that a lot of things (like keyboard macros) are often slower than
expected in modern Emacs. We may, as a developer community, have
allowed the speed of modern hardware to lull us into complacency on
issues like performance. Emacs should be the best editor it can
be.

I suppose this ties back in to earlier discussions about automated
testing (as, among other things, automated testing can find
performance regressions before they matter.)

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Speed of keyboard macro execution?

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1011 bytes --]

>>>>> Perry E Metzger <perry@piermont.com> writes:

> I don't know if it was disingenuous or not, but I think that being able to
> improve performance would not be a bad thing. I've noticed that a lot of
> things (like keyboard macros) are often slower than expected in modern
> Emacs. We may, as a developer community, have allowed the speed of modern
> hardware to lull us into complacency on issues like performance. Emacs
> should be the best editor it can be.

I would very much like someone to step up as our "performance czar", and to
create a benchmarking suite to run alongside our test suite, to establish a
baseline for operations which should always be quick in Emacs.

Efficiency matters a lot to me, although at the moment it falls because
correctness and not crashing. But I'd like to improve in this area as well
over time.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Speed of keyboard macro execution?

[David Engster]

Perry E. Metzger writes:
> I don't know if it was disingenuous or not, but I think that being
> able to improve performance would not be a bad thing.

No one is claiming that. AFAIK, the very-long-line problem is not easy
to fix.

> I've noticed that a lot of things (like keyboard macros) are often
> slower than expected in modern Emacs.

Then please make a bug report about it.

-David

Re: Speed of keyboard macro execution?

[Perry E. Metzger]

On Thu, 10 Dec 2015 18:27:49 +0100 David Engster
<deng@randomsample.de> wrote:
> Perry E. Metzger writes:
> > I don't know if it was disingenuous or not, but I think that being
> > able to improve performance would not be a bad thing.
> 
> No one is claiming that. AFAIK, the very-long-line problem is not
> easy to fix.

Perhaps it isn't. Looking at the old thread, it seems like the bidi
code is one of the culprits. OTOH, if I read this right and the
problem is partially slow redisplay, I'm wondering why the redisplay
code needs to be involved at all if a long keyboard macro is being
run. Perhaps there should be some way to suspend redisplay, perform
the macro, and then restart it? I have to admit I haven't looked at
the code involved and it might be utterly infeasible.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Speed of keyboard macro execution?

[David Engster]

Perry E. Metzger writes:
> OTOH, if I read this right and the problem is partially slow
> redisplay, I'm wondering why the redisplay code needs to be involved
> at all if a long keyboard macro is being run. Perhaps there should be
> some way to suspend redisplay, perform the macro, and then restart it?

From the video, it doesn't look like there's any redisplay happening
during macro execution.

-David

Re: Speed of keyboard macro execution?

[Eli Zaretskii]

> Date: Thu, 10 Dec 2015 12:33:12 -0500
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> On Thu, 10 Dec 2015 18:27:49 +0100 David Engster
> <deng@randomsample.de> wrote:
> > Perry E. Metzger writes:
> > > I don't know if it was disingenuous or not, but I think that being
> > > able to improve performance would not be a bad thing.
> > 
> > No one is claiming that. AFAIK, the very-long-line problem is not
> > easy to fix.
> 
> Perhaps it isn't. Looking at the old thread, it seems like the bidi
> code is one of the culprits.

No, if this is your conclusion, you are not reading it right.  The
bidi code makes the awfully slow cases slower, but they are awfully,
unbearably slow even when bidi is turned off.  IOW, in practical
terms, bidi didn't change anything in those cases: half the infinity
is still infinity.

> OTOH, if I read this right and the problem is partially slow
> redisplay, I'm wondering why the redisplay code needs to be involved
> at all if a long keyboard macro is being run. Perhaps there should
> be some way to suspend redisplay, perform the macro, and then
> restart it? I have to admit I haven't looked at the code involved
> and it might be utterly infeasible.

This I can explain.  "Redisplay" comes into play not only for actually
redrawing the screen, it also comes into play when, e.g., a command
moves cursor down, as in C-n.  With (the default) line-move-visual
mode, Emacs no longer has an easy way of knowing what character is
directly below the current one, what with all the variable-pitch fonts
and long lines broken between several screen lines.  The only way of
finding out where is "one visual line down" is to _simulate_
redisplay.  IOW, Emacs runs code that does exactly what redisplay
does, but without displaying anything; it just accumulates pixel
coordinates until it finds itself one visual line down.

Any keyboard macro that moves point will at some point invoke this
code, because moving point in a variable-pitch multi-font display is a
hard problem, and this is even before we consider complications like
display properties, overlay strings, embedded images, and the other
calamities.

Re: Speed of keyboard macro execution?

[Perry E. Metzger]

On Thu, 10 Dec 2015 19:53:11 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> This I can explain.  "Redisplay" comes into play not only for
> actually redrawing the screen, it also comes into play when, e.g.,
> a command moves cursor down, as in C-n.  With (the default)
> line-move-visual mode, Emacs no longer has an easy way of knowing
> what character is directly below the current one, what with all the
> variable-pitch fonts and long lines broken between several screen
> lines.  The only way of finding out where is "one visual line down"
> is to _simulate_ redisplay.  IOW, Emacs runs code that does exactly
> what redisplay does, but without displaying anything; it just
> accumulates pixel coordinates until it finds itself one visual line
> down.

Oh, my. Now I understand. Especially once you have issues like (as
just one example) complicated Unicode rendering involved, this could
get arbitrarily messy.

I suppose there are only two obvious ways to handle that:

1) radically speed up the redisplay code.
2) memoize the output of the redisplay code.

Have I missed an obvious approach or are those the choices?

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Speed of keyboard macro execution?

[Eli Zaretskii]

> Date: Thu, 10 Dec 2015 13:10:37 -0500
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: deng@randomsample.de, emacs-devel@gnu.org
> 
> I suppose there are only two obvious ways to handle that:
> 
> 1) radically speed up the redisplay code.
> 2) memoize the output of the redisplay code.

I think there's not much place for optimizations left, at least not by
large factors (you need something like 100-fold speedup to make long
lines bearable).

So 2) is the way to go, IMO.  As usual, the devil is in the details:
the design of a cache (or caches) that will hold the necessary data,
and don't hold too much of it.

> Have I missed an obvious approach or are those the choices?

No, I don't think you have.

Thanks.

Re: Speed of keyboard macro execution?

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

> This I can explain.  "Redisplay" comes into play not only for actually
> redrawing the screen, it also comes into play when, e.g., a command
> moves cursor down, as in C-n.  With (the default) line-move-visual
> mode, Emacs no longer has an easy way of knowing what character is
> directly below the current one, what with all the variable-pitch fonts
> and long lines broken between several screen lines.

I don't think there is much sense in having line-move-visual mode active
when recording/replaying keyboard macros.

Tying the operation of keyboard macros to the current display/font
selection is just meaningless.  Its purpose is for _aiming_ positioning
by keyboard, and that's just not useful at keyboard replay.

-- 
David Kastrup

Re: Speed of keyboard macro execution?

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: "Perry E. Metzger" <perry@piermont.com>,  deng@randomsample.de,  emacs-devel@gnu.org
> Date: Thu, 10 Dec 2015 19:44:54 +0100
> 
> I don't think there is much sense in having line-move-visual mode active
> when recording/replaying keyboard macros.
> 
> Tying the operation of keyboard macros to the current display/font
> selection is just meaningless.  Its purpose is for _aiming_ positioning
> by keyboard, and that's just not useful at keyboard replay.

I think it depends on the keyboard macro.  The ones I saw in that demo
did move point, moreover they moved it to the end of a very long line,
so both the actual redisplay and its simulation were at work,
including auto-hscroll.

I don't see how you can prevent that: macros are just a mechanical way
of recording and replaying commands, and commands sure do need to run
this code.

In any case, keyboard macros are not relevant to the discussion
(contrary to the subject line).  The issue is slow redisplay with long
lines.

Re: Speed of keyboard macro execution?

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: "Perry E. Metzger" <perry@piermont.com>, deng@randomsample.de,
>> emacs-devel@gnu.org
>> Date: Thu, 10 Dec 2015 19:44:54 +0100
>> 
>> I don't think there is much sense in having line-move-visual mode active
>> when recording/replaying keyboard macros.
>> 
>> Tying the operation of keyboard macros to the current display/font
>> selection is just meaningless.  Its purpose is for _aiming_ positioning
>> by keyboard, and that's just not useful at keyboard replay.
>
> I think it depends on the keyboard macro.  The ones I saw in that demo
> did move point, moreover they moved it to the end of a very long line,
> so both the actual redisplay and its simulation were at work,
> including auto-hscroll.

So how did line-move-visual accomplish anything useful here?  It is not
the question whether line-move-visual was involved here or not:
obviously it was or it could hardly have affected the benchmark.

> I don't see how you can prevent that: macros are just a mechanical way
> of recording and replaying commands, and commands sure do need to run
> this code.
>
> In any case, keyboard macros are not relevant to the discussion
> (contrary to the subject line).  The issue is slow redisplay with long
> lines.

Which occured during keyboard macro execution due to line-move-visual
being active.  Yes, improving the display engine speed would have sped
up this benchmark.  But if the benchmark did not actually accomplish
anything that would be useful in the course of editing, speeding the
display engine up will not lead to a corresponding speedup to anything
that would be useful in the course of editing.

So I think it would make excellent sense to disable visual positioning
modes while recording and replaying keyboard macros.  Not in order to
cheat at benchmarks, but to have actually useful behavior for a keyboard
macro that is intended to run without human intervention afterwards and
produce meaningful results.

I'll grant that paragraph adjustment according to visual width might
make sense in a keyboard macro and that _would_ exercise the display
engine by necessity.  But visual movement?  In a keyboard macro?  Not
really.

-- 
David Kastrup

Re: Speed of keyboard macro execution?

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: emacs-devel@gnu.org,  deng@randomsample.de,  perry@piermont.com
> Date: Thu, 10 Dec 2015 20:38:58 +0100
> 
> >> Tying the operation of keyboard macros to the current display/font
> >> selection is just meaningless.  Its purpose is for _aiming_ positioning
> >> by keyboard, and that's just not useful at keyboard replay.
> >
> > I think it depends on the keyboard macro.  The ones I saw in that demo
> > did move point, moreover they moved it to the end of a very long line,
> > so both the actual redisplay and its simulation were at work,
> > including auto-hscroll.
> 
> So how did line-move-visual accomplish anything useful here?

One example is vertical-motion, and many commands that call it.  E.g.,
if the macro invokes C-n.

> > In any case, keyboard macros are not relevant to the discussion
> > (contrary to the subject line).  The issue is slow redisplay with long
> > lines.
> 
> Which occured during keyboard macro execution due to line-move-visual
> being active.

No, not necessarily.  Rather, the original issue with keyboard macros
is just one of the scenarios that brings the broader problem to light,
if the macro invokes one of the affected commands.

> So I think it would make excellent sense to disable visual positioning
> modes while recording and replaying keyboard macros.

No, I think you are wrong.  Think C-n and C-p again.

Re: Speed of keyboard macro execution?

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: emacs-devel@gnu.org,  deng@randomsample.de,  perry@piermont.com
>> Date: Thu, 10 Dec 2015 20:38:58 +0100
>> 
>> >> Tying the operation of keyboard macros to the current display/font
>> >> selection is just meaningless.  Its purpose is for _aiming_ positioning
>> >> by keyboard, and that's just not useful at keyboard replay.
>> >
>> > I think it depends on the keyboard macro.  The ones I saw in that demo
>> > did move point, moreover they moved it to the end of a very long line,
>> > so both the actual redisplay and its simulation were at work,
>> > including auto-hscroll.
>> 
>> So how did line-move-visual accomplish anything useful here?
>
> One example is vertical-motion, and many commands that call it.  E.g.,
> if the macro invokes C-n.

Uh WHAT?

    C-n runs the command next-line (found in global-map), which is an
    interactive compiled Lisp function in ‘simple.el’.

    This function is for interactive use only;
    in Lisp code use ‘forward-line’ instead.

There is a reason this function is for interactive use only.  And
replaying a keyboard macro is not interactive.

    If the variable ‘line-move-visual’ is non-nil, this command moves
    by display lines.  Otherwise, it moves by buffer lines, without
    taking variable-width characters or continued lines into account.

And this is _exactly_ why line-move-visual should be nil during macro
recording/replay because otherwise _no_ _useful_ behavior for
non-interactive use results.

>> Which occured during keyboard macro execution due to line-move-visual
>> being active.
>
> No, not necessarily.  Rather, the original issue with keyboard macros
> is just one of the scenarios that brings the broader problem to light,
> if the macro invokes one of the affected commands.

No argument here.  I'm not against fixing the redisplay problem.  I'm
against involving display-based movement in keyboard macros.

>> So I think it would make excellent sense to disable visual
>> positioning modes while recording and replaying keyboard macros.
>
> No, I think you are wrong.  Think C-n and C-p again.

That's exactly the reason for me wanting to disable visual positioning
modes while recording and replaying keyboard macros.

I mean, obviously I would not proposing it if I thought it should not
make a difference.

So I have no idea what your argument here is supposed to be.

-- 
David Kastrup

Re: Speed of keyboard macro execution?

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Cc: emacs-devel@gnu.org,  deng@randomsample.de,  perry@piermont.com
> Date: Thu, 10 Dec 2015 21:09:59 +0100
> 
>     C-n runs the command next-line (found in global-map), which is an
>     interactive compiled Lisp function in ‘simple.el’.
> 
>     This function is for interactive use only;
>     in Lisp code use ‘forward-line’ instead.
> 
> There is a reason this function is for interactive use only.  And
> replaying a keyboard macro is not interactive.
> 
>     If the variable ‘line-move-visual’ is non-nil, this command moves
>     by display lines.  Otherwise, it moves by buffer lines, without
>     taking variable-width characters or continued lines into account.
> 
> And this is _exactly_ why line-move-visual should be nil during macro
> recording/replay because otherwise _no_ _useful_ behavior for
> non-interactive use results.

But then recording a macro and replaying it will do different things,
no?

Re: Speed of keyboard macro execution?

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Cc: emacs-devel@gnu.org,  deng@randomsample.de,  perry@piermont.com
>> Date: Thu, 10 Dec 2015 21:09:59 +0100
>> 
>>     C-n runs the command next-line (found in global-map), which is an
>>     interactive compiled Lisp function in ‘simple.el’.
>> 
>>     This function is for interactive use only;
>>     in Lisp code use ‘forward-line’ instead.
>> 
>> There is a reason this function is for interactive use only.  And
>> replaying a keyboard macro is not interactive.
>> 
>>     If the variable ‘line-move-visual’ is non-nil, this command moves
>>     by display lines.  Otherwise, it moves by buffer lines, without
>>     taking variable-width characters or continued lines into account.
>> 
>> And this is _exactly_ why line-move-visual should be nil during macro
>> recording/replay because otherwise _no_ _useful_ behavior for
>> non-interactive use results.
>
> But then recording a macro and replaying it will do different things,
> no?

As I apparently did not state clearly enough, one would switch off
visual line mode during recording already (after all, during recording
one tries to achieve the effect that replay should have).

Yes, this will means that the same key sequence outside of macro
recording/replay and inside of it will have different effects.

I readily agree that this is ugly, and if you want to, even "wrong".
The alternative is "useless".  Which serves no point.

-- 
David Kastrup

Re: Speed of keyboard macro execution?

[Perry E. Metzger]

On Thu, 10 Dec 2015 22:00:05 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> > From: David Kastrup <dak@gnu.org>
> > So I think it would make excellent sense to disable visual
> > positioning modes while recording and replaying keyboard macros.
> 
> No, I think you are wrong.  Think C-n and C-p again.

Although I loathe visual positioning (and I disable it), if a user has
that turned on, they'll expect that keyboard macros will operate as
expected, which is to say, that they'll follow the visual up and down
when C-n and C-p are invoked. I think Eli is right that just disabling
it can't be done.

Further, I think Eli is right that the broader problem needs to be
fixed. It almost certainly causes lots of other slowdowns as well.

It seems that the best way to do this effectively is to add a
memoization data structure that caches the redisplay information for
the the buffer. Presumably, when there aren't "unusual" things in the
buffer bits could be set in the structure to simplify the caching (and
allow it to be more compact).

Having been thinking about this for essentially no time and with
little knowledge of the information needed I'm not yet prepared to
make any sort of concrete proposal. Input from people with
significantly more knowledge than me is therefore quite welcome.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Speed of keyboard macro execution?

[John Wiegley]

>>>>> Perry E Metzger <perry@piermont.com> writes:

> On Thu, 10 Dec 2015 22:00:05 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
>> > From: David Kastrup <dak@gnu.org>
>> > So I think it would make excellent sense to disable visual
>> > positioning modes while recording and replaying keyboard macros.
>> 
>> No, I think you are wrong.  Think C-n and C-p again.

Recording a macro has the value that it does what the user would have done,
just automated. If you disable visual positioning, then in many cases it no
longer does what I would have done manually.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Speed of keyboard macro execution?

[David Kastrup]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> Perry E Metzger <perry@piermont.com> writes:
>
>> On Thu, 10 Dec 2015 22:00:05 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
>>> > From: David Kastrup <dak@gnu.org>
>>> > So I think it would make excellent sense to disable visual
>>> > positioning modes while recording and replaying keyboard macros.
>>> 
>>> No, I think you are wrong.  Think C-n and C-p again.
>
> Recording a macro has the value that it does what the user would have
> done, just automated. If you disable visual positioning, then in many
> cases it no longer does what I would have done manually.

Visual positioning makes no sense without visual feedback.  Can you show
a single example of a task to be done by a keyboard macro where visual
positioning would _help_ achieve that task rather than sabotage it?

Just one?  For your argument's sake?  I certainly get the ugliness of
changing a setting during macro record/replay.  No question about that.
But the alternative is keyboard macros being essentially useless because
their effects on the text become unpredictable.

Yes, they are then perfectly equivalent to pressing the same keys
outside of macro recording/replay, but since the pressing of the keys is
done based on visual feedback, there is just no point in recording key
presses during visual movement mode and replay them at some other buffer
position, or even the same buffer position but with different visuals.

In this case, I prefer "wrong" to "useless".

-- 
David Kastrup

Re: Speed of keyboard macro execution?

[John Wiegley]

>>>>> David Kastrup <dak@gnu.org> writes:

> Visual positioning makes no sense without visual feedback. Can you show a
> single example of a task to be done by a keyboard macro where visual
> positioning would _help_ achieve that task rather than sabotage it?

I don't get this argument. What is the real problem here? I've never faced on
issue on this point with macros before, and I've been using them for a long
time. In cases where visual positioning became a problem, I used
`toggle-truncate-lines'. I'd much rather have intervention be manual, than
changing the core behavior in ways that are hard to understand. It may well be
that there are times when I want visual positioning without feedback, simply
because it matches what I'm expecting to happen.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Speed of keyboard macro execution?

[David Kastrup]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> David Kastrup <dak@gnu.org> writes:
>
>> Visual positioning makes no sense without visual feedback. Can you show a
>> single example of a task to be done by a keyboard macro where visual
>> positioning would _help_ achieve that task rather than sabotage it?
>
> I don't get this argument. What is the real problem here? I've never
> faced on issue on this point with macros before, and I've been using
> them for a long time.

Have you ever had a case where visual positioning during macro
recording/replay was helpful?  Or have you "never faced this issue"
merely because your lines were short enough that the difference was
irrelevant?

> In cases where visual positioning became a problem, I used
> `toggle-truncate-lines'.

Depending on what font highlighting may do, vertical movement based on
visual entities is still different.

> I'd much rather have intervention be manual, than changing the core
> behavior in ways that are hard to understand. It may well be that
> there are times when I want visual positioning without feedback,
> simply because it matches what I'm expecting to happen.

So you don't have an example where having macros recorded/replayed using
visual movements would be useful, but you would not want it different on
principle.

Can we have an option to customize for less principled people?

-- 
David Kastrup

Re: Speed of keyboard macro execution?

[John Wiegley]

>>>>> David Kastrup <dak@gnu.org> writes:

> So you don't have an example where having macros recorded/replayed using
> visual movements would be useful, but you would not want it different on
> principle.

I still don't see this as significant enough even to warrant a customization
variable. Is there a clear example of why a change should be made? So far all
I've heard are arguments about what seems "right" to one person.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Speed of keyboard macro execution?

[David Kastrup]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> David Kastrup <dak@gnu.org> writes:
>
>> So you don't have an example where having macros recorded/replayed using
>> visual movements would be useful, but you would not want it different on
>> principle.
>
> I still don't see this as significant enough even to warrant a
> customization variable. Is there a clear example of why a change
> should be made? So far all I've heard are arguments about what seems
> "right" to one person.

    C-x e runs the command kmacro-end-and-call-macro (found in
    global-map), which is an interactive autoloaded compiled Lisp function
    in ‘kmacro.el’.

    It is bound to C-x e.

    (kmacro-end-and-call-macro ARG &optional NO-REPEAT)

    Call last keyboard macro, ending it first if currently being defined.
    With numeric prefix ARG, repeat macro that many times.
    Zero argument means repeat until there is an error.

Repeating a keyboard macro without intervening other actions means that
the macro, after achieving its changes, has to progress to the next
position where executing it makes sense as its last part.  This progress
will often involve moving to the next line.  The next _visual_ line is
meaningless for that purpose in almost all situations.  Under manual
control, you'll just move across some continuation line.  But a repeated
keyboard macro is not under manual control.

My claim is not about "right" but "useful" behavior.  We've had one
other person state that he switches off visual-line-mode always since
keyboard macros would otherwise be useless.  Of course that's one
possibility, but visual-line-mode is actually useful for _visual_
navigation.  Which is not what happens during macro execution since the
computer executes the recorded keys without human intervention.  Which
makes sense in _similar_ but not identical situations (for identical
situations, one could just copy and paste the result repeatedly).  And
non-identical situations will sometimes have lines wrapped visually and
sometimes not without that being related to the structure of the
modified text.

I am surprised that I should be the only user of keyboard macros having
noticed that.

-- 
David Kastrup

Re: Speed of keyboard macro execution?

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1461 bytes --]

>>>>> David Kastrup <dak@gnu.org> writes:

> My claim is not about "right" but "useful" behavior. We've had one other
> person state that he switches off visual-line-mode always since keyboard
> macros would otherwise be useless. Of course that's one possibility, but
> visual-line-mode is actually useful for _visual_ navigation. Which is not
> what happens during macro execution since the computer executes the recorded
> keys without human intervention. Which makes sense in _similar_ but not
> identical situations (for identical situations, one could just copy and
> paste the result repeatedly). And non-identical situations will sometimes
> have lines wrapped visually and sometimes not without that being related to
> the structure of the modified text.

I think we're presupposing what users want, and that is no reason to change a
long-standing default. You can always disable visual movement during recording
and playback manually; I don't see why it needs to happen automatically now.

If you want a customization option to auto-disable visual movement during
recording and playback, it should be easy to achieve with advice. Give it a
try for a few months, and then tell me if you really find it to be a quality
of life improvement. If so, I'd welcome the customization.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Speed of keyboard macro execution?

[David Kastrup]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> David Kastrup <dak@gnu.org> writes:
>
>> My claim is not about "right" but "useful" behavior. We've had one other
>> person state that he switches off visual-line-mode always since keyboard
>> macros would otherwise be useless. Of course that's one possibility, but
>> visual-line-mode is actually useful for _visual_ navigation. Which is not
>> what happens during macro execution since the computer executes the recorded
>> keys without human intervention. Which makes sense in _similar_ but not
>> identical situations (for identical situations, one could just copy and
>> paste the result repeatedly). And non-identical situations will sometimes
>> have lines wrapped visually and sometimes not without that being related to
>> the structure of the modified text.
>
> I think we're presupposing what users want, and that is no reason to change a
> long-standing default.

I'm still waiting for a single example where the current behavior would
be actually useful for keyboard macro execution.

> You can always disable visual movement during recording and playback
> manually; I don't see why it needs to happen automatically now.

line-move-visual is not a minor mode.  Changing it requires setting of a
variable.  Setting a variable and resetting it afterwards every time one
wants to use keyboard macros requires a different level of expertise
than that required to use keyboard macros in the first place.  Apart
from being a nuisance.

> If you want a customization option to auto-disable visual movement
> during recording and playback, it should be easy to achieve with
> advice. Give it a try for a few months, and then tell me if you really
> find it to be a quality of life improvement. If so, I'd welcome the
> customization.

Can we get a single case where the current behavior would be useful?
Just one?  Or is this really just an academic exercise?

-- 
David Kastrup

Re: Speed of keyboard macro execution?

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1001 bytes --]

>>>>> David Kastrup <dak@gnu.org> writes:

>> I think we're presupposing what users want, and that is no reason to change a
>> long-standing default.

> I'm still waiting for a single example where the current behavior would be
> actually useful for keyboard macro execution.

If I don't come up with an example right now, it doesn't mean that it's not
useful, only that I don't have an example.

You're proposing a change to existing behavior, and I fail to see a compelling
enough reason for that change. Not only is the new behavior not compelling to
me yet, but the safety of the change (because it *does* change how existing
macros behave) is definitely not compelling enough.

Asking me to defend why the status quo shouldn't be changed has nothing to do
with whether your suggested feature should become the new default.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Speed of keyboard macro execution?

[David Kastrup]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> David Kastrup <dak@gnu.org> writes:
>
>>> I think we're presupposing what users want, and that is no reason to change a
>>> long-standing default.
>
>> I'm still waiting for a single example where the current behavior would be
>> actually useful for keyboard macro execution.
>
> If I don't come up with an example right now, it doesn't mean that it's not
> useful, only that I don't have an example.
>
> You're proposing a change to existing behavior, and I fail to see a
> compelling enough reason for that change. Not only is the new behavior
> not compelling to me yet, but the safety of the change (because it
> *does* change how existing macros behave) is definitely not compelling
> enough.

Changing the default font will already change how existing macros
behave.  Does that mean that we will not ever change the default font of
Emacs again?  Since that would make visual line movement end up in
different places than previously?

> Asking me to defend why the status quo shouldn't be changed has
> nothing to do with whether your suggested feature should become the
> new default.

Because currently keyboard macros become unpredictable as soon as line
movement commands are involved?

At any rate, it's obvious that there will neither be agreement, nor any
example where the current mode of operation would be desirable.

So we might as well let the matter drop.  It's only tangential to the
"Speed of keyboard macro execution" subject anyway.

-- 
David Kastrup

Re: Speed of keyboard macro execution?

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 647 bytes --]

>>>>> David Kastrup <dak@gnu.org> writes:

> At any rate, it's obvious that there will neither be agreement, nor any
> example where the current mode of operation would be desirable.

Yes, there can be agreement: Try it locally, and come back in a few and tell
me how wonderful it was. Then I'd be much more interested in supporting a new
customization option in core. Perhaps even in a future version of Emacs, after
others have had a chance to use it, it could become the default.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Speed of keyboard macro execution?

[David Kastrup]

John Wiegley <jwiegley@gmail.com> writes:

>>>>>> David Kastrup <dak@gnu.org> writes:
>
>> At any rate, it's obvious that there will neither be agreement, nor any
>> example where the current mode of operation would be desirable.
>
> Yes, there can be agreement: Try it locally, and come back in a few
> and tell me how wonderful it was.

It couldn't be as wonderful as the communication style on this list, so
let's just drop this.

-- 
David Kastrup

Re: Speed of keyboard macro execution?

[Per Starbäck]

> Repeating a keyboard macro without intervening other actions means that
> the macro, after achieving its changes, has to progress to the next
> position where executing it makes sense as its last part.  This progress
> will often involve moving to the next line.  The next _visual_ line is
> meaningless for that purpose in almost all situations.  Under manual
> control, you'll just move across some continuation line.  But a repeated
> keyboard macro is not under manual control.
>
> My claim is not about "right" but "useful" behavior.  We've had one
> other person state that he switches off visual-line-mode always since
> keyboard macros would otherwise be useless.  Of course that's one
> possibility, but visual-line-mode is actually useful for _visual_
> navigation.  Which is not what happens during macro execution since the
> computer executes the recorded keys without human intervention.  Which
> makes sense in _similar_ but not identical situations (for identical
> situations, one could just copy and paste the result repeatedly).  And
> non-identical situations will sometimes have lines wrapped visually and
> sometimes not without that being related to the structure of the
> modified text.
>
> I am surprised that I should be the only user of keyboard macros having
> noticed that.

Of course you are not! Your proposed change would make it a easier to
write macros. Now I often resort to work-arounds like C-e C-f and then
more navigation to get to the right place in the new line to be sure
to actually get to the next line in the typical case where you run a
macro on one line at a time.

Since I teach a course for GNU beginners with some Emacs in it, I see
what kind of mistakes beginners make when using macros. There is no
surprise there, I think. They navigate with visual feedback in a way
that works fine for the line they are in, but that doesn't do the
"same" thing on (some) other lines.

First it can be totally obvious like just typing <right> <right>
<right> ... <right> to get to the end of line. Later perhaps using
word navigation in a CSV file where most fields are one word, but
where there are some fields with several words where their macros
won't work.

line-move-visual invites this sort of using visual feedback that only
works for a particular line and can lead to the same kind of
surprises. It has been written here that not changing any settings
when defining macros leads to less surprises. Yes, less surprises
*while defining*, but more surprises when using those macros. And then
it's certainly possible that you don't notice it, because you just ran
your macro on all lines in a buffer without checking every one.

I think one reason not many have written in support for your position
is that some experienced Emacs users turn off visual-line-mode and
think that it's mostly there for less experienced users, so they don't
have any problems. And the less experienced users don't use macros
that much (except when forced to in my course :-) .

Re: Speed of keyboard macro execution?

[Perry E. Metzger]

On Thu, 10 Dec 2015 13:26:39 -0800 John Wiegley <jwiegley@gmail.com>
wrote:
> >>>>> David Kastrup <dak@gnu.org> writes:
> 
> > So you don't have an example where having macros
> > recorded/replayed using visual movements would be useful, but you
> > would not want it different on principle.
> 
> I still don't see this as significant enough even to warrant a
> customization variable. Is there a clear example of why a change
> should be made? So far all I've heard are arguments about what
> seems "right" to one person.

If it isn't made the default, I think it is fairly harmless to have
a variable you could set to trigger David's proposed behavior. I
doubt many people are going to use it, but if they do, they've
explicitly chosen they want it, and it isn't in general our business
to tell people how they should enjoy editing. Emacs is a highly
programmable system, and if this the (non-default!) behavior is what
someone wants, well, adding it means that people don't have to write
the hack to do it on their own.

That said, the default has to follow the principle of least surprise I
think.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Speed of keyboard macro execution?

[David Kastrup]

"Perry E. Metzger" <perry@piermont.com> writes:

> If it isn't made the default, I think it is fairly harmless to have
> a variable you could set to trigger David's proposed behavior. I
> doubt many people are going to use it, but if they do, they've
> explicitly chosen they want it, and it isn't in general our business
> to tell people how they should enjoy editing. Emacs is a highly
> programmable system, and if this the (non-default!) behavior is what
> someone wants, well, adding it means that people don't have to write
> the hack to do it on their own.
>
> That said, the default has to follow the principle of least surprise I
> think.

We wouldn't have Emacs under that premise.  Joking aside I'm not that
sure that there will be a lot of surprise involved here since the whole
point of line-move-visual is to operate under visual feedback so people
will tend to move as far as necessary.

There also is the possibility of a have-one's-cake-and-eat-it-too
approach: when possible macro recording omits any "purely visual"
movements and replaces the recorded sequences with equivalents that
don't move visually.

It would be my guess that this would likely match naive expectations
_better_ than straightforward recording/playback.  However, it would be
underhanded of the "I'll do what you want instead of what you say" kind.
Switching off visual movement explicitly during recording/replay seems
less sketchy in comparison.

-- 
David Kastrup

Disabling visual lines for macros (was: Speed of keyboard macro execution?)

[John Wiegley]

[-- Attachment #1: Type: text/plain, Size: 1310 bytes --]

>>>>> Perry E Metzger <perry@piermont.com> writes:

> If it isn't made the default, I think it is fairly harmless to have a
> variable you could set to trigger David's proposed behavior. I doubt many
> people are going to use it, but if they do, they've explicitly chosen they
> want it, and it isn't in general our business to tell people how they should
> enjoy editing. Emacs is a highly programmable system, and if this the
> (non-default!) behavior is what someone wants, well, adding it means that
> people don't have to write the hack to do it on their own.

Such functionality does not require a new option in core, which the
maintainers then have to care for (documentation, testing, ensuring
correctness through future changes, etc). For it to receive that level of
"support", I want more evidence of its value.

Meanwhile, it can easily be done as an external module using advice. This
should be done first anyway, by David, to test whether it's as useful as he
thinks it is. Otherwise, we're just debating an idea without facts. We don't
even know if this would address any of the performance issues raised by this
thread.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 629 bytes --]

Re: Disabling visual lines for macros

[David Kastrup]

John Wiegley <jwiegley@gmail.com> writes:

> This should be done first anyway, by David, to test whether it's as
> useful as he thinks it is. Otherwise, we're just debating an idea
> without facts. We don't even know if this would address any of the
> performance issues raised by this thread.

The "performance issue" should be beside the point.  However, I rather
doubt that if the performance issue was related to visual line movement
that the keyboard macro in the test did the same job as the Emacs
competitor: visual line movement would likely cause the results to
differ even when run in two frames with different fonts and/or frame
width in the same Emacs session.

-- 
David Kastrup

Re: Speed of keyboard macro execution?

[Perry E. Metzger]

On Thu, 10 Dec 2015 21:36:14 +0100 David Kastrup <dak@gnu.org> wrote:
> Visual positioning makes no sense without visual feedback.  Can you
> show a single example of a task to be done by a keyboard macro
> where visual positioning would _help_ achieve that task rather than
> sabotage it?

Probably not. This is one reason I turn off visual positioning at all
times -- I use keyboard macros heavily and find that visual
positioning interferes with my workflow. However, this is how keyboard
macros work -- they record what the user would have typed -- and it
would violate expectations for that to change.

Regardless, I think that this isn't going to fix the underlying
performance problems.

Perry
-- 
Perry E. Metzger		perry@piermont.com

Re: Speed of keyboard macro execution?

[John Wiegley]

>>>>> Perry E Metzger <perry@piermont.com> writes:

> However, this is how keyboard macros work -- they record what the user would
> have typed -- and it would violate expectations for that to change.

Precisely. If visual positioning influences what a person would type, it
should influence the macros they record equally. So if one wants better
consistency, it's up to the user to disable visual positioning -- rather than
make the possibility of such macros unavailable to everyone.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Speed of keyboard macro execution?

[Eli Zaretskii]

> From: David Kastrup <dak@gnu.org>
> Date: Thu, 10 Dec 2015 21:36:14 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> John Wiegley <jwiegley@gmail.com> writes:
> 
> >>>>>> Perry E Metzger <perry@piermont.com> writes:
> >
> >> On Thu, 10 Dec 2015 22:00:05 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
> >>> > From: David Kastrup <dak@gnu.org>
> >>> > So I think it would make excellent sense to disable visual
> >>> > positioning modes while recording and replaying keyboard macros.
> >>> 
> >>> No, I think you are wrong.  Think C-n and C-p again.
> >
> > Recording a macro has the value that it does what the user would have
> > done, just automated. If you disable visual positioning, then in many
> > cases it no longer does what I would have done manually.
> 
> Visual positioning makes no sense without visual feedback.  Can you show
> a single example of a task to be done by a keyboard macro where visual
> positioning would _help_ achieve that task rather than sabotage it?

Even if I agree with you, it doesn't help to fix the _interactive_ use
case of pressing C-p.  With very long lines, that can take an
eternity.  _That_ is what we should strive to fix.


> 
> Just one?  For your argument's sake?  I certainly get the ugliness of
> changing a setting during macro record/replay.  No question about that.
> But the alternative is keyboard macros being essentially useless because
> their effects on the text become unpredictable.
> 
> Yes, they are then perfectly equivalent to pressing the same keys
> outside of macro recording/replay, but since the pressing of the keys is
> done based on visual feedback, there is just no point in recording key
> presses during visual movement mode and replay them at some other buffer
> position, or even the same buffer position but with different visuals.
> 
> In this case, I prefer "wrong" to "useless".
> 
> -- 
> David Kastrup
> 
> 

Re: Speed of keyboard macro execution?

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

> Even if I agree with you, it doesn't help to fix the _interactive_ use case
> of pressing C-p. With very long lines, that can take an eternity. _That_ is
> what we should strive to fix.

+1

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Speed of keyboard macro execution?

[David Kastrup]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: David Kastrup <dak@gnu.org>
>> Date: Thu, 10 Dec 2015 21:36:14 +0100
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>> 
>> John Wiegley <jwiegley@gmail.com> writes:
>> 
>> >>>>>> Perry E Metzger <perry@piermont.com> writes:
>> >
>> >> On Thu, 10 Dec 2015 22:00:05 +0200 Eli Zaretskii <eliz@gnu.org> wrote:
>> >>> > From: David Kastrup <dak@gnu.org>
>> >>> > So I think it would make excellent sense to disable visual
>> >>> > positioning modes while recording and replaying keyboard macros.
>> >>> 
>> >>> No, I think you are wrong.  Think C-n and C-p again.
>> >
>> > Recording a macro has the value that it does what the user would have
>> > done, just automated. If you disable visual positioning, then in many
>> > cases it no longer does what I would have done manually.
>> 
>> Visual positioning makes no sense without visual feedback.  Can you show
>> a single example of a task to be done by a keyboard macro where visual
>> positioning would _help_ achieve that task rather than sabotage it?
>
> Even if I agree with you, it doesn't help to fix the _interactive_ use
> case of pressing C-p.  With very long lines, that can take an
> eternity.  _That_ is what we should strive to fix.

No argument here.  My proposal changing the keyboard macro execution to
switch off visual positioning was not intended as a performance fix: it
might have that effect on some benchmarks, but that's purely incidental.

Stuff like fill-paragraph will still require display engine calculations
and I'm fine with that.  Its purpose is not in any manner different
during macro recording/replay so it's definitely both useful and
consistent to consult the display engine.

With regard to visual movement, I don't see that consistency will ever
be useful in macro recording/replay.  So I'd at least like to have an
option (effective during both macro recording and replay since a
difference between the two would be both inconsistent and useless) to
get inconsistent but useful behavior here.

-- 
David Kastrup

Re: Speed of keyboard macro execution?

[Eli Zaretskii]

> Date: Thu, 10 Dec 2015 12:00:51 -0500
> From: "Perry E. Metzger" <perry@piermont.com>
> Cc: deng@randomsample.de, emacs-devel@gnu.org
> 
> > It's a bit disingenuous on the part of that person to compare Emacs
> > with another editor _solely_ where there's a known inefficiency in
> > Emacs, and pretend that this is somehow representative of the
> > general differences in performance between these two editors.
> > After all, how frequently do you need to edit files with lines in
> > excess of 11K characters?
> 
> I don't know if it was disingenuous or not, but I think that being
> able to improve performance would not be a bad thing.

Of course.  We agree.  This particular problem doesn't have a solution
yet, but if someone has ideas, feel free to describe them and submit
patches.

> I've noticed that a lot of things (like keyboard macros) are often
> slower than expected in modern Emacs.

Please file bugs for those, with recipes to reproduce the slowness,
and I assure you they will be looked at.

> We may, as a developer community, have allowed the speed of modern
> hardware to lull us into complacency on issues like
> performance. Emacs should be the best editor it can be.

I agree.  The issue is how to solve those problems, not whether we
want to or should.

> I suppose this ties back in to earlier discussions about automated
> testing (as, among other things, automated testing can find
> performance regressions before they matter.)

The particular issue with long lines doesn't need any test suite to
reveal it.  It's a known issue, ever since Emacs 21.1 was developed;
it's not a regression that somehow slipped into Emacs.  I filed bug
#13675 for it to make a statement: we don't want to live with this
problem, we want to solve it.  Ideas for how are welcome.

Thanks.

Re: Speed of keyboard macro execution?

[John Wiegley]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

>> We may, as a developer community, have allowed the speed of modern hardware
>> to lull us into complacency on issues like performance. Emacs should be the
>> best editor it can be.

> I agree. The issue is how to solve those problems, not whether we want to or
> should.

Yes, and one of the best ways to begin is principled and consistent
measurement.

-- 
John Wiegley                  GPG fingerprint = 4710 CF98 AF9B 327B B80F
http://newartisans.com                          60E1 46C4 BD1A 7AC1 4BA2

Re: Speed of keyboard macro execution?

[Eli Zaretskii]

> From: John Wiegley <jwiegley@gmail.com>
> Cc: "Perry E. Metzger" <perry@piermont.com>,  deng@randomsample.de,  emacs-devel@gnu.org
> Date: Thu, 10 Dec 2015 09:43:19 -0800
> 
> >>>>> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> We may, as a developer community, have allowed the speed of modern hardware
> >> to lull us into complacency on issues like performance. Emacs should be the
> >> best editor it can be.
> 
> > I agree. The issue is how to solve those problems, not whether we want to or
> > should.
> 
> Yes, and one of the best ways to begin is principled and consistent
> measurement.

No argument here.  Volunteers are welcome.

Re: Speed of keyboard macro execution?

[Achim Gratz]

Eli Zaretskii writes:
> After all, how frequently do you need to edit files with lines in
> excess of 11K characters?

If editing includes just opening the file and navigating around most of
the time, then the answer to that question for me is "daily".  And the
lines are in fact a lot longer than 11k characters also, being tabulated
data with a few thousand columns, each 20 characters wide typically.  I
can guarantee that there are no multi-byte or non-printing characters in
those files if that would make a difference.


Regards,
Achim.
-- 
+<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+

Wavetables for the Waldorf Blofeld:
http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables

Re: Speed of keyboard macro execution?

[Eli Zaretskii]

> From: Achim Gratz <Stromeko@nexgo.de>
> Date: Thu, 10 Dec 2015 19:15:53 +0100
> 
> Eli Zaretskii writes:
> > After all, how frequently do you need to edit files with lines in
> > excess of 11K characters?
> 
> If editing includes just opening the file and navigating around most of
> the time, then the answer to that question for me is "daily".

Yes, some of us do that a lot (myself included).  But not most of us.

> I can guarantee that there are no multi-byte or non-printing
> characters in those files if that would make a difference.

No, it doesn't help.  Detecting a pure-ASCII buffer is very easy, so
if this would help, we would be using it long ago.

The main problem, in a nutshell, is that to get to the next visual
line, you need to traverse all the characters of the preceding line.
Going backward (which happens every time you type C-p, and also many
times behind the scenes, e.g., when Emacs needs to determine where to
set the window start for next redisplay) is even more problematic,
since Emacs can currently only do that by starting from the beginning
of the previous _physical_ line (there's an optimization in this
regard for preposterously long lines, but it loses accuracy -- it
could cause horizontal scrolling when you move point vertically).

Re: Speed of keyboard macro execution?

[Joseph Mingrone]

[-- Attachment #1: Type: text/plain, Size: 1422 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:
> It's a bit disingenuous on the part of that person to compare Emacs
> with another editor _solely_ where there's a known inefficiency in
> Emacs, and pretend that this is somehow representative of the general
> differences in performance between these two editors.  After all, how
> frequently do you need to edit files with lines in excess of 11K
> characters?

The intention wasn't to be misleading or trash Emacs, but to highlight
an area ripe for improvement.  I also assumed it was common knowledge
for anyone that would be interested in such an esoteric topic that mg
is a shell of Emacs in terms of features, so the comparison is
"contrived".

Below is a comment I added to the video shortly after it was posted.

Following the discussions on Reddit [1] and more testing, it's clear
that the major difference in speed was a result of certain operations
within the macros than the macros themselves.

emacs -nw -Q:

Slow:
M-: (while (< (point) (point-max)) (end-of-line) (insert ",")
(next-line) (beginning-of-line nil))

Within a second or two:
Define a macro with M-x end-of-line / , / C-f and run it 3869 times.

Instantaneous:
M-: (while (< (point) (point-max)) (end-of-line) (insert ",")
(forward-line 1))
M-: (replace-regexp "$" ",")

[1] https://www.reddit.com/r/emacs/comments/3vp583/macros_in_gnu_emacs_and_mg_speed_comparison/


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Speed of keyboard macro execution?

[Eli Zaretskii]

> From: Joseph Mingrone <jrm@ftfl.ca>
> Date: Fri, 11 Dec 2015 22:14:28 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > It's a bit disingenuous on the part of that person to compare Emacs
> > with another editor _solely_ where there's a known inefficiency in
> > Emacs, and pretend that this is somehow representative of the general
> > differences in performance between these two editors.  After all, how
> > frequently do you need to edit files with lines in excess of 11K
> > characters?
> 
> The intention wasn't to be misleading or trash Emacs, but to highlight
> an area ripe for improvement.

That's not what the text there says.  It says that mg is simpler, but
faster, and gives this particular example of how it is faster.  The
impression that a naïve observer will end up after that is that Emacs
is simply unworkable, since it cannot even scroll through a file in
some reasonably short time.

A more honest comparison would be to show scrolling through some
source file (with "normal" line lengths), which is something any Emacs
user does every day many times.  I'm guessing that Emacs will be
slower there as well, but instead of showing how it takes Emacs an
infinite time to do something, such an example would provide a much
more realistic comparison of relative speed between the two.

> I also assumed it was common knowledge for anyone that would be
> interested in such an esoteric topic that mg is a shell of Emacs in
> terms of features, so the comparison is "contrived".

That's exactly my point: it's so contrived that it loses credibility
in the eyes of anyone who knows a little about Emacs.

> Below is a comment I added to the video shortly after it was posted.

No amount of commentary will make that video less lopsided.  A fair
comparison should simply compare other use cases, that's all.

Re: Speed of keyboard macro execution?

[Joseph Mingrone]

[-- Attachment #1: Type: text/plain, Size: 623 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:
> After all, how frequently do you need to edit files with lines in
> excess of 11K characters?

The lines in the file are 1454 characters long and there are 3870 of
them.

Eli Zaretskii <eliz@gnu.org> writes:
> That's not what the text there says.  It says that mg is simpler, but
> faster, and gives this particular example of how it is faster.  The
> impression that a naïve observer will end up after that is that Emacs
> is simply unworkable, since it cannot even scroll through a file in
> some reasonably short time.

I neither wrote nor said that mg was faster.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]

Re: Speed of keyboard macro execution?

[Eli Zaretskii]

> From: Joseph Mingrone <jrm@ftfl.ca>
> Cc: Eli Zaretskii <eliz@gnu.org>
> Date: Sat, 12 Dec 2015 13:28:52 -0400
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> > After all, how frequently do you need to edit files with lines in
> > excess of 11K characters?
> 
> The lines in the file are 1454 characters long and there are 3870 of
> them.

The text said 1454 "cells", so I multiplied that by the number of
characters in each number (8).  Apologies if I misunderstood, but the
demo didn't really supply enough info to be sure.

> Eli Zaretskii <eliz@gnu.org> writes:
> > That's not what the text there says.  It says that mg is simpler, but
> > faster, and gives this particular example of how it is faster.  The
> > impression that a naïve observer will end up after that is that Emacs
> > is simply unworkable, since it cannot even scroll through a file in
> > some reasonably short time.
> 
> I neither wrote nor said that mg was faster.

Really?  The demo says it implements only part of the features of GNU
Emacs, but in return i is "more efficient and fast".  If that doesn't
mean mg is faster, then what does it mean?

Re: Speed of keyboard macro execution?

[Joseph Mingrone]

[-- Attachment #1: Type: text/plain, Size: 1309 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:
> The text said 1454 "cells", so I multiplied that by the number of
> characters in each number (8).  Apologies if I misunderstood, but the
> demo didn't really supply enough info to be sure.

I say, "...there are just over 1450 columns."

>> Eli Zaretskii <eliz@gnu.org> writes:
>> > That's not what the text there says.  It says that mg is simpler, but
>> > faster, and gives this particular example of how it is faster.  The
>> > impression that a naïve observer will end up after that is that Emacs
>> > is simply unworkable, since it cannot even scroll through a file in
>> > some reasonably short time.

>> I neither wrote nor said that mg was faster.

> Really?  The demo says it implements only part of the features of GNU
> Emacs, but in return i is "more efficient and fast".  If that doesn't
> mean mg is faster, then what does it mean?

I say, "It only implements a subset of the features of GNU Emacs, but
in return it's quite efficient and fast."  In retrospect I do wish I
had been clearer and elaborated on the "contrived" part of the test.

Again, the goal was to promote discussion, and in the end, hopefully
improve Emacs.  I will take the video down at some point.  I also
don't want to give new users the wrong impression.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 818 bytes --]