scratch/accurate-warning-pos: First tentative successes.

scratch/accurate-warning-pos: First tentative successes.

[Alan Mackenzie]

Hello, Eli and Stefan.

I have just committed another commit to scratch/accurate-warning-pos.  It
now "works"!

To see this, create one or both of these test files:

(i) winkler.el, from bug #9109:

    (unwind-protect
        (let ((foo "foo"))
          (insert foo))
      (setq foo "bar"))

(ii) pit-claudel.el, from bug #22288:

    (defun test ()
      (let (a))
        a)

Then build the SW, start it (with or without -Q) and run the byte
compiler with the current mechanism for warnings:

    M-: (byte-compile-file "~/winkler.el")

.  Then run the byte compiler again, with the new mechanism:

    M-: (let ((symbols-with-pos-enabled t)) (byte-compile-file "~/winkler.el"))

.  Note the positions in the warning messages.

Eli, in bug #24128, you complained that in compiling progmodes/gud-mi.el
and progmodes/gud.el, you got warning message positions which were
clearly wrong.  Would it be possible to reconstruct the versions of those
files which gave these errors?  It would be great to be able to try them
out on this new software.

However...

This approach to fixing the bug has disadvantages, namely:
(i) The meanings of `eq' and `symbolp' have been modified.
(ii) It has slowed Emacs down somewhat.  (By about 8% in an unoptimised
build; I haven't yet timed an optimsed build).

On the other hand...

Given the difficulty of the problem, there is probably no way of solving
it without such disadvantages.  We've searched for and explored several
alternatives, and none of them were good.  And the approach in
scratch/accurate-warning-pos DOES solve all these bugs, not being in any
sense a workaround.  There might well be optimisations which could reduce
the slowdown.

So....

Are we going to carry on developing this thing, with a view to
integrating it into Emacs, or are we not?  It would be good to decide
this before spending much more time on it.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: First tentative successes.

[Eli Zaretskii]

> Date: Sat, 17 Nov 2018 12:45:34 +0000
> Cc: Michael Heerdegen <michael_heerdegen@web.de>,
>   Clément Pit-Claudel <cpitclaudel@gmail.com>,
>   emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> Eli, in bug #24128, you complained that in compiling progmodes/gud-mi.el
> and progmodes/gud.el, you got warning message positions which were
> clearly wrong.  Would it be possible to reconstruct the versions of those
> files which gave these errors?  It would be great to be able to try them
> out on this new software.

The current version of gdb-mi.el still emits the warning.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Eli.

On Sat, Nov 17, 2018 at 15:13:51 +0200, Eli Zaretskii wrote:
> > Date: Sat, 17 Nov 2018 12:45:34 +0000
> > Cc: Michael Heerdegen <michael_heerdegen@web.de>,
> >   Clément Pit-Claudel <cpitclaudel@gmail.com>,
> >   emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > Eli, in bug #24128, you complained that in compiling progmodes/gud-mi.el
> > and progmodes/gud.el, you got warning message positions which were
> > clearly wrong.  Would it be possible to reconstruct the versions of those
> > files which gave these errors?  It would be great to be able to try them
> > out on this new software.

> The current version of gdb-mi.el still emits the warning.

I didn't see this, but never mind.

The branch scratch/accurate-warning-pos now bootstraps.  (It also now
runs on optimised builds. ;-)

Its bootstrap is a mere 7.1% slower than master's.

Currently, warning messages contain two source positions, that of the
traditional scheme, and of the new one.  A typical message currently
looks like this:

    cedet/srecode/find.el:248:57:247:35:Warning: Unknown slot `:application'

.  The old position was line 248 column 57, the new L247 C35.

The question arises, how different are the new positions from the old?
I saved the stderr from bootstrap to a file, and fed it through the
following AWK filter:

#########################################################################
#! /usr/bin/gawk -f
BEGIN { FS = ":" }

NF >= 6 {
    if (($2 == $4) && ($3 == $5))
        same++
    else {
        different++
        printf("%d: ", NR)
        print $0
    }
}

END {
    print
    printf("same: %d\n", same)
    printf("different: %d\n", different)
}
#########################################################################

.  There were 81 positions the same (old and new), there were 254
different.  A large part of these 254 different were "Warning: Unknown
slot".  In these warnings, the new position is correct, the old one is
incorrect.  In fact, of all the differences I've looked at, the new one
is always the correct one.

Thus, this branch works, and the idea behind it is sound.

There are one or two nastinesses in it: principally, I had to duplicate
the function byte-compile-strip-symbol-positions as
cl-macs-strip-symbol-positions to get the thing to bootstrap.  There is
surely a straightforward solution to this, but it escapes me at the
moment.

Although the branch works, it has been tested solely on 64-bit
GNU/Linux, the only system I have.  This despite some fairly hefty
changes in lisp.h.

So, where do we go from here?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Charles A. Roelli]

> Date: Fri, 23 Nov 2018 13:09:05 +0000
> From: Alan Mackenzie <acm@muc.de>
> 
> Although the branch works, it has been tested solely on 64-bit
> GNU/Linux, the only system I have.  This despite some fairly hefty
> changes in lisp.h.

"make bootstrap" prints the following errors here on the
scratch/accurate-warning-pos branch:

  CC       dispnew.o
In file included from dispnew.c:27:
lisp.h: In function ‘SYMBOLP’:
lisp.h:1052: error: used struct type value where scalar is required
lisp.h: In function ‘XSYMBOL’:
lisp.h:1083: error: wrong type argument to unary exclamation mark
lisp.h: In function ‘EQ’:
lisp.h:1233: error: used struct type value where scalar is required
lisp.h:1233: error: invalid operands to binary == (have ‘Lisp_Object’ and ‘Lisp_Object’)
lisp.h:1233: error: invalid operands to binary == (have ‘Lisp_Object’ and ‘Lisp_Object’)

(This is macOS with GCC 4.2.1.)

Seems like the compiler is raising errors when the var
"Vsymbols_with_pos_enabled" (a Lisp_Object) is used in a true/false
context.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Charles.

Thanks for trying this out!

On Sun, Nov 25, 2018 at 12:26:45 +0100, Charles A. Roelli wrote:
> > Date: Fri, 23 Nov 2018 13:09:05 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > 
> > Although the branch works, it has been tested solely on 64-bit
> > GNU/Linux, the only system I have.  This despite some fairly hefty
> > changes in lisp.h.

> "make bootstrap" prints the following errors here on the
> scratch/accurate-warning-pos branch:

>   CC       dispnew.o
> In file included from dispnew.c:27:
> lisp.h: In function ‘SYMBOLP’:
> lisp.h:1052: error: used struct type value where scalar is required
> lisp.h: In function ‘XSYMBOL’:
> lisp.h:1083: error: wrong type argument to unary exclamation mark
> lisp.h: In function ‘EQ’:
> lisp.h:1233: error: used struct type value where scalar is required
> lisp.h:1233: error: invalid operands to binary == (have ‘Lisp_Object’ and ‘Lisp_Object’)
> lisp.h:1233: error: invalid operands to binary == (have ‘Lisp_Object’ and ‘Lisp_Object’)

> (This is macOS with GCC 4.2.1.)

> Seems like the compiler is raising errors when the var
> "Vsymbols_with_pos_enabled" (a Lisp_Object) is used in a true/false
> context.

Yes, indeed.

I hope the following patch will fix it, and hope also there won't be too
many similar bugs to fix.



diff --git a/src/lisp.h b/src/lisp.h
index 4dfd065629..802c9e4259 100644
--- a/src/lisp.h
+++ b/src/lisp.h
@@ -395,7 +395,7 @@ typedef EMACS_INT Lisp_Word;
 
 /* verify (NIL_IS_ZERO) */
 #define lisp_h_EQ(x, y) ((XLI ((x)) == XLI ((y)))       \
-  || (Vsymbols_with_pos_enabled  \
+  || (XLI (Vsymbols_with_pos_enabled)    \
   && (SYMBOL_WITH_POS_P ((x))                        \
       ? BARE_SYMBOL_P ((y))                               \
         ? (XSYMBOL_WITH_POS((x)))->sym == (y)          \
@@ -424,7 +424,7 @@ typedef EMACS_INT Lisp_Word;
 #define lisp_h_BARE_SYMBOL_P(x) TAGGEDP ((x), Lisp_Symbol)
 /* verify (NIL_IS_ZERO) */
 #define lisp_h_SYMBOLP(x) ((BARE_SYMBOL_P ((x)) ||               \
-                            (Vsymbols_with_pos_enabled && (SYMBOL_WITH_POS_P ((x))))))
+                            (XLI (Vsymbols_with_pos_enabled) && (SYMBOL_WITH_POS_P ((x))))))
 #define lisp_h_TAGGEDP(a, tag) \
    (! (((unsigned) (XLI (a) >> (USE_LSB_TAG ? 0 : VALBITS)) \
 	- (unsigned) (tag)) \
@@ -463,7 +463,7 @@ typedef EMACS_INT Lisp_Word;
 /* verify (NIL_IS_ZERO) */
 # define lisp_h_XSYMBOL(a)                      \
      (eassert (SYMBOLP ((a))),                      \
-      (!Vsymbols_with_pos_enabled                \
+      (!XLI (Vsymbols_with_pos_enabled)             \
       ? (XBARE_SYMBOL ((a)))             \
        : (BARE_SYMBOL_P ((a)))           \
       ? (XBARE_SYMBOL ((a)))                                    \


-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Andreas Schwab]

On Nov 25 2018, Alan Mackenzie <acm@muc.de> wrote:

> diff --git a/src/lisp.h b/src/lisp.h
> index 4dfd065629..802c9e4259 100644
> --- a/src/lisp.h
> +++ b/src/lisp.h
> @@ -395,7 +395,7 @@ typedef EMACS_INT Lisp_Word;
>  
>  /* verify (NIL_IS_ZERO) */
>  #define lisp_h_EQ(x, y) ((XLI ((x)) == XLI ((y)))       \
> -  || (Vsymbols_with_pos_enabled  \
> +  || (XLI (Vsymbols_with_pos_enabled)    \

You should make that a DEFVAR_BOOL.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Andreas.

On Sun, Nov 25, 2018 at 16:12:57 +0100, Andreas Schwab wrote:
> On Nov 25 2018, Alan Mackenzie <acm@muc.de> wrote:

> > diff --git a/src/lisp.h b/src/lisp.h
> > index 4dfd065629..802c9e4259 100644
> > --- a/src/lisp.h
> > +++ b/src/lisp.h
> > @@ -395,7 +395,7 @@ typedef EMACS_INT Lisp_Word;
> >  
> >  /* verify (NIL_IS_ZERO) */
> >  #define lisp_h_EQ(x, y) ((XLI ((x)) == XLI ((y)))       \
> > -  || (Vsymbols_with_pos_enabled  \
> > +  || (XLI (Vsymbols_with_pos_enabled)    \

> You should make that a DEFVAR_BOOL.

I'm not sure.  If I understand correctly, DEFVAR_BOOLS are bitfields in
integers, and they are slow to unpack and test.  Or am I wrong, here?
Even if it's not the case now, it might be in the future.

The idea behind symbols-with-pos-enabled is that it will be a 32/64 bit
integer permanently in cache, hence can be tested for zero/non-zero
rapidly.

Even so, the mechanism has slowed Emacs down by seven or eight per cent.

> Andreas.

> -- 
> Andreas Schwab, schwab@linux-m68k.org
> GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
> "And now for something completely different."

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Stefan Monnier]

> -  || (Vsymbols_with_pos_enabled  \
> +  || (XLI (Vsymbols_with_pos_enabled)    \

Either use NILP here rather than XLI, or change
Vsymbols_with_pos_enabled to type `bool` rather than Lisp_Object as
Andreas suggests.


        Stefan

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Eli Zaretskii]

> Date: Sun, 25 Nov 2018 14:31:25 +0000
> Cc: michael_heerdegen@web.de, eliz@gnu.org, emacs-devel@gnu.org,
>   cpitclaudel@gmail.com, monnier@IRO.UMontreal.CA
> From: Alan Mackenzie <acm@muc.de>
> 
> -  || (Vsymbols_with_pos_enabled  \
> +  || (XLI (Vsymbols_with_pos_enabled)    \


??? Why not NILP instead?  Please don't use XLI for testing a Lisp
variable, it isn't clean.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Eli Zaretskii]

> Date: Sun, 25 Nov 2018 15:42:03 +0000
> Cc: "Charles A. Roelli" <charles@aurox.ch>, michael_heerdegen@web.de,
>   eliz@gnu.org, cpitclaudel@gmail.com, monnier@IRO.UMontreal.CA,
>   emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> The idea behind symbols-with-pos-enabled is that it will be a 32/64 bit
> integer permanently in cache, hence can be tested for zero/non-zero
> rapidly.

This is premature optimization, we have gazillions of such tests in
our inner loops (e.g., see bidi.c), one more or one less shouldn't
matter.

> Even so, the mechanism has slowed Emacs down by seven or eight per cent.

I'd be very surprised if using a boolean Lisp variable would slow that
down more.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Andreas Schwab]

On Nov 25 2018, Eli Zaretskii <eliz@gnu.org> wrote:

>> Date: Sun, 25 Nov 2018 14:31:25 +0000
>> Cc: michael_heerdegen@web.de, eliz@gnu.org, emacs-devel@gnu.org,
>>   cpitclaudel@gmail.com, monnier@IRO.UMontreal.CA
>> From: Alan Mackenzie <acm@muc.de>
>> 
>> -  || (Vsymbols_with_pos_enabled  \
>> +  || (XLI (Vsymbols_with_pos_enabled)    \
>
>
> ??? Why not NILP instead?

You can't use NILP inside the definition of NILP.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Eli.

On Sun, Nov 25, 2018 at 18:38:16 +0200, Eli Zaretskii wrote:
> > Date: Sun, 25 Nov 2018 14:31:25 +0000
> > Cc: michael_heerdegen@web.de, eliz@gnu.org, emacs-devel@gnu.org,
> >   cpitclaudel@gmail.com, monnier@IRO.UMontreal.CA
> > From: Alan Mackenzie <acm@muc.de>

> > -  || (Vsymbols_with_pos_enabled  \
> > +  || (XLI (Vsymbols_with_pos_enabled)    \


> ??? Why not NILP instead?  Please don't use XLI for testing a Lisp
> variable, it isn't clean.

Well the current definition of EQ in master uses XLI on both sides of
the == operator.  So I'm not doing anything new here.

I tried to make NILP work for about two hours this morning, without
success.

The problem is circular dependencies between the macros and the inline
functions in lisp.h.  NILP uses EQ, and EQ tests
Vsymbols_with_pos_enabled.  So we can't use NILP for this second test.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Stefan.

On Sun, Nov 25, 2018 at 11:22:00 -0500, Stefan Monnier wrote:
> > -  || (Vsymbols_with_pos_enabled  \
> > +  || (XLI (Vsymbols_with_pos_enabled)    \

> Either use NILP here rather than XLI, or change
> Vsymbols_with_pos_enabled to type `bool` rather than Lisp_Object as
> Andreas suggests.

symbols-with-pos-enabled gets bound to t by Lisp code (that's its point).
So Vsymbols_with_pos_enabled has to be a Lisp_Object of some sort.

NILP won't work here (as I pointed out to Eli), because of circular
dependencies inside lisp.h.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Eli Zaretskii]

> Date: Sun, 25 Nov 2018 17:31:24 +0000
> Cc: charles@aurox.ch, michael_heerdegen@web.de, emacs-devel@gnu.org,
>   cpitclaudel@gmail.com, monnier@IRO.UMontreal.CA
> From: Alan Mackenzie <acm@muc.de>
> 
> I tried to make NILP work for about two hours this morning, without
> success.
> 
> The problem is circular dependencies between the macros and the inline
> functions in lisp.h.  NILP uses EQ, and EQ tests
> Vsymbols_with_pos_enabled.  So we can't use NILP for this second test.

Than make that variable a boolean, and test the C equivalents for
being non-zero.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Eli.

On Sun, Nov 25, 2018 at 18:40:54 +0200, Eli Zaretskii wrote:
> > Date: Sun, 25 Nov 2018 15:42:03 +0000
> > Cc: "Charles A. Roelli" <charles@aurox.ch>, michael_heerdegen@web.de,
> >   eliz@gnu.org, cpitclaudel@gmail.com, monnier@IRO.UMontreal.CA,
> >   emacs-devel@gnu.org
> > From: Alan Mackenzie <acm@muc.de>

> > The idea behind symbols-with-pos-enabled is that it will be a 32/64 bit
> > integer permanently in cache, hence can be tested for zero/non-zero
> > rapidly.

> This is premature optimization, we have gazillions of such tests in
> our inner loops (e.g., see bidi.c), one more or one less shouldn't
> matter.

It's a straightforward way of writing it, without using any contorted
code, so even if it is optimisation, it's harmless optimisation.

But right from the beginning of the exercise, I've had speed at the back
of my mind.  If the slowdown had been 30%, the Emacs project leadership
would have been very unhappy about the idea.

My current trouble arises from the need to access the variable both from
C and from Lisp.

> > Even so, the mechanism has slowed Emacs down by seven or eight per cent.

> I'd be very surprised if using a boolean Lisp variable would slow that
> down more.

I'm still a little confused as to what a DEFVAR_BOOL is.  Is it a full
Lisp_Object, of which only one bit is used, or is it one of many
DEFVAR_BOOLs packed into a single word?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

On Sun, Nov 25, 2018 at 19:39:46 +0200, Eli Zaretskii wrote:
> > Date: Sun, 25 Nov 2018 17:31:24 +0000
> > Cc: charles@aurox.ch, michael_heerdegen@web.de, emacs-devel@gnu.org,
> >   cpitclaudel@gmail.com, monnier@IRO.UMontreal.CA
> > From: Alan Mackenzie <acm@muc.de>

> > I tried to make NILP work for about two hours this morning, without
> > success.

> > The problem is circular dependencies between the macros and the inline
> > functions in lisp.h.  NILP uses EQ, and EQ tests
> > Vsymbols_with_pos_enabled.  So we can't use NILP for this second test.

> Than make that variable a boolean, and test the C equivalents for
> being non-zero.

The variable has to be accessible from Lisp.  If I declare it with
DEFVAR_BOOL, would that make it a bool for C code?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Eli Zaretskii]

> Date: Sun, 25 Nov 2018 17:59:29 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: cpitclaudel@gmail.com, charles@aurox.ch, michael_heerdegen@web.de,
> 	emacs-devel@gnu.org, schwab@linux-m68k.org, monnier@IRO.UMontreal.CA
> 
> > > The idea behind symbols-with-pos-enabled is that it will be a 32/64 bit
> > > integer permanently in cache, hence can be tested for zero/non-zero
> > > rapidly.
> 
> > This is premature optimization, we have gazillions of such tests in
> > our inner loops (e.g., see bidi.c), one more or one less shouldn't
> > matter.
> 
> It's a straightforward way of writing it, without using any contorted
> code, so even if it is optimisation, it's harmless optimisation.

The premature optimisation here is to worry about performance without
timing the code.

> But right from the beginning of the exercise, I've had speed at the back
> of my mind.  If the slowdown had been 30%, the Emacs project leadership
> would have been very unhappy about the idea.

As I said, I'd be very surprised if using DEFVAR_BOOL would slow down
what you did in any significant way, beyond the slowdown due to the
comparison and struct member access.

> My current trouble arises from the need to access the variable both from
> C and from Lisp.

DEFVAR_BOOL defines a variable accessible both from C and Lisp.

> I'm still a little confused as to what a DEFVAR_BOOL is.

It's a C 'bool'.  Look at defvar_bool the function: its last argument
is a pointer to the C counterpart of a DEFVAR_BOOL.

You can also use ptype in the debugger to see what it really is, if
you want to be absolutely sure.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Stefan Monnier]

> symbols-with-pos-enabled gets bound to t by Lisp code (that's its point).
> So Vsymbols_with_pos_enabled has to be a Lisp_Object of some sort.

No, it doesn't.  If you use DEFVAR_BOOL, then the underlying C var will
be of type `bool` but it's still exported as a normal Elisp variable
that can be `let` bound like any other (except that any non-nil value
will result in the variable taking value t).

> NILP won't work here (as I pointed out to Eli), because of circular
> dependencies inside lisp.h.

Oh, right:

    #define lisp_h_NILP(x) EQ (x, Qnil)

Reminds me of another question I had: in your code, do you need NILP to
pay attention to symbols-with-pos-enabled?

That would significantly increase the impact in terms of code-size and
CPU time (could explain why you see 8% slowdown which is more expansive
than my patch which turns EQ into EQL).


        Stefan

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Eli.

On Sun, Nov 25, 2018 at 20:15:30 +0200, Eli Zaretskii wrote:
> > Date: Sun, 25 Nov 2018 17:59:29 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: cpitclaudel@gmail.com, charles@aurox.ch, michael_heerdegen@web.de,
> > 	emacs-devel@gnu.org, schwab@linux-m68k.org, monnier@IRO.UMontreal.CA

> > > > The idea behind symbols-with-pos-enabled is that it will be a 32/64 bit
> > > > integer permanently in cache, hence can be tested for zero/non-zero
> > > > rapidly.

> > > This is premature optimization, we have gazillions of such tests in
> > > our inner loops (e.g., see bidi.c), one more or one less shouldn't
> > > matter.

> > It's a straightforward way of writing it, without using any contorted
> > code, so even if it is optimisation, it's harmless optimisation.

> The premature optimisation here is to worry about performance without
> timing the code.

> > But right from the beginning of the exercise, I've had speed at the back
> > of my mind.  If the slowdown had been 30%, the Emacs project leadership
> > would have been very unhappy about the idea.

> As I said, I'd be very surprised if using DEFVAR_BOOL would slow down
> what you did in any significant way, beyond the slowdown due to the
> comparison and struct member access.

I'll time it, then we'll know.

> > My current trouble arises from the need to access the variable both from
> > C and from Lisp.

> DEFVAR_BOOL defines a variable accessible both from C and Lisp.

> > I'm still a little confused as to what a DEFVAR_BOOL is.

> It's a C 'bool'.  Look at defvar_bool the function: its last argument
> is a pointer to the C counterpart of a DEFVAR_BOOL.

> You can also use ptype in the debugger to see what it really is, if
> you want to be absolutely sure.

OK, thanks for the explanation.  I'll change the declaration of that
variable (and another one) to DEFVAR_BOOL, and tidy up lisp.h to match.

Then I'll time the resulting Emacs again.  :-)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

I didn't realize until now that this branch makes EQ slower for everybody, for 
what seems to be a relatively minor issue (i.e., more accurate diagnostics). 
Surely there is a better way.

Why not add an option to 'read' so that its returned value contains 
symbol-with-pos objects instead of symbol objects? That way, programs like the 
Elisp compiler that read objects can generate more accurate diagnostics without 
having to slow down EQ for everybody.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Gemini Lasswell]

[-- Attachment #1: Type: text/plain, Size: 1340 bytes --]

Alan Mackenzie <acm@muc.de> writes:

> The branch scratch/accurate-warning-pos now bootstraps.  (It also now
> runs on optimised builds. ;-)
>
> Its bootstrap is a mere 7.1% slower than master's.

Hi Alan,

Great progress!  There are clearly benefits to Emacs Lisp programmers
(especially if errors and debugger backtraces can someday have accurate
source positions), to be weighed against the cost of a slower Emacs for
all the non-Lisp programmers and non-programmers who work with Emacs.  I
don't know how to begin to sort that out, but since I've been working on
creating a benchmark suite for Emacs, I'm thrilled you've given me such
an interesting test case. :)

Here's how much longer in percentage terms the tasks in my nascent
benchmark suite take in your branch compared to 5b218be0c3, the commit
your branch is based on, running on my newish Linux laptop.  I'll push
my benchmarking project to a scratch branch soon so you can see the code
which was benchmarked.

cus-theme-tasks-load-tango: 9%
bytecomp-tasks-compile-doc: 24%
kmacro-tasks-edit-lines: 18%
fill-tasks-fill-paragraph: 15%
cmds-tasks-forward-line: 5%
data-tasks-prime-p: 1%
emacs-tasks-startup: 4%
syntax-tasks-backward-word: 23%
syntax-tasks-forward-word: 20%

Here's a benchmark report, which contains the measurements from which I
calculated those percentages:


[-- Attachment #2: report.org --]
[-- Type: text/plain, Size: 13969 bytes --]

#+TITLE: Emacs Benchmarks
#+OPTIONS: toc:2 num:2 author:nil
#+LATEX_HEADER: \usepackage[margin=0.5in]{geometry}
* Benchmark results
** Custom Themes
=lisp/cus-theme-tasks.el=
*** cus-theme-tasks-load-tango
Load the tango theme.
#+BEGIN_SRC gnuplot :var data=cus-theme-tasks-load-tango-1.0() :file plots/cus-theme-tasks-load-tango-1.0.png
 :noweb yesreset
set terminal png size 800, 600
set title "cus-theme-tasks-load-tango"
set xlabel "Commit"
set xrange [1459356037:1542976351]
<<xtics>>
set ylabel "Run time (seconds)"
set yrange [0:0.003452230265000001]
set key right bottom
plot data u 2:3 w lp lw 2 title 'sockeye'
#+END_SRC

*** Measurements for cus-theme-tasks-load-tango-1.0 :noexport:
#+NAME: cus-theme-tasks-load-tango-1.0
#+BEGIN_SRC emacs-lisp
'(("09462b95484ebf75899c64e8ddb7ffce50ef6ca0" 1459356037 0.0021430170333333336)
  ("33f856ba01d13f649cf5c848b322ecb0dbfc02fc" 1474022875 0.0022962552999999998)
  ("d4a97088f69eb5729261ee4581cfb7d60c673ebd" 1484271619 0.0021830009666666673)
  ("d1da8e534db54bb82b6f2b16bba86a6d202ac675" 1504966847 0.002482565333333333)
  ("0676633e8fd842554e91d0756115fa2d6b7cf69c" 1518488627 0.0025596994)
  ("e18600dfbd31316f30cf98f6d7a90aad33bc1981" 1527019548 0.002526405933333334)
  ("bd013a448b152a84cff9b18292d8272faf265447" 1539100642 0.002774463566666667)
  ("5b218be0c362316384f5c9ef57a3bef02f742e94" 1541077242 0.0028888464666666667)
  ("75b18e07e57da7ee4362db800352d6650f5f7290" 1542976351 0.0031383911500000006))
#+END_SRC

** Byte Compilation
=lisp/emacs-lisp/bytecomp-tasks.el=
*** bytecomp-tasks-compile-doc
Byte compile a function.
#+BEGIN_SRC gnuplot :var data=bytecomp-tasks-compile-doc-1.0() :file plots/bytecomp-tasks-compile-doc-1.0.png
 :noweb yesreset
set terminal png size 800, 600
set title "bytecomp-tasks-compile-doc"
set xlabel "Commit"
set xrange [1459356037:1542976351]
<<xtics>>
set ylabel "Run time (seconds)"
set yrange [0:0.0026953711399999997]
set key right bottom
plot data u 2:3 w lp lw 2 title 'sockeye'
#+END_SRC

*** Measurements for bytecomp-tasks-compile-doc-1.0 :noexport:
#+NAME: bytecomp-tasks-compile-doc-1.0
#+BEGIN_SRC emacs-lisp
'(("09462b95484ebf75899c64e8ddb7ffce50ef6ca0" 1459356037 0.0014085693666666668)
  ("33f856ba01d13f649cf5c848b322ecb0dbfc02fc" 1474022875 0.0014425310000000002)
  ("d4a97088f69eb5729261ee4581cfb7d60c673ebd" 1484271619 0.0014799697333333331)
  ("d1da8e534db54bb82b6f2b16bba86a6d202ac675" 1504966847 0.0019974256000000004)
  ("0676633e8fd842554e91d0756115fa2d6b7cf69c" 1518488627 0.0020345819333333335)
  ("e18600dfbd31316f30cf98f6d7a90aad33bc1981" 1527019548 0.0020481948999999996)
  ("bd013a448b152a84cff9b18292d8272faf265447" 1539100642 0.0019689730333333332)
  ("5b218be0c362316384f5c9ef57a3bef02f742e94" 1541077242 0.001975632566666667)
  ("75b18e07e57da7ee4362db800352d6650f5f7290" 1542976351 0.0024503373999999996))
#+END_SRC

** Enhanced Keyboard Macros
=lisp/kmacro-tasks.el=
*** kmacro-tasks-edit-lines
Edit lines of text using a keyboard macro.
#+BEGIN_SRC gnuplot :var data=kmacro-tasks-edit-lines-1.0() :file plots/kmacro-tasks-edit-lines-1.0.png
 :noweb yesreset
set terminal png size 800, 600
set title "kmacro-tasks-edit-lines"
set xlabel "Commit"
set xrange [1459356037:1542976351]
<<xtics>>
set ylabel "Run time (seconds)"
set yrange [0:0.004164748775]
set key right bottom
plot data u 2:3 w lp lw 2 title 'sockeye'
#+END_SRC

*** Measurements for kmacro-tasks-edit-lines-1.0 :noexport:
#+NAME: kmacro-tasks-edit-lines-1.0
#+BEGIN_SRC emacs-lisp
'(("09462b95484ebf75899c64e8ddb7ffce50ef6ca0" 1459356037 0.0031633584333333334)
  ("33f856ba01d13f649cf5c848b322ecb0dbfc02fc" 1474022875 0.0032088737000000003)
  ("d4a97088f69eb5729261ee4581cfb7d60c673ebd" 1484271619 0.0031356906333333333)
  ("d1da8e534db54bb82b6f2b16bba86a6d202ac675" 1504966847 0.0031409135666666662)
  ("0676633e8fd842554e91d0756115fa2d6b7cf69c" 1518488627 0.0031141107333333335)
  ("e18600dfbd31316f30cf98f6d7a90aad33bc1981" 1527019548 0.0031232734000000003)
  ("bd013a448b152a84cff9b18292d8272faf265447" 1539100642 0.003379109366666667)
  ("5b218be0c362316384f5c9ef57a3bef02f742e94" 1541077242 0.0032031665999999997)
  ("75b18e07e57da7ee4362db800352d6650f5f7290" 1542976351 0.0037861352500000002))
#+END_SRC

** Fill Commands
=lisp/textmodes/fill-tasks.el=
*** fill-tasks-fill-paragraph
Create a long single line paragraph and use fill-paragraph.
#+BEGIN_SRC gnuplot :var data=fill-tasks-fill-paragraph-1.0() :file plots/fill-tasks-fill-paragraph-1.0.png
 :noweb yesreset
set terminal png size 800, 600
set title "fill-tasks-fill-paragraph"
set xlabel "Commit"
set xrange [1459356037:1542976351]
<<xtics>>
set ylabel "Run time (seconds)"
set yrange [0:0.0021485162599999994]
set key right bottom
plot data u 2:3 w lp lw 2 title 'sockeye'
#+END_SRC

*** Measurements for fill-tasks-fill-paragraph-1.0 :noexport:
#+NAME: fill-tasks-fill-paragraph-1.0
#+BEGIN_SRC emacs-lisp
'(("09462b95484ebf75899c64e8ddb7ffce50ef6ca0" 1459356037 0.0016103219333333334)
  ("33f856ba01d13f649cf5c848b322ecb0dbfc02fc" 1474022875 0.0015916505999999996)
  ("d4a97088f69eb5729261ee4581cfb7d60c673ebd" 1484271619 0.0016998693666666663)
  ("d1da8e534db54bb82b6f2b16bba86a6d202ac675" 1504966847 0.0017253751666666667)
  ("0676633e8fd842554e91d0756115fa2d6b7cf69c" 1518488627 0.0016796258333333332)
  ("e18600dfbd31316f30cf98f6d7a90aad33bc1981" 1527019548 0.0017355549333333332)
  ("bd013a448b152a84cff9b18292d8272faf265447" 1539100642 0.0016574472666666669)
  ("5b218be0c362316384f5c9ef57a3bef02f742e94" 1541077242 0.0016994107333333332)
  ("75b18e07e57da7ee4362db800352d6650f5f7290" 1542976351 0.0019531965999999993))
#+END_SRC

** Simple Editing Commands
=src/cmds-tasks.el=
*** cmds-tasks-forward-line
Use forward-line to navigate through a buffer.
#+BEGIN_SRC gnuplot :var data=cmds-tasks-forward-line-1.0() :file plots/cmds-tasks-forward-line-1.0.png
 :noweb yesreset
set terminal png size 800, 600
set title "cmds-tasks-forward-line"
set xlabel "Commit"
set xrange [1459356037:1542976351]
<<xtics>>
set ylabel "Run time (seconds)"
set yrange [0:0.002669931146666667]
set key right bottom
plot data u 2:3 w lp lw 2 title 'sockeye'
#+END_SRC

*** Measurements for cmds-tasks-forward-line-1.0 :noexport:
#+NAME: cmds-tasks-forward-line-1.0
#+BEGIN_SRC emacs-lisp
'(("09462b95484ebf75899c64e8ddb7ffce50ef6ca0" 1459356037 0.002290958133333333)
  ("33f856ba01d13f649cf5c848b322ecb0dbfc02fc" 1474022875 0.0024272101333333335)
  ("d4a97088f69eb5729261ee4581cfb7d60c673ebd" 1484271619 0.0022335262666666666)
  ("d1da8e534db54bb82b6f2b16bba86a6d202ac675" 1504966847 0.0022870859)
  ("0676633e8fd842554e91d0756115fa2d6b7cf69c" 1518488627 0.002244811133333333)
  ("e18600dfbd31316f30cf98f6d7a90aad33bc1981" 1527019548 0.002371730899999999)
  ("bd013a448b152a84cff9b18292d8272faf265447" 1539100642 0.0021013971)
  ("5b218be0c362316384f5c9ef57a3bef02f742e94" 1541077242 0.0022137415666666663)
  ("75b18e07e57da7ee4362db800352d6650f5f7290" 1542976351 0.0023198416499999996))
#+END_SRC

** Primitive Operations on Lisp Data Types
=src/data-tasks.el=
*** data-tasks-prime-p
Verify that a large prime number is prime.
#+BEGIN_SRC gnuplot :var data=data-tasks-prime-p-1.0() :file plots/data-tasks-prime-p-1.0.png
 :noweb yesreset
set terminal png size 800, 600
set title "data-tasks-prime-p"
set xlabel "Commit"
set xrange [1459356037:1542976351]
<<xtics>>
set ylabel "Run time (seconds)"
set yrange [0:0.003587532355]
set key right bottom
plot data u 2:3 w lp lw 2 title 'sockeye'
#+END_SRC

*** Measurements for data-tasks-prime-p-1.0 :noexport:
#+NAME: data-tasks-prime-p-1.0
#+BEGIN_SRC emacs-lisp
'(("09462b95484ebf75899c64e8ddb7ffce50ef6ca0" 1459356037 0.003044770766666666)
  ("33f856ba01d13f649cf5c848b322ecb0dbfc02fc" 1474022875 0.0028769088333333334)
  ("d4a97088f69eb5729261ee4581cfb7d60c673ebd" 1484271619 0.002816468666666667)
  ("d1da8e534db54bb82b6f2b16bba86a6d202ac675" 1504966847 0.0028207245333333342)
  ("0676633e8fd842554e91d0756115fa2d6b7cf69c" 1518488627 0.0027013283000000003)
  ("e18600dfbd31316f30cf98f6d7a90aad33bc1981" 1527019548 0.002730308333333333)
  ("bd013a448b152a84cff9b18292d8272faf265447" 1539100642 0.0030692047000000005)
  ("5b218be0c362316384f5c9ef57a3bef02f742e94" 1541077242 0.0032145395666666673)
  ("75b18e07e57da7ee4362db800352d6650f5f7290" 1542976351 0.00326139305))
#+END_SRC

** Emacs Startup and Shutdown
=src/emacs-tasks.el=
*** emacs-tasks-startup
Start up and shut down Emacs.
#+BEGIN_SRC gnuplot :var data=emacs-tasks-startup-1.0() :file plots/emacs-tasks-startup-1.0.png
 :noweb yesreset
set terminal png size 800, 600
set title "emacs-tasks-startup"
set xlabel "Commit"
set xrange [1459356037:1542976351]
<<xtics>>
set ylabel "Run time (seconds)"
set yrange [0:0.10571710112333335]
set key right bottom
plot data u 2:3 w lp lw 2 title 'sockeye'
#+END_SRC

*** Measurements for emacs-tasks-startup-1.0 :noexport:
#+NAME: emacs-tasks-startup-1.0
#+BEGIN_SRC emacs-lisp
'(("09462b95484ebf75899c64e8ddb7ffce50ef6ca0" 1459356037 0.08497184696666667)
  ("33f856ba01d13f649cf5c848b322ecb0dbfc02fc" 1474022875 0.0853800205)
  ("d4a97088f69eb5729261ee4581cfb7d60c673ebd" 1484271619 0.0874049044)
  ("d1da8e534db54bb82b6f2b16bba86a6d202ac675" 1504966847 0.09610645556666668)
  ("0676633e8fd842554e91d0756115fa2d6b7cf69c" 1518488627 0.07799585656666667)
  ("e18600dfbd31316f30cf98f6d7a90aad33bc1981" 1527019548 0.07763947026666666)
  ("bd013a448b152a84cff9b18292d8272faf265447" 1539100642 0.07827372816666665)
  ("5b218be0c362316384f5c9ef57a3bef02f742e94" 1541077242 0.07896257613333332)
  ("75b18e07e57da7ee4362db800352d6650f5f7290" 1542976351 0.08181464855000001))
#+END_SRC

** Syntax Tables
=src/syntax-tasks.el=
*** syntax-tasks-backward-word
Use ‘forward-word’ to navigate backwards through a buffer.
#+BEGIN_SRC gnuplot :var data=syntax-tasks-backward-word-1.0() :file plots/syntax-tasks-backward-word-1.0.png
 :noweb yesreset
set terminal png size 800, 600
set title "syntax-tasks-backward-word"
set xlabel "Commit"
set xrange [1459356037:1542976351]
<<xtics>>
set ylabel "Run time (seconds)"
set yrange [0:0.00436608931]
set key right bottom
plot data u 2:3 w lp lw 2 title 'sockeye'
#+END_SRC

*** Measurements for syntax-tasks-backward-word-1.0 :noexport:
#+NAME: syntax-tasks-backward-word-1.0
#+BEGIN_SRC emacs-lisp
'(("09462b95484ebf75899c64e8ddb7ffce50ef6ca0" 1459356037 0.003393012233333333)
  ("33f856ba01d13f649cf5c848b322ecb0dbfc02fc" 1474022875 0.003425965433333333)
  ("d4a97088f69eb5729261ee4581cfb7d60c673ebd" 1484271619 0.0035167778999999986)
  ("d1da8e534db54bb82b6f2b16bba86a6d202ac675" 1504966847 0.0035477279999999996)
  ("0676633e8fd842554e91d0756115fa2d6b7cf69c" 1518488627 0.0033622731333333333)
  ("e18600dfbd31316f30cf98f6d7a90aad33bc1981" 1527019548 0.0033781541666666665)
  ("bd013a448b152a84cff9b18292d8272faf265447" 1539100642 0.003293653366666666)
  ("5b218be0c362316384f5c9ef57a3bef02f742e94" 1541077242 0.003221600399999999)
  ("75b18e07e57da7ee4362db800352d6650f5f7290" 1542976351 0.0039691721))
#+END_SRC

*** syntax-tasks-forward-word
Use ‘forward-word’ to navigate through a buffer.
#+BEGIN_SRC gnuplot :var data=syntax-tasks-forward-word-1.0() :file plots/syntax-tasks-forward-word-1.0.png
 :noweb yesreset
set terminal png size 800, 600
set title "syntax-tasks-forward-word"
set xlabel "Commit"
set xrange [1459356037:1542976351]
<<xtics>>
set ylabel "Run time (seconds)"
set yrange [0:0.004305837690000001]
set key right bottom
plot data u 2:3 w lp lw 2 title 'sockeye'
#+END_SRC

*** Measurements for syntax-tasks-forward-word-1.0 :noexport:
#+NAME: syntax-tasks-forward-word-1.0
#+BEGIN_SRC emacs-lisp
'(("09462b95484ebf75899c64e8ddb7ffce50ef6ca0" 1459356037 0.0034616826333333335)
  ("33f856ba01d13f649cf5c848b322ecb0dbfc02fc" 1474022875 0.0034043496)
  ("d4a97088f69eb5729261ee4581cfb7d60c673ebd" 1484271619 0.0034625005666666666)
  ("d1da8e534db54bb82b6f2b16bba86a6d202ac675" 1504966847 0.0036520705)
  ("0676633e8fd842554e91d0756115fa2d6b7cf69c" 1518488627 0.0033964009333333333)
  ("e18600dfbd31316f30cf98f6d7a90aad33bc1981" 1527019548 0.003364131566666667)
  ("bd013a448b152a84cff9b18292d8272faf265447" 1539100642 0.0033124766333333332)
  ("5b218be0c362316384f5c9ef57a3bef02f742e94" 1541077242 0.0032527114666666664)
  ("75b18e07e57da7ee4362db800352d6650f5f7290" 1542976351 0.0039143979))
#+END_SRC

* Benchmark machine information
** sockeye
*** Configuration
Arguments used to invoke Emacs: =-Q --batch=
*** System information
#+OPTIONS: ^:nil
#+BEGIN_SRC emacs-lisp :results value table :exports results
'(("OS" "NixOS 18.09.git.7bac6c1 (Jellyfish) x86_64 ")
  ("Kernel" "4.14.83 ")
  ("CPU" "Intel i5-7200U (4) @ 3.100GHz ")
  ("GPU" "Intel HD Graphics 620 ")
  ("Memory" "7869MiB")
  ("WM" "stumpwm ")
  ("DE" "none+stumpwm ")
  ("Resolution" "1920x1080 ")
  ("Shell" "bash 4.4.23 "))
#+END_SRC

System information last updated: 2018-11-24 Sat 12:52
* Xtics :noexport:
#+BEGIN_SRC gnuplot :export none
set xtics rotate by -45 \
    ("emacs-26.1" 1527266287, \
     "emacs-25.3" 1505243110, \
     "emacs-25.2" 1492702151, \
     "emacs-25.1" 1473975009, \
     "emacs-24.5" 1428344408, \
     "emacs-24.4" 1413832875, \
     "emacs-24.3" 1362969323, \
     "emacs-24.2" 1345802019, \
     "emacs-24.1" 1339314106, \
     "emacs-23.4" 1327469705, \
     "emacs-23.3" 1299528353, \
     "emacs-23.2" 1273289306, \
     "emacs-23.1" 1248916455, \
     "emacs-22.3" 1220631826, \
     "emacs-22.2" 1206539448, \
     "emacs-22.1" 1180782708, \
     "emacs-21.3" 1048000793, \
     "emacs-21.2" 1016280953, \
     "emacs-21.1" 1003594590, \
     "emacs-20.4" 931968057, \
     "emacs-20.3" 903548661, \
     "emacs-20.2" 874730909, \
     "emacs-20.1" 874478501)
#+END_SRC

[-- Attachment #3: Type: text/plain, Size: 169 bytes --]


Each timing value in this file is the average of 30 runs of the task.
Here's the same report, exported with graphs from gnuplot (and page
breaks in the wrong places):


[-- Attachment #4: report.pdf --]
[-- Type: application/pdf, Size: 220268 bytes --]

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Paul.

On Sun, Nov 25, 2018 at 10:45:09 -0800, Paul Eggert wrote:
> I didn't realize until now that this branch makes EQ slower for everybody, ....

Yes.  That is why I have been so concerned about the speed, right from
the beginning.

> .... for what seems to be a relatively minor issue (i.e., more accurate
> diagnostics). ....

Well in the branch I'm working on, there are 335 warnings with a
position, and the old way of doing things gave the correct position in
only 81 of these, being wrong in the other 254.

There have been quite a lot of bug reports about wrong warning positions.

> Surely there is a better way.

Stefan, Eli, and I had quite an extensive discussion about possible
approaches two or three or four weeks ago.  Our conclusion (formulated by
me) was that there was no good way of resolving the bug, only varying
degrees of bad ways.

How do other Lisp compilers work?  They must have ways of keeping track
of line and column numbers.

> Why not add an option to 'read' so that its returned value contains 
> symbol-with-pos objects instead of symbol objects?

That's more or less what's been done, except that there's a second
function called read-positioning-symbols alongside read.

> That way, programs like the Elisp compiler that read objects can
> generate more accurate diagnostics without having to slow down EQ for
> everybody.

The problem is that when compilation is in progress,

    (eq 'foo '#<symbol foo at 666>)

has got to return t.  I can't see a better way of doing this than testing
the variable symbols-with-position-enabled at each failing == operation.

If you have any better suggestions, I'd love to hear of them.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello again, Eli.

On Sun, Nov 25, 2018 at 18:23:30 +0000, Alan Mackenzie wrote:
> On Sun, Nov 25, 2018 at 20:15:30 +0200, Eli Zaretskii wrote:
> > > Date: Sun, 25 Nov 2018 17:59:29 +0000
> > > From: Alan Mackenzie <acm@muc.de>
> > > Cc: cpitclaudel@gmail.com, charles@aurox.ch, michael_heerdegen@web.de,
> > > 	emacs-devel@gnu.org, schwab@linux-m68k.org, monnier@IRO.UMontreal.CA

[ .... ]

> > As I said, I'd be very surprised if using DEFVAR_BOOL would slow down
> > what you did in any significant way, beyond the slowdown due to the
> > comparison and struct member access.

> I'll time it, then we'll know.

[ .... ]

> > > I'm still a little confused as to what a DEFVAR_BOOL is.

> > It's a C 'bool'.  Look at defvar_bool the function: its last argument
> > is a pointer to the C counterpart of a DEFVAR_BOOL.

> > You can also use ptype in the debugger to see what it really is, if
> > you want to be absolutely sure.

> OK, thanks for the explanation.  I'll change the declaration of that
> variable (and another one) to DEFVAR_BOOL, and tidy up lisp.h to match.

> Then I'll time the resulting Emacs again.  :-)

DONE.  The resulting Emacs with DEFVAR_BOOL took just the same amount of
time to bootstrap as the same code with DEFVAR_LISP, to within
measurement error.

So, I'll commit the version with the DEFVAR_BOOLs.

> -- 
> Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Stefan.

On Sun, Nov 25, 2018 at 13:22:38 -0500, Stefan Monnier wrote:

[ .... ]

> > NILP won't work here (as I pointed out to Eli), because of circular
> > dependencies inside lisp.h.

> Oh, right:

>     #define lisp_h_NILP(x) EQ (x, Qnil)

> Reminds me of another question I had: in your code, do you need NILP to
> pay attention to symbols-with-pos-enabled?

I think so, yes.  The compiler has to recognise an explicit #<symbol nil
at 666> as being nil.  I had quite some fun in the hacking with the (foo
. nil)s in fontset.el.

> That would significantly increase the impact in terms of code-size and
> CPU time (could explain why you see 8% slowdown which is more expansive
> than my patch which turns EQ into EQL).

Would it?  Currently, an EQ does a binary == test between the
Lisp_Objects, and only if that fails does it test
symbols-with-pos-enabled.  So if the number of successful and failing
tests were equal (I've really no idea), EQ would be slowed down by 33%.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Stefan Monnier]

> cus-theme-tasks-load-tango: 9%
> bytecomp-tasks-compile-doc: 24%
> kmacro-tasks-edit-lines: 18%
> fill-tasks-fill-paragraph: 15%
> cmds-tasks-forward-line: 5%
> data-tasks-prime-p: 1%
> emacs-tasks-startup: 4%
> syntax-tasks-backward-word: 23%
> syntax-tasks-forward-word: 20%

Ugh!  That's worse than I expected.
Unless we can bring those numbers down I'm afraid we're going to have to
look for another one of the "bad choices".


        Stefan

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Stefan Monnier]

>> That would significantly increase the impact in terms of code-size and
>> CPU time (could explain why you see 8% slowdown which is more expansive
>> than my patch which turns EQ into EQL).
> Would it?

Yes: I think NILP is much more frequent than EQ.


        Stefan

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

Alan Mackenzie wrote:

> How do other Lisp compilers work?  They must have ways of keeping track
> of line and column numbers.

I'm pretty sure they don't redefine the meaning of 'eq'....

> The problem is that when compilation is in progress,
> 
>      (eq 'foo '#<symbol foo at 666>)
> 
> has got to return t.

Why not change the byte-compiler to use 'eq-ignoring-symbol-pos' (or whatever 
you want to call it) instead of 'eq'? The byte compiler could do this for every 
occurrence of 'eq' in every user macro, as well as in its own code. This should 
let ordinary Lisp code run at full speed; only compilation would be slowed down.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Paul.

On Sun, Nov 25, 2018 at 12:12:56 -0800, Paul Eggert wrote:
> Alan Mackenzie wrote:

> > How do other Lisp compilers work?  They must have ways of keeping track
> > of line and column numbers.

> I'm pretty sure they don't redefine the meaning of 'eq'....

Maybe, maybe not.  How do they actually do it?  I simply don't know, I
thought you might.

As I said, when we discussed this bug a few weeks ago, we could only
come up with bad ways of dealing with it.  The current bad way has the
advantage of being practical and implementable.

> > The problem is that when compilation is in progress,

> >      (eq 'foo '#<symbol foo at 666>)

> > has got to return t.

> Why not change the byte-compiler to use 'eq-ignoring-symbol-pos' (or
> whatever you want to call it) instead of 'eq'?

Because of macros.  These macros are typically already compiled.

> The byte compiler could do this for every occurrence of 'eq' in every
> user macro, ....

No it couldn't.  In this context, "user macro"s include any helper
defuns that they call.  They would need to be converted, too.  To say
nothing of quite a few standard Emacs functions, probably.  I've no idea
how any of this could be done, but suspect it would be more work than
reimplementing the byte compiler from scratch.

> .... as well as in its own code. This should let ordinary Lisp code
> run at full speed; only compilation would be slowed down.

The actual slow down is currently uncertain.  On benchmarks, Gemini
Lasswell has measured between 1% and (?)30% slowdown.  On realistic
applications (a make bootstrap and scrolling through xdisp.c) I've seen
7 to 8% slowdown.

If you could come up with a solid proposal which would fix the bug
without slowing down Emacs at all, we'd all be most appreciative.

One of the bad ways of dealing with the bug was not to bother fixing it
at all.  Given how poor the current method is (under 25% hit rate in a
bootstrap), I don't think this is a good idea.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

Alan Mackenzie wrote:

> Because of macros.  These macros are typically already compiled.

Even a compiled macro operates via the interpreter. So we could have a separate 
interpreter used only by the byte compiler. The byte-compiler interpreter would 
operate more slowly than the normal interpreter, but that's OK. The main and the 
byte-compiler interpreter could mostly be written with shared code, without 
slowing down the main interpreter. Admittedly this would not be a project for 
the fainthearted.

> If you could come up with a solid proposal which would fix the bug
> without slowing down Emacs at all, we'd all be most appreciative.

I'm afraid I don't understand the bug well enough yet to know whether any 
proposal I can come up with would be "solid". For one thing, any method of 
outputting source-code locations will founder in the presence of macros. Even 
GCC, which tries to do a reasonably good job of this and isn't limited by the 
Lisp reader, doesn't do well with the sort of C macros I tend to write. My 
admittedly uninformed guess is that there is no such thing as a "solid" solution 
here, only solutions that work better and/or worse on various example sets.

That being said, here's another possibility: don't bother attaching source-code 
positions to symbols, since duplicate symbols can be appear in the input and the 
source-code positions can't be retrieved reliably. Instead, attach positions to 
input objects that are guaranteed to be unique so that retrieval is trivial. Do 
the attachment via a hash table so that the input objects are unchanged and we 
don't need to change much of anything except the byte-compiler's diagnostic code 
(plus a read function that fills in the hash table as it reads). When the byte 
compiler needs the source code location corresponding to a symbol, it looks for 
the closest unique object nearby and uses its location.

For example, the source expression for the bug#22288 test case:

   (defun test () (let (a)) a)

has five conses in its top level list, two conses at the top of its second level 
list (let (a)), and one cons in its third level list (a). Each cons corresponds 
to a source code position (or if you prefer more accuracy, multiple positions 
for the start and end of the corresponding source-code and/or for the starts and 
ends of the source code corresponding to the cons's car and cdr). This will let 
the byte compiler narrow down where every subexpression lies, with significantly 
more accuracy than what we have now. In the bug#22288 example, the last cons in 
the top-level list should be attached to the precise source code location for 
the 'a' that we want to issue a diagnostic about.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Eli Zaretskii]

> Cc: michael_heerdegen@web.de, eliz@gnu.org, cpitclaudel@gmail.com,
>  monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Sun, 25 Nov 2018 17:41:39 -0800
> 
> we could have a separate interpreter used only by the byte compiler.

I don't think I like this idea: it will be a maintenance burden to
maintain two interpreters, and we will have subtle long-lived bugs.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Paul.

On Sun, Nov 25, 2018 at 17:41:39 -0800, Paul Eggert wrote:
> Alan Mackenzie wrote:

> > Because of macros.  These macros are typically already compiled.

> Even a compiled macro operates via the interpreter. So we could have a
> separate interpreter used only by the byte compiler. The byte-compiler
> interpreter would operate more slowly than the normal interpreter, but
> that's OK. The main and the byte-compiler interpreter could mostly be
> written with shared code, without slowing down the main interpreter.
> Admittedly this would not be a project for the fainthearted.

Indeed not.  Where and how would this help with getting accurate source
code positions?

> > If you could come up with a solid proposal which would fix the bug
> > without slowing down Emacs at all, we'd all be most appreciative.

> I'm afraid I don't understand the bug well enough yet to know whether
> any proposal I can come up with would be "solid". For one thing, any
> method of outputting source-code locations will founder in the
> presence of macros.

scratch/accurate-warning-pos seems to do rather well in this regard.

> Even GCC, which tries to do a reasonably good job of this and isn't
> limited by the Lisp reader, doesn't do well with the sort of C macros
> I tend to write. My admittedly uninformed guess is that there is no
> such thing as a "solid" solution here, only solutions that work better
> and/or worse on various example sets.

The example set scratch/accurate-warning-pos works well on is the Lisp
code comprising Emacs.

> That being said, here's another possibility: don't bother attaching
> source-code positions to symbols, since duplicate symbols can be
> appear in the input and the source-code positions can't be retrieved
> reliably.

The source code positions are attached not to symbols, but to symbol
_occurrences_.

> Instead, attach positions to input objects that are guaranteed to be
> unique so that retrieval is trivial.

I think you mean conses here.  I've tried this approach, spending a lot
of time on it but not getting very far.  The problem is, Lisp objects
flow through lots of different conses as they are transformed by the
compiler.  Have a look at cconv-convert, which processes every function.
I'm not sure that even a single cons in the input form survives through
to the output.  The symbols do survive, though, in the main.

> Do the attachment via a hash table so that the input objects are
> unchanged and we don't need to change much of anything except the
> byte-compiler's diagnostic code (plus a read function that fills in
> the hash table as it reads).

Using conses as keys?  See previous paragraph.  The approach I tried
before to implement this was to ensure that after any source
transformation, the result was written back to the original cons using
setcar and setcdr.  This rapidly became unwieldy, with, for example,
versions of setq and mapcar which had extra parameters indicating the
result cons, and so on.  This involved extensive amendment of large
portions of the compiler.

> When the byte compiler needs the source code location corresponding to
> a symbol, it looks for the closest unique object nearby and uses its
> location.

"Nearby"?  Warning messages are typically associated with symbol
occurences (not conses), and are found when a recursive compiler routine
is presented with a symbol rather than a cons.  Not all the time, but a
lot of the time.  Hence the scratch/accurate-warning-pos approach of
attaching source positions to symbol occurrences.

> For example, the source expression for the bug#22288 test case:

>    (defun test () (let (a)) a)

> has five conses in its top level list, two conses at the top of its
> second level list (let (a)), and one cons in its third level list (a).
> Each cons corresponds to a source code position (or if you prefer more
> accuracy, multiple positions for the start and end of the
> corresponding source-code and/or for the starts and ends of the source
> code corresponding to the cons's car and cdr). This will let the byte
> compiler narrow down where every subexpression lies, with
> significantly more accuracy than what we have now. In the bug#22288
> example, the last cons in the top-level list should be attached to the
> precise source code location for the 'a' that we want to issue a
> diagnostic about.

Yes, in theory.  In practice, as already said, the source code flows
between lots of cons cells as it is transformed by functions like
cconv-convert and those in byte-opt.el.  As said, countering this would
involve lots of tedious amendment to the compiler, with the emphasis on
"lots".  I've already tried this, and given up.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Stefan.

On Sun, Nov 25, 2018 at 15:08:52 -0500, Stefan Monnier wrote:
> >> That would significantly increase the impact in terms of code-size and
> >> CPU time (could explain why you see 8% slowdown which is more expansive
> >> than my patch which turns EQ into EQL).
> > Would it?

> Yes: I think NILP is much more frequent than EQ.

Ah right, I see what you mean, now:  Leave EQ more or less as it is in
scratch/accurate-warning-pos, but have NILP use "BASE_EQ" rather than EQ
to compare with Qnil.

My gut feeling is that this won't work, but I'm going to try it, anyway.

Thanks for the suggestion!

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello again, Stefan.

On Mon, Nov 26, 2018 at 09:52:19 +0000, Alan Mackenzie wrote:
> On Sun, Nov 25, 2018 at 15:08:52 -0500, Stefan Monnier wrote:
> > >> That would significantly increase the impact in terms of code-size and
> > >> CPU time (could explain why you see 8% slowdown which is more expansive
> > >> than my patch which turns EQ into EQL).
> > > Would it?

> > Yes: I think NILP is much more frequent than EQ.

> Ah right, I see what you mean, now:  Leave EQ more or less as it is in
> scratch/accurate-warning-pos, but have NILP use "BASE_EQ" rather than EQ
> to compare with Qnil.

> My gut feeling is that this won't work, but I'm going to try it, anyway.

I've just tried it in a make bootstrap.

There was a binding stack overflow whilst compiling
.../lisp/composite.el.  I don't know exactly why.

> Thanks for the suggestion!

> >         Stefan

> -- 
> Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Gemini.

On Sun, Nov 25, 2018 at 10:48:38 -0800, Gemini Lasswell wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > The branch scratch/accurate-warning-pos now bootstraps.  (It also now
> > runs on optimised builds. ;-)
> >
> > Its bootstrap is a mere 7.1% slower than master's.

> Hi Alan,

> Great progress!  There are clearly benefits to Emacs Lisp programmers
> (especially if errors and debugger backtraces can someday have accurate
> source positions), to be weighed against the cost of a slower Emacs for
> all the non-Lisp programmers and non-programmers who work with Emacs.  I
> don't know how to begin to sort that out, but since I've been working on
> creating a benchmark suite for Emacs, I'm thrilled you've given me such
> an interesting test case. :)

:-)

> Here's how much longer in percentage terms the tasks in my nascent
> benchmark suite take in your branch compared to 5b218be0c3, the commit
> your branch is based on, running on my newish Linux laptop.  I'll push
> my benchmarking project to a scratch branch soon so you can see the code
> which was benchmarked.

> cus-theme-tasks-load-tango: 9%
> bytecomp-tasks-compile-doc: 24%
> kmacro-tasks-edit-lines: 18%
> fill-tasks-fill-paragraph: 15%
> cmds-tasks-forward-line: 5%
> data-tasks-prime-p: 1%
> emacs-tasks-startup: 4%
> syntax-tasks-backward-word: 23%
> syntax-tasks-forward-word: 20%

Ouch!  That's slower than I had anticipated or measured myself.

From this point on, I probably look a bit defensive.  Forgive me for
this.

I'm guessing what each benchmark tested.  I see you've pushed a scratch
branch for this, but I haven't looked at this yet.

syntax-tasks-{backward,forward}-word presumably time backward-word and
forward-word.  bytecomp-tasks-compile-doc I'm not sure about, is that
the extraction of doc strings from .elc files?

After getting over my knee-jerk reaction, my thoughts are that these
timings are not of things which, of themselves, would directly concern a
user.  A fill-paragraph will take 15% more time, but would a user
actually notice this, given that the operation is usually instantaneous?

Good things to benchmark would be interactive commands which feel a bit
sluggish anyway.  CC Mode redisplay is a good candidate for this (see
below).  ;-)

From your timings, my gut feeling is that the 15% slowdown in
fill-tasks-fill-paragraph probably represents fairly closely the
slowdown in Emacs as a whole.  It is testing a "macro" operation rather
than an isolated primitive.

> Here's a benchmark report, which contains the measurements from which I
> calculated those percentages:

[ .... ]

> Each timing value in this file is the average of 30 runs of the task.
> Here's the same report, exported with graphs from gnuplot (and page
> breaks in the wrong places):

Thanks for doing this!  One positive thing is that the new Emacs hasn't
fallen over.  ;-)

My 8% (approximate) I measured by scrolling through xdisp.c, displaying
each window's worth on the way.  Before repeating this timing, it's
necessary to insert a character at BOB then undo the insertion, so as to
erase the face and other properties throughout the buffer.

Perhaps you might be able to adapt this exercise into your branch.  At
any rate, I'd be interested if you could compare the timings on your
machine between old and new Emacs versions.

The code to run this follows.  Do M-: (time-scroll) to start it.

(defmacro time-it (&rest forms)
  "Time the running of a sequence of forms using `float-time'.
Call like this: \"M-: (time-it (foo ...) (bar ...) ...)\"."
  `(let ((start (float-time)))
    ,@forms
    (- (float-time) start)))

(defun time-scroll (&optional arg)
  (interactive "P")
  (message "%s"
           (time-it
            (condition-case nil
                (while t
                  (if arg (scroll-down) (scroll-up))
                  (sit-for 0))
              (error nil)))))

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Gemini Lasswell]

Alan Mackenzie <acm@muc.de> writes:

Hi Alan,

> syntax-tasks-{backward,forward}-word presumably time backward-word and
> forward-word.  bytecomp-tasks-compile-doc I'm not sure about, is that
> the extraction of doc strings from .elc files?

You are correct about syntax-tasks-*.  bytecomp-tasks-compile-doc
measures the time it takes to compile the function 'doctor-doc' from
doctor.el (since I want benchmark tasks to run in old versions of Emacs,
I went looking in lisp/play for something that hadn't changed in a long
time.)

> After getting over my knee-jerk reaction, my thoughts are that these
> timings are not of things which, of themselves, would directly concern a
> user.  A fill-paragraph will take 15% more time, but would a user
> actually notice this, given that the operation is usually instantaneous?
>
> Good things to benchmark would be interactive commands which feel a bit
> sluggish anyway.  CC Mode redisplay is a good candidate for this (see
> below).  ;-)
>
> From your timings, my gut feeling is that the 15% slowdown in
> fill-tasks-fill-paragraph probably represents fairly closely the
> slowdown in Emacs as a whole.  It is testing a "macro" operation rather
> than an isolated primitive.

Every time I use a keyboard macro to edit every line in a file, I wonder
why it takes so long.  kmacro-tasks-edit-lines changes 10 lines of
"1. Flintstone, Fred" to read "Fred Flintstone" using basic editing
commands.  From the current short list of benchmarks, it would be my
choice for representing Emacs as a whole.

> My 8% (approximate) I measured by scrolling through xdisp.c, displaying
> each window's worth on the way.  Before repeating this timing, it's
> necessary to insert a character at BOB then undo the insertion, so as to
> erase the face and other properties throughout the buffer.
>
> Perhaps you might be able to adapt this exercise into your branch.  At
> any rate, I'd be interested if you could compare the timings on your
> machine between old and new Emacs versions.

Thanks for the suggestion and the code.  The benchmark numbers in my
last email were all created with --batch, meaning no display.  But
running benchmarks without --batch, so that redisplay can be
benchmarked, is one of the things I'm working on.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello again, Gemini.

On Mon, Nov 26, 2018 at 08:14:28 -0800, Gemini Lasswell wrote:
> Alan Mackenzie <acm@muc.de> writes:

> Hi Alan,

> > syntax-tasks-{backward,forward}-word presumably time backward-word and
> > forward-word.  bytecomp-tasks-compile-doc I'm not sure about, is that
> > the extraction of doc strings from .elc files?

> You are correct about syntax-tasks-*.  bytecomp-tasks-compile-doc
> measures the time it takes to compile the function 'doctor-doc' from
> doctor.el (since I want benchmark tasks to run in old versions of Emacs,
> I went looking in lisp/play for something that hadn't changed in a long
> time.)

Ah, "doc" as in "doctor", not "document".  :-)

When I compile doctor.el with (time-it (byte-compile-file
".../play/doctor.el")), I see a slow down of 9.9%, which is surprisingly
little, given how much extra processing is involved when
symbols-with-positions-enabled is bound to non-nil.

Compiling a single function from that file, you see a 24% slowdown (if I
remember the figures correctly).  There's some explaining to do there to
account for the difference between what we see.

My machine is a desktop with an AMD Ryzen processor, and I run Gentoo
GNU/Linux.  Maybe the processor architecture accounts for some of the
difference.

> > After getting over my knee-jerk reaction, my thoughts are that these
> > timings are not of things which, of themselves, would directly concern a
> > user.  A fill-paragraph will take 15% more time, but would a user
> > actually notice this, given that the operation is usually instantaneous?

> > Good things to benchmark would be interactive commands which feel a bit
> > sluggish anyway.  CC Mode redisplay is a good candidate for this (see
> > below).  ;-)

> > From your timings, my gut feeling is that the 15% slowdown in
> > fill-tasks-fill-paragraph probably represents fairly closely the
> > slowdown in Emacs as a whole.  It is testing a "macro" operation rather
> > than an isolated primitive.

> Every time I use a keyboard macro to edit every line in a file, I wonder
> why it takes so long.  kmacro-tasks-edit-lines changes 10 lines of
> "1. Flintstone, Fred" to read "Fred Flintstone" using basic editing
> commands.  From the current short list of benchmarks, it would be my
> choice for representing Emacs as a whole.

That one was 18%.  I think it would be a fair summary to say that you
are seeing a 15 - 20% slowdown in "real life" measurements (which
excludes the byte compilation, which everybody agrees can take as long
as it likes).  I seem to be seeing a lesser slowdown, of around 10%,
although I haven't yet got around to running your new branch and doing a
solid test.

Quite likely, Eli is keeping a keen eye on this part of the thread to
see how things develop.  ;-)  My feeling is that the new branch is up to
being integrated into master eventually, with "only" 10 - 20% slowdown,
but then I'm not exactly impartial.

[ .... ]

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Gemini.

On Mon, Nov 26, 2018 at 17:06:52 +0000, Alan Mackenzie wrote:
> When I compile doctor.el with (time-it (byte-compile-file
> ".../play/doctor.el")), I see a slow down of 9.9%, which is surprisingly
> little, given how much extra processing is involved when
> symbols-with-positions-enabled is bound to non-nil.

> Compiling a single function from that file, you see a 24% slowdown (if I
> remember the figures correctly).  There's some explaining to do there to
> account for the difference between what we see.

> My machine is a desktop with an AMD Ryzen processor, and I run Gentoo
> GNU/Linux.  Maybe the processor architecture accounts for some of the
> difference.

I'll take that back.  On the first run of byte-compile-file (on master),
the file had to fetched from a drive.  On the second run (on the branch)
it would already have been in cache.

When I repeated the exercise, I saw a 21% slowdown, the same as what you
saw, to within measurement error.

It shows just how careful one must be.

Sorry about this mistake.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

On 11/25/18 7:41 PM, Eli Zaretskii wrote:
>> we could have a separate interpreter used only by the byte compiler.
> I don't think I like this idea: it will be a maintenance burden to
> maintain two interpreters, and we will have subtle long-lived bugs.

Yes, the only way it's plausible is if we have just one source-code 
instance of the interpreter, parameterized by whether it's doing things 
the slower (byte-compiler-oriented) way or the faster (current) way, and 
then partially evaluate the interpreter for the two cases. In that case 
the source code should be not much more complicated than 
scratch/accurate-warning-pos, and the number of subtle long-lived bugs 
should be about the same as scratch/accurate-warning-pos has now.

As I said, this approach is not for the fainthearted. But it should fix 
the significant performance problems in scratch/accurate-warning-pos.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

On 11/26/18 1:48 AM, Alan Mackenzie wrote:
>
>> The byte-compiler
>> interpreter would operate more slowly than the normal interpreter, but
>> that's OK. The main and the byte-compiler interpreter could mostly be
>> written with shared code, without slowing down the main interpreter.
>> Admittedly this would not be a project for the fainthearted.
> Indeed not.  Where and how would this help with getting accurate source
> code positions?

 From the point of view of the byte-compiler, it should help just as 
well as scratch/accurate-warning-pos does, because it's just another way 
to get the same information. Admittedly it's more work to implement this.


>> any method of outputting source-code locations will founder in the
>> presence of macros.
> scratch/accurate-warning-pos seems to do rather well in this regard.

No doubt it does better than master does. However, macros inevitably 
cause problems when diagnosing problems. If there's a bug in a macro, it 
could well be that the byte compiler will report the line number of the 
call to the macro rather than of the macro itself (where the real bug 
is). So any definition of a "solid" solution must provide some leeway 
for the compiler to issue a "wrong" diagnostic when macros are involved. 
Otherwise, if the goal is to provide a "solid" solution then the 
goalposts will keep moving and there's no way that I could suggest a 
solution that is unambiguously "solid".


>> Instead, attach positions to input objects that are guaranteed to be
>> unique so that retrieval is trivial.
> I think you mean conses here.  I've tried this approach, spending a lot
> of time on it but not getting very far.  The problem is, Lisp objects
> flow through lots of different conses as they are transformed by the
> compiler.  Have a look at cconv-convert, which processes every function.

I took a brief look there and see a lot of calls to mapcar, which of 
course would lose track of locations. But this should be fixable by 
defining and using a new function mapcar-pos, which acts like mapcar 
except it also copies location information from the input list's conses 
to the corresponding conses in the output list.

You're right that this would involve many changes to the byte compiler, 
but the changes should be reasonably mechanical (e.g., change mapcar to 
mapcar-pos) and the overall approach should be preferable to changing 
how 'eq' works.

> The approach I tried
> before to implement this was to ensure that after any source
> transformation, the result was written back to the original cons using
> setcar and setcdr.

If location info is kept in a hash table that maps objects to locations, 
then mapcar-pos can preserve location info merely by updating a hash 
table. There should be no need for setcar and setcdr on the original cons.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Paul.

On Mon, Nov 26, 2018 at 09:43:46 -0800, Paul Eggert wrote:
> On 11/25/18 7:41 PM, Eli Zaretskii wrote:
> >> we could have a separate interpreter used only by the byte compiler.
> > I don't think I like this idea: it will be a maintenance burden to
> > maintain two interpreters, and we will have subtle long-lived bugs.

> Yes, the only way it's plausible is if we have just one source-code 
> instance of the interpreter, parameterized by whether it's doing things 
> the slower (byte-compiler-oriented) way or the faster (current) way, and 
> then partially evaluate the interpreter for the two cases.

I think you're going to be using the symbols-with-position approach, but
instead of evaluating symbols-with-pos-enabled on each instance of EQ,
NILP, and SYMBOLP, you're going to evaluate it just once, and branch to
the appropriate version of the interpreter for that case.

If you're not thinking of using the symbols-with-position approach, I
earnestly urge you to read the thread "Thoughts on getting correct line
numbers in the byte compiler's warning messages", which began here on
2018-11-01.  It might well persuade you.

> In that case the source code should be not much more complicated than
> scratch/accurate-warning-pos, and the number of subtle long-lived bugs
> should be about the same as scratch/accurate-warning-pos has now.

It will be more complicated, but not by an order of magnitude, perhaps.

> As I said, this approach is not for the fainthearted.

You're not kidding, there.

> But it should fix the significant performance problems in
> scratch/accurate-warning-pos.

The slowdown seems to be chrystallising out at 20 - 25% while byte
compiling, and 5 - 20% otherwise.

This may be significant, but is it really important?  Most of the time,
Emacs is waiting for the next key depression anyway.  If a batch like
operation takes 11.5 minutes rather than 10 minutes, who would really
notice?  Similarly if an instantaneous operation, like filling a
paragraph increases by 15%, would anybody care?  It is surely only
those interactive operations which are sluggish which might become more
so, where the slowdown would be important.  How many of these really
exist in Emacs?

As a data point, my 1.5 year old machine is about 150% faster (per core)
than its predecessor.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Clément Pit-Claudel]

On 26/11/2018 13.43, Alan Mackenzie wrote:
> This may be significant, but is it really important?  Most of the time,
> Emacs is waiting for the next key depression anyway.  If a batch like
> operation takes 11.5 minutes rather than 10 minutes, who would really
> notice?  Similarly if an instantaneous operation, like filling a
> paragraph increases by 15%, would anybody care?  It is surely only
> those interactive operations which are sluggish which might become more
> so, where the slowdown would be important.  How many of these really
> exist in Emacs?

This is a tricky question. Places where Emacs is too slow for my taste, and that I wouldn't like to see become slower, include indentation (which often requires many position-related operations, as well as macro editing, and syntax highlighting.

I do have one question: I understand why attaching positions to code fragments make reading the code slower, but why does it make interpreting it slower as well?

Another related question: this work is for warnings, but will it extend to having positions in backtraces?

Congrats on the had work, btw :) This would be a very useful feature to have.
Clément.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Clément.

On Mon, Nov 26, 2018 at 14:18:36 -0500, Clément Pit-Claudel wrote:
> On 26/11/2018 13.43, Alan Mackenzie wrote:
> > This may be significant, but is it really important?  Most of the
> > time, Emacs is waiting for the next key depression anyway.  If a
> > batch like operation takes 11.5 minutes rather than 10 minutes, who
> > would really notice?  Similarly if an instantaneous operation, like
> > filling a paragraph increases by 15%, would anybody care?  It is
> > surely only those interactive operations which are sluggish which
> > might become more so, where the slowdown would be important.  How
> > many of these really exist in Emacs?

> This is a tricky question. Places where Emacs is too slow for my
> taste, and that I wouldn't like to see become slower, include
> indentation (which often requires many position-related operations, as
> well as macro editing, and syntax highlighting.

The pertinent position information doesn't exist at runtime.  The only
slowing down results from .....

> I do have one question: I understand why attaching positions to code
> fragments make reading the code slower, but why does it make
> interpreting it slower as well?

Mainly because of EQ, the C macro corresponding to `eq'.  In master, EQ
just does a binary comparison between two Lisp_Objects, and that's it.
In scratch/accurate-warning-pos, should that binary comparison fail, EQ
additionally has to check the flag variable `symbols-with-pos-enabled'
to see whether it needs to do any additional comparisons.  Except while
byte compiling, it won't, but the check of that variable is what is
causing the slowdown.

> Another related question: this work is for warnings, but will it
> extend to having positions in backtraces?

I hadn't actually got to thinking about that, but the answer is
proabably not, at least for now.  The .elc format for compiled functions
has a single entry for "each" symbol in its constant vector, and it
would be quite an exercise to augment that with symbols-with-position.

Maybe for the future.

> Congrats on the had work, btw :) This would be a very useful feature
> to have.

Thanks!

> Clément.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Stefan Monnier]

> Yes, the only way it's plausible is if we have just one source-code instance
> of the interpreter, parameterized by whether it's doing things the slower
> (byte-compiler-oriented) way or the faster (current) way, and then partially

You going to have to define "the interpreter".
Is the code of Fmemq part of your definition of "the interpreter"?


        Stefan

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Gemini Lasswell]

Alan Mackenzie <acm@muc.de> writes:

Hi Alan,

> Mainly because of EQ, the C macro corresponding to `eq'.  In master, EQ
> just does a binary comparison between two Lisp_Objects, and that's it.
> In scratch/accurate-warning-pos, should that binary comparison fail, EQ
> additionally has to check the flag variable `symbols-with-pos-enabled'
> to see whether it needs to do any additional comparisons.  Except while
> byte compiling, it won't, but the check of that variable is what is
> causing the slowdown.

I'm having trouble accepting that this is the problem.

When I hear a story about a C program, where adding an always failing
check of a boolean variable to it causes it to run slower by 15% or 20%,
then when I read the code I expect to find that variable check inside a
really tight loop with only a few other instructions besides the
variable check.

But Fforward_word is significantly slower in your branch and I can't
find any tight loops with an EQ in it.  exec_byte_code might fit that
description, if you gave it a Lisp function with a loop that hammered on
'eq', but that doesn't describe fill-region-as-paragraph, which is 15%
slower.  Could there be anything else causing a slowdown besides EQ?

>> Another related question: this work is for warnings, but will it
>> extend to having positions in backtraces?
>
> I hadn't actually got to thinking about that, but the answer is
> proabably not, at least for now.  The .elc format for compiled functions
> has a single entry for "each" symbol in its constant vector, and it
> would be quite an exercise to augment that with symbols-with-position.
>
> Maybe for the future.

Positions in error messages would be a major improvement as well.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Gemini.

On Mon, Nov 26, 2018 at 17:07:07 -0800, Gemini Lasswell wrote:
> Alan Mackenzie <acm@muc.de> writes:

> Hi Alan,

> > Mainly because of EQ, the C macro corresponding to `eq'.  In master, EQ
> > just does a binary comparison between two Lisp_Objects, and that's it.
> > In scratch/accurate-warning-pos, should that binary comparison fail, EQ
> > additionally has to check the flag variable `symbols-with-pos-enabled'
> > to see whether it needs to do any additional comparisons.  Except while
> > byte compiling, it won't, but the check of that variable is what is
> > causing the slowdown.

> I'm having trouble accepting that this is the problem.

> When I hear a story about a C program, where adding an always failing
> check of a boolean variable to it causes it to run slower by 15% or 20%,
> then when I read the code I expect to find that variable check inside a
> really tight loop with only a few other instructions besides the
> variable check.

> But Fforward_word is significantly slower in your branch and I can't
> find any tight loops with an EQ in it.  exec_byte_code might fit that
> description, if you gave it a Lisp function with a loop that hammered on
> 'eq', but that doesn't describe fill-region-as-paragraph, which is 15%
> slower.  Could there be anything else causing a slowdown besides EQ?

In ..../src/lisp.h, we have:

    INLINE bool
    (NILP) (Lisp_Object x)
    {
      return lisp_h_NILP (x);
    }

, and earlier on in the file, we have

    #define lisp_h_NILP(x) EQ (x, Qnil).

So every NILP is substituted by EQ (x, Qnil).  There are a fair number
of NILPs in loops in Fforward_char and its subroutines.

> >> Another related question: this work is for warnings, but will it
> >> extend to having positions in backtraces?
> >
> > I hadn't actually got to thinking about that, but the answer is
> > proabably not, at least for now.  The .elc format for compiled functions
> > has a single entry for "each" symbol in its constant vector, and it
> > would be quite an exercise to augment that with symbols-with-position.
> >
> > Maybe for the future.

> Positions in error messages would be a major improvement as well.

Yes.  I wonder why this wasn't done long ago.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

On 11/26/18 10:43 AM, Alan Mackenzie wrote:
> read the thread "Thoughts on getting correct line
> numbers in the byte compiler's warning messages", which began here on
> 2018-11-01.

I read that thread, but to be honest my eyes began glazing over. I'm 
sure I missed some points.

> The slowdown seems to be chrystallising out at 20 - 25% while byte
> compiling, and 5 - 20% otherwise.
>
> This may be significant, but is it really important?  Most of the time,
> Emacs is waiting for the next key depression anyway.

And most of the time, my office at work is empty. That doesn't mean I 
don't care how well my office performs when a half-dozen people squeeze 
into it and try to get work done (which happened multiple times 
today...). Emacs regularly reacts too slowly for me when I use it in 
ordinary interaction, and I'd rather not see it get significantly slower.


> As a data point, my 1.5 year old machine is about 150% faster (per core)
> than its predecessor.
My desktop uses a circa 2010 design and is no doubt significantly slower 
than your machine. No doubt that partly explains why I'm more sensitive 
about performance issues.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

On 11/26/18 12:04 PM, Stefan Monnier wrote:
> You going to have to define "the interpreter".
> Is the code of Fmemq part of your definition of "the interpreter"?

It would have to be, yes. I take your point that the project would 
require quite a few mechanical changes to the interpreter.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Eli Zaretskii]

> From: Gemini Lasswell <gazally@runbox.com>
> Cc: Clément Pit-Claudel <cpitclaudel@gmail.com>,
>   michael_heerdegen@web.de,  Eli Zaretskii <eliz@gnu.org>,  Paul Eggert
>  <eggert@cs.ucla.edu>,  monnier@IRO.UMontreal.CA,  emacs-devel@gnu.org
> Date: Mon, 26 Nov 2018 17:07:07 -0800
> 
> When I hear a story about a C program, where adding an always failing
> check of a boolean variable to it causes it to run slower by 15% or 20%,
> then when I read the code I expect to find that variable check inside a
> really tight loop with only a few other instructions besides the
> variable check.
> 
> But Fforward_word is significantly slower in your branch and I can't
> find any tight loops with an EQ in it.

That could be rarely_quit, which is called inside the inner loops of
scan_words.  But this must be verified, e.g., by running Emacs under
'perf' and examining the profile.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Paul.

On Mon, Nov 26, 2018 at 18:48:49 -0800, Paul Eggert wrote:
> On 11/26/18 10:43 AM, Alan Mackenzie wrote:
> > read the thread "Thoughts on getting correct line
> > numbers in the byte compiler's warning messages", which began here on
> > 2018-11-01.

> I read that thread, but to be honest my eyes began glazing over. I'm 
> sure I missed some points.

:-)  Yes, I get that too, sometimes.

Perhaps the main conclusion we came to was that it is far better working
with symbols than with conses.  Even after macro-expansion, and
processing by byte-opt.el and cconv.el, the symbols persist.  Mostly.
The cons cells don't, not at all.

Creating the double Lisp interpreter is orthogonal to this choice
between symbols and conses.  i would recommend you to use the mechanism
in scratch/accurate-warning-pos in your new version.  It works, and all
you have to do is get over your distaste for EQ ignoring source
positions when comparing symbols.

> > The slowdown seems to be chrystallising out at 20 - 25% while byte
> > compiling, and 5 - 20% otherwise.
> >
> > This may be significant, but is it really important?  Most of the time,
> > Emacs is waiting for the next key depression anyway.

> And most of the time, my office at work is empty. That doesn't mean I 
> don't care how well my office performs when a half-dozen people squeeze 
> into it and try to get work done (which happened multiple times 
> today...). Emacs regularly reacts too slowly for me when I use it in 
> ordinary interaction, and I'd rather not see it get significantly slower.

OK, that's a valid statement.

> > As a data point, my 1.5 year old machine is about 150% faster (per core)
> > than its predecessor.
> My desktop uses a circa 2010 design and is no doubt significantly slower 
> than your machine. No doubt that partly explains why I'm more sensitive 
> about performance issues.

Not really.  My previous machine was from 2009, and I don't recall
problems with sluggishness.  Possibly slowness in batch-like operations,
yes, but not sluggishness.  Maybe your brain just works faster than mine
does.

Emacs used to work OK on machines 10, 100 times slower than what we have
today.  Were there really that many complaints about its speed?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

On 11/26/18 11:43 PM, Alan Mackenzie wrote:
> Emacs used to work OK on machines 10, 100 times slower than what we have today.

That was back before Emacs did things like automatic syntax highlighting. Today's Emacs would have been rejected for use on those older machines.

Alas, we cannot assume that single-threaded computation will continue to 
improve at the same rate it improved in the past, as the single-threaded 
performance-vs-time curve is flattening out. See, for example:

https://imgur.com/a/FVLjs

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Paul.

On Tue, Nov 27, 2018 at 12:27:36 -0800, Paul Eggert wrote:
> On 11/26/18 11:43 PM, Alan Mackenzie wrote:
> > Emacs used to work OK on machines 10, 100 times slower than what we have today.

> That was back before Emacs did things like automatic syntax highlighting. Today's Emacs would have been rejected for use on those older machines.

> Alas, we cannot assume that single-threaded computation will continue to 
> improve at the same rate it improved in the past, as the single-threaded 
> performance-vs-time curve is flattening out. See, for example:

> https://imgur.com/a/FVLjs

That's all pretty much irrelevant to the task at hand.  The fact is,
Emacs users will be using computers of a wide range of power.  Probably
as much as a factor of 10.  Compared with this, a 10%, or even a 20%
slowdown, while not being good, will hardly be noticed.

Anyhow, as far as creating a better solution than
scratch/accurate-warning-pos goes, have you had any further thoughts?

I've thought about the double interpreter idea, and must confess I've no
idea how one might go about implementing it in any reasonable time.
grep reveals approximately 6728 instances of EQ/NILP/SYMBOLP/XSYMBOL in
98 out of 132 .c files, and all of these instances are compiled in,
rather than being function calls.  Are you still serious about
implementing this?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

On 11/27/18 1:15 PM, Alan Mackenzie wrote:
> Anyhow, as far as creating a better solution than
> scratch/accurate-warning-pos goes, have you had any further thoughts?

Hash tables for cons locations would work well enough, if someone had 
the time to put into it. The main objection seems to be that the 
byte-compiler would lose track of location information. But we can fix 
the byte-compiler so that it doesn't lose track of the location 
information. That's purely a byte-compiler issue, which is a good thing 
for software engineering reasons as well as for performance reasons.

> I've thought about the double interpreter idea, and must confess I've no
> idea how one might go about implementing it in any reasonable time.

There's a simpler way to do a double interpreter. Use a compile-time 
test to decide whether eq operates slowly. Build two copies of Emacs, 
the slow one and the fast one, by compiling the slow one with -DSLOW_EQ 
(or whatever). Use the slow Emacs to byte-compile Emacs (and any other 
program that wants to be byte-compiled); this will output nicer 
diagnostics. Use the fast Emacs for everything else.

I'm not saying I *like* this solution (I don't). But it would address 
the problem of poor diagnostic locations when building Emacs, and it 
shouldn't slow Emacs down one bit in ordinary use.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Paul.

On Tue, Nov 27, 2018 at 13:37:29 -0800, Paul Eggert wrote:
> On 11/27/18 1:15 PM, Alan Mackenzie wrote:
> > Anyhow, as far as creating a better solution than
> > scratch/accurate-warning-pos goes, have you had any further thoughts?

> Hash tables for cons locations would work well enough, if someone had 
> the time to put into it.

I don't think they would.  Macros would mess things up.  For example,
after expanding a macro like the contrived

   (defmacro dehash (form)
     (copy-tree form))

, how would the recorded cons locations be linked up with the resulting
form?

> The main objection seems to be that the byte-compiler would lose track
> of location information. But we can fix the byte-compiler so that it
> doesn't lose track of the location information. That's purely a
> byte-compiler issue, which is a good thing for software engineering
> reasons as well as for performance reasons.

I don't think this is right.  See above.

> > I've thought about the double interpreter idea, and must confess I've no
> > idea how one might go about implementing it in any reasonable time.

> There's a simpler way to do a double interpreter. Use a compile-time 
> test to decide whether eq operates slowly. Build two copies of Emacs, 
> the slow one and the fast one, by compiling the slow one with -DSLOW_EQ 
> (or whatever). Use the slow Emacs to byte-compile Emacs (and any other 
> program that wants to be byte-compiled); this will output nicer 
> diagnostics. Use the fast Emacs for everything else.

> I'm not saying I *like* this solution (I don't). But it would address 
> the problem of poor diagnostic locations when building Emacs, and it 
> shouldn't slow Emacs down one bit in ordinary use.

Of course, the "slow" Emacs would somehow need to be used by the byte
compiler after Emacs has been built, to continue to get accurate warning
locations.  I don't like this approach either.  ;-)

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Stefan Monnier]

> Hash tables for cons locations would work well enough, if someone had the
> time to put into it.

Experiment with this approach seemed to indicate that it would slowdown
compilation tremendously (not just a few percents) because of the cost
of all those hash table operations.

> The main objection seems to be that the byte-compiler would lose track
> of location information.  But we can fix the byte-compiler so that it
> doesn't lose track of the location information.  That's purely
> a byte-compiler issue, which is a good thing for software engineering
> reasons as well as for performance reasons.

I agree that making modifications to the byte-compiler is fundamentally
what we want to do.  My current thinking is that we could try to do
something like the following:

`read-with-positions` could return annotated-sexps that would look like:

    (locinfo FILE POS (foo (locinfo FILE POS a) (locinfo FILE POS 4)))

for an input like

    (foo a 4)

It would significantly increase the size of the sexp and hence incur
a slowdown, but it won't be nearly as bad as with the hash-table hack.
Preservation of this info through the byte-compiler itself wouldn't
take too much effort (most of the code can treat `locinfo` as a kind of
function which returns its 3 argument).

The main problem is for macro expansion: when we call a macro we can't
pass it these annotated sexps, so we have to de-annotate the sexp
arguments, which is both extra work and a loss of important info.

So I was thinking of reducing the pain by re-using the edebug info so as
to find those arguments (or parts of arguments) which are treated as
normal expressions, which we don't need to de-annotate.

FWIW, these are just idle thoughts.  I don't really have the intention
to work on it in the foreseeable future.


        Stefan

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Gemini Lasswell]

Alan Mackenzie <acm@muc.de> writes:

> On Tue, Nov 27, 2018 at 12:27:36 -0800, Paul Eggert wrote:
>> On 11/26/18 11:43 PM, Alan Mackenzie wrote:
>> > Emacs used to work OK on machines 10, 100 times slower than what we have today.
>
>> That was back before Emacs did things like automatic syntax
>> highlighting. Today's Emacs would have been rejected for use on
>> those older machines.
>
>> Alas, we cannot assume that single-threaded computation will continue to
>> improve at the same rate it improved in the past, as the single-threaded
>> performance-vs-time curve is flattening out. See, for example:
>
>> https://imgur.com/a/FVLjs
>
> That's all pretty much irrelevant to the task at hand.  The fact is,
> Emacs users will be using computers of a wide range of power.  Probably
> as much as a factor of 10.  Compared with this, a 10%, or even a 20%
> slowdown, while not being good, will hardly be noticed.

It might not be if you're running emacs -Q.  But I'd rather we make
Emacs core as fast as possible so that the package authors who give us
org-mode and fuzzy search and in-buffer completion can make their
packages do more before responsiveness becomes a problem.

Also, new Spectre variants keep making the news, as well as news
of the performance costs of their mitigation in the kernel.  CPU
performance might be going backwards for a while.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

On 11/27/18 1:53 PM, Alan Mackenzie wrote:
> Macros would mess things up. For example, after expanding a macro like 
> the contrived
>     (defmacro dehash (form)
>       (copy-tree form))
>
> , how would the recorded cons locations be linked up with the resulting
> form?

People typically don't write macros like that, so we needn't tune our 
design for it. More commonly, people write macros containing errors, 
such as:

   (defmacro face-attribute-specified-or (value default &rest body)
     (let ((temp (make-symbol "value")))
       `(or
          (let ((,temp ,value))
            (if (eq ,temp 'unspecified)
                (progn ,@body)
              default))
          ,temp)))

This has a parenthesization typo and the macro doesn't mean what the 
user probably intended. For these macros, as I understand it the 
symbols-with-pos approach messes up because the symbol associated with 
the error was built by make-symbol and doesn't have a source code 
position. In contrast, the cons-with-pos approach should return a good 
location in the macro itself for the undeclared symbol (i.e., just 
before the last ",temp").

One can also come up with examples where the symbols-with-pos approach 
is better than the cons-with-pos approach. That is, neither approach 
dominates the other. So, which approach would be better in practice? I 
don't think we know.

> the "slow" Emacs would somehow need to be used by the byte compiler 
> after Emacs has been built, to continue to get accurate warning locations.

That's easy enough: the fast Emacs can fork and exec the slow one, which 
can be installed next to movemail etc. (Again, not that I like this 
approach.)

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

On 11/27/18 2:09 PM, Stefan Monnier wrote:
>> Hash tables for cons locations would work well enough, if someone had the
>> time to put into it.
> Experiment with this approach seemed to indicate that it would slowdown
> compilation tremendously (not just a few percents) because of the cost
> of all those hash table operations.

That's too bad. Why is hashing so slow? Can we speed it up?

Another possibility is to have a special hashing mechanism just for 
pairs that are allocated to be hashable. This would work by having a 
special alternative to struct cons_block for which it would be trivial 
to convert the address of a hashable pair to the address of the 
corresponding value (simply add a constant to the address, say). The new 
'read' function could return hashable pairs. A hashable pair would have 
the same structure as an ordinary pair so all existing functions 
(including C macros) would work with it. This should be fast enough.


> So I was thinking of reducing the pain by re-using the edebug info so as
> to find those arguments (or parts of arguments) which are treated as
> normal expressions, which we don't need to de-annotate.
Hmm, I know little about edebug so I'm afraid this suggestion is mostly 
Greek to me.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Stefan Monnier]

> That's too bad. Why is hashing so slow? Can we speed it up?

Possibly.

> Another possibility is to have a special hashing mechanism just for pairs
> that are allocated to be hashable. This would work by having a special
> alternative to struct cons_block for which it would be trivial to convert
> the address of a hashable pair to the address of the corresponding value
> (simply add a constant to the address, say). The new 'read' function could
> return hashable pairs. A hashable pair would have the same structure as an
> ordinary pair so all existing functions (including C macros) would work with
> it. This should be fast enough.

If we only do it for cons cells, then we don't need hashing: we can make
"fat-cons-cells" which contain extra position info after car/cdr.

>> So I was thinking of reducing the pain by re-using the edebug info so as
>> to find those arguments (or parts of arguments) which are treated as
>> normal expressions, which we don't need to de-annotate.
> Hmm, I know little about edebug so I'm afraid this suggestion is mostly
> Greek to me.

E.g. for a macro like `when` where the edebug spec is just `t` (which
means all the arguments are Elisp expressions) we don't need to
de-annotate anything at all.  And for `dolist` whose edebug spec is
((symbolp form &optional form) body), we can use that debug spec to find
that the first arg needs to be partially de-annotated, and the rest can
be left fully annotated.


        Stefan

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

Stefan Monnier wrote:
> If we only do it for cons cells, then we don't need hashing: we can make
> "fat-cons-cells" which contain extra position info after car/cdr.

Yes, that's basically what I was proposing. Also, we'd probably need a boolean 
flag in each struct cons_block so that C (or maybe Lisp) code can quickly 
distinguish between a fat pair and a thin (normal) one, which should also be 
cheap and easy to do. (Not that I'm volunteering to do it right now....)

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Gemini Lasswell]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> The main problem is for macro expansion: when we call a macro we can't
> pass it these annotated sexps, so we have to de-annotate the sexp
> arguments, which is both extra work and a loss of important info.
>
> So I was thinking of reducing the pain by re-using the edebug info so as
> to find those arguments (or parts of arguments) which are treated as
> normal expressions, which we don't need to de-annotate.

The trouble with using the edebug info is that there's a lot of it
that's wrong.

I spent some time last year trying to get Emacs's test suite running
under Testcover, which uses Edebug's instrumentation, and found a lot of
incorrect Edebug specs.

I reported maybe 30 of them, 8 are still on the buglist, and I have a
list of about 50 files which if instrumented cause tests to fail, which
are probably mostly Edebug spec bugs, but I haven't gone through the
time-consuming process of tracking down and reporting them yet.  And
that's just in the approximately 1/3 of all the Lisp files in Emacs
which get loaded by the test suite.

But in thinking about this, I've had an idea for how it could work:

What if the byte compiler first compiles as usual, and when it finds a
function with warnings or errors it doesn't issue them immediately, but
does another pass where it rereads the function (and maybe also any
macros it uses which are defined in the same file) with Edebug or with
Stefan's read-with-position + de-annotate, and compiles the result
(throwing away the byte-compiled code from this pass), using the
instrumentation/annotation to track source position.

If it gets the same warnings/errors the second time, it can issue them
with a more accurate position.  If it gets a different warning or error
on the second pass, that would indicate an Edebug spec problem.

Unlike my Testcover project, broken Edebug specs wouldn't lead to broken
code, because the byte compiler would always use the bytecode it made on
the first pass.  In this scheme, the only thing that gets slower is the
byte-compilation of functions with warnings or errors.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Gemini Lasswell]

Stefan Monnier <monnier@IRO.UMontreal.CA> writes:

> E.g. for a macro like `when` where the edebug spec is just `t` (which
> means all the arguments are Elisp expressions) we don't need to
> de-annotate anything at all.  And for `dolist` whose edebug spec is
> ((symbolp form &optional form) body), we can use that debug spec to find
> that the first arg needs to be partially de-annotated, and the rest can
> be left fully annotated.

Why not just use Edebug's reader?  It's already solved the de-annotation
problem.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Eli Zaretskii]

> Cc: Eli Zaretskii <eliz@gnu.org>, michael_heerdegen@web.de,
>  cpitclaudel@gmail.com, monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Tue, 27 Nov 2018 13:37:29 -0800
> 
> There's a simpler way to do a double interpreter. Use a compile-time 
> test to decide whether eq operates slowly. Build two copies of Emacs, 
> the slow one and the fast one, by compiling the slow one with -DSLOW_EQ 
> (or whatever). Use the slow Emacs to byte-compile Emacs (and any other 
> program that wants to be byte-compiled); this will output nicer 
> diagnostics. Use the fast Emacs for everything else.

So end-users will also build 2 binaries, and use the other one to
byte-compile their Lisp programs and Emacs Lisp code they've changed?

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Paul

On Tue, Nov 27, 2018 at 17:11:48 -0800, Paul Eggert wrote:
> On 11/27/18 1:53 PM, Alan Mackenzie wrote:
> > Macros would mess things up. For example, after expanding a macro like 
> > the contrived
> >     (defmacro dehash (form)
> >       (copy-tree form))

> > , how would the recorded cons locations be linked up with the resulting
> > form?

> People typically don't write macros like that, ....

For the pertinent value of "like", they do.  The essential feature of
that macro is that it returns a list form whose conses are not the input
conses.

In a typical macro, only some of the conses survive the processing.  For
example, every time the macro result is something like:

    `(progn ,@(mapcar 'foo body))

, the conses in the return list are new, and the old conses are lost.
Look at cconv-closure-convert.  I know this isn't, strictly speaking, a
macro, but it rearranges code the way macros do.  In its output there
isn't a single cons which was present in the input.

> .... so we needn't tune our design for it.

It's not a matter of "tuning" the design.  It's a matter of having a
scheme which will work at all.

> .... More commonly, people write macros containing errors, such as:

>    (defmacro face-attribute-specified-or (value default &rest body)
>      (let ((temp (make-symbol "value")))
>        `(or
>           (let ((,temp ,value))
>             (if (eq ,temp 'unspecified)
>                 (progn ,@body)
>               default))
>           ,temp)))

> This has a parenthesization typo and the macro doesn't mean what the 
> user probably intended. For these macros, as I understand it the 
> symbols-with-pos approach messes up because the symbol associated with 
> the error was built by make-symbol and doesn't have a source code 
> position.

It doesn't "mess up".  It's that the compiler isn't designed to give
warning messages about already compiled macros being used, and this is
the case regardless of whether symbols-with-pos or conses-with-pos is
used.  To change this is a separate question which we should deal with
separately, if at all.

> In contrast, the cons-with-pos approach should return a good location
> in the macro itself for the undeclared symbol (i.e., just before the
> last ",temp").

No.  Assuming the macro is compiled, just where are these cons positions
going to be stored?  Byte compiled functions/macros don't consist of
conses.  Merely evaluated functions/macros do consist of conses, but
they also consist of symbols, so ons-with-pos doesn't come out ahead,
here.

Anyhow, this is all a diversion from the main point, which is how to
generate warning positions for the source currently being compiled.

> One can also come up with examples where the symbols-with-pos approach 
> is better than the cons-with-pos approach. That is, neither approach 
> dominates the other.

You're wrong, here.  Symbols-with-pos works on the output from macros.
Conses-with-pos will only do so for some "nice" macros, ones which
preserve their input conses.

> So, which approach would be better in practice? I don't think we know.

Well, in practice, the symbols-with-pos approach exists and works, even
if it is currently unfinished.  cons-with-pos has been tried, by me, and
never reached anything like a working version.  I'm convinced the idea
can't work.

Maybe it needs somebody more capable than me to implement cons-with-pos.

#########################################################################

> > the "slow" Emacs would somehow need to be used by the byte compiler 
> > after Emacs has been built, to continue to get accurate warning locations.

> That's easy enough: the fast Emacs can fork and exec the slow one, which 
> can be installed next to movemail etc. (Again, not that I like this 
> approach.)

Thinking about it, what other approach is there?  Because of the ~6728
macro calls of EQ, NILP, .... scattered throughout the Emacs sources,
almost all that source will need to be compiled twice, and one way or
another be made available to users in two forms.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Stefan Monnier]

> The trouble with using the edebug info is that there's a lot of it
> that's wrong.

I was looking at it from the other side: it would be an opportunity to
make sure they're right ;-)

But yes, some transition would be needed.


        Stefan

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

On 11/27/18 10:28 PM, Eli Zaretskii wrote:
> So end-users will also build 2 binaries, and use the other one to
> byte-compile their Lisp programs and Emacs Lisp code they've changed?

That's the most obvious way to do it, yes. That is, we currently build 
Emacs three times (temacs, bootstrap-emacs, emacs) and we'd add a fourth 
one and install it.

(Again, not that I think this is a good idea....)

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Eli Zaretskii]

> Cc: acm@muc.de, michael_heerdegen@web.de, cpitclaudel@gmail.com,
>  monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> From: Paul Eggert <eggert@cs.ucla.edu>
> Date: Wed, 28 Nov 2018 14:50:36 -0800
> 
> On 11/27/18 10:28 PM, Eli Zaretskii wrote:
> > So end-users will also build 2 binaries, and use the other one to
> > byte-compile their Lisp programs and Emacs Lisp code they've changed?
> 
> That's the most obvious way to do it, yes. That is, we currently build 
> Emacs three times (temacs, bootstrap-emacs, emacs) and we'd add a fourth 
> one and install it.

But only one gets installed currently, whereas under this proposal
there will be 2.

> (Again, not that I think this is a good idea....)

Then maybe we should stop discussing it ;-)

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Paul.

On Wed, Nov 28, 2018 at 14:50:36 -0800, Paul Eggert wrote:
> On 11/27/18 10:28 PM, Eli Zaretskii wrote:
> > So end-users will also build 2 binaries, and use the other one to
> > byte-compile their Lisp programs and Emacs Lisp code they've changed?

> That's the most obvious way to do it, yes. That is, we currently build 
> Emacs three times (temacs, bootstrap-emacs, emacs) and we'd add a fourth 
> one and install it.

> (Again, not that I think this is a good idea....)

Is the idea of the double interpreter still a live idea?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Eli Zaretskii]

> Date: Thu, 29 Nov 2018 10:54:34 +0000
> From: Alan Mackenzie <acm@muc.de>
> Cc: michael_heerdegen@web.de, Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
> 	cpitclaudel@gmail.com, monnier@IRO.UMontreal.CA
> 
> Is the idea of the double interpreter still a live idea?

If you are asking me, then I'm not sure what is the idea of the double
interpreter.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Eli.

On Thu, Nov 29, 2018 at 13:13:17 +0200, Eli Zaretskii wrote:
> > Date: Thu, 29 Nov 2018 10:54:34 +0000
> > From: Alan Mackenzie <acm@muc.de>
> > Cc: michael_heerdegen@web.de, Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org,
> > 	cpitclaudel@gmail.com, monnier@IRO.UMontreal.CA
> > 
> > Is the idea of the double interpreter still a live idea?

> If you are asking me, then I'm not sure what is the idea of the double
> interpreter.

I was really asking Paul, since it was originally his idea.

The "double interpreter" would be when we'd have one software instance
running at full speed, on which everything but byte compilation would be
done, and a second instance, using the ideas from branch
scratch/accurate-warning-pos, on which byte compilation would be done.

For some value of "instance".

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Some real world timings

[Alan Mackenzie]

Hello, Eli.

On Fri, Nov 23, 2018 at 13:09:05 +0000, Alan Mackenzie wrote:

[ .... ]

> The branch scratch/accurate-warning-pos now bootstraps.  (It also now
> runs on optimised builds. ;-)

> Its bootstrap is a mere 7.1% slower than master's.

Over the last week or so, there have been some wildly different timings
posted about the slowdown caused by scratch/accurate-warning-pos.

I have measured this on some "real world" situations - actions which
could be or are sluggish in interactive or "semi-batch" operation.  Here
are the results:

(i) time-scroll (see below) on the biggest .el file,
.../leim/ja-dic/ja-dic.el.  This measures mainly fontification:
  o - Slowdown of 4.9% and 5.1%.

(ii) Indenting entire buffer progmodes/cc-engine.el, after having
removed all space in the LH margin:
  o - Slowdown of 3.4%

(iii) Indenting entire buffer src/xdisp.c, after having removed all
space in the LH margin:
  o - Slowdown of 46%.  (This is anomalous, and not yet understood.)

(iv) Indenting first 50% of buffer src/xdisp.c, after having removed all
space in the LH margin:
  o - Slowdown of 4%, 7.3%.

(v) Same, but last 50% of src/xdisp.c, ...
  o - Slowdown of 10.8%, 9.4%.

(vi) time-scroll, measuring mainly fontification on src/xdisp.c:
  o - Slowdown of 3.4%, 3.6%.


These results were measured by the macro time-it:

(defmacro time-it (&rest forms)
  "Time the running of a sequence of forms using `float-time'.
Call like this: \"M-: (time-it (foo ...) (bar ...) ...)\"."
  `(let ((start (float-time)))
    ,@forms
    (- (float-time) start)))

The first and last of the timings were measured by time-scroll:

(defun time-scroll (&optional arg)
  (interactive "P")
  (message "%s"
           (time-it
            (condition-case nil
                (while t
                  (if arg (scroll-down) (scroll-up))
                  (sit-for 0))
              (error nil)))))


CONCLUSION: On the "real world" operations of fontification and
indentation, scratch/accurate-warning-pos is around 3.5% - 10% slower
than master.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

On 11/29/18 3:37 AM, Alan Mackenzie wrote:
> I was really asking Paul, since it was originally his idea.

I don't like the idea of having two instances of Emacs installed, a 
"main" one and a "debugging" used only for byte-compilation. I suggested 
the idea only because it's better than having only a "debugging" Emacs 
that is slower for everything.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

On Thu, Nov 29, 2018 at 13:12:36 -0800, Paul Eggert wrote:
> On 11/29/18 3:37 AM, Alan Mackenzie wrote:
> > I was really asking Paul, since it was originally his idea.

> I don't like the idea of having two instances of Emacs installed, a 
> "main" one and a "debugging" used only for byte-compilation. I suggested 
> the idea only because it's better than having only a "debugging" Emacs 
> that is slower for everything.

A better description than "debugging" Emacs would be "debugged" Emacs;
it's the one whose compiler works.  The "main" one's compiler is buggy.

The degree of slow down is minimal in practical use.

Again, the question, Paul: is the idea of the two instances of code
still live?

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

On 11/28/18 4:04 AM, Alan Mackenzie wrote:

> Assuming the macro is compiled, just where are these cons positions
> going to be stored?

The same place we store the rest of compiled output. That is, when we 
byte-compile macros, we store the uncompiled macros (with source-pos 
info) into the .elc file. Then, any code that wants accurate line number 
or similar debugging information for the macro can use the uncompiled 
version. It's pretty common to bloat object files to support debugging 
in this way.

This approach wouldn't work for .elc files generated by older versions 
of Emacs, but that's OK; the old macros should still work, it's just 
that the byte-compiler's diagnostics will be just as bad as before.

(Sorry, I don't know how edebug works so I don't know how this would 
affect edebug.)

> You're wrong, here.  Symbols-with-pos works on the output from macros.
> Conses-with-pos will only do so for some "nice" macros, ones which
> preserve their input conses.
I'm afraid we'll have to disagree here, as I can see examples where 
symbols-with-pos fails and conses-with-pos works (as well as vice 
versa). The only good way I see to resolve the disagreement would be to 
build it both ways and see which typically works better in the real 
world. A daunting prospect, admittedly.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Paul.

On Thu, Nov 29, 2018 at 13:28:06 -0800, Paul Eggert wrote:
> On 11/28/18 4:04 AM, Alan Mackenzie wrote:

> > Assuming the macro is compiled, just where are these cons positions
> > going to be stored?

> The same place we store the rest of compiled output. That is, when we 
> byte-compile macros, we store the uncompiled macros (with source-pos 
> info) into the .elc file. Then, any code that wants accurate line number 
> or similar debugging information for the macro can use the uncompiled 
> version. It's pretty common to bloat object files to support debugging 
> in this way.

> This approach wouldn't work for .elc files generated by older versions 
> of Emacs, but that's OK; the old macros should still work, it's just 
> that the byte-compiler's diagnostics will be just as bad as before.

The above is completely irrelevant to the question at hand.  We're not
talking about macros generating messages for invalid input.  we're
talking about generating message positions for the source being
compiled.

[ .... ]

> > You're wrong, here.  Symbols-with-pos works on the output from macros.
> > Conses-with-pos will only do so for some "nice" macros, ones which
> > preserve their input conses.

> I'm afraid we'll have to disagree here, as I can see examples where 
> symbols-with-pos fails ....

I can't.  Please give an example, so I can see what you're talking about.

> .... and conses-with-pos works (as well as vice versa). The only good
> way I see to resolve the disagreement would be to build it both ways
> and see which typically works better in the real world. A daunting
> prospect, admittedly.

Symbols-with-pos exists, works, and works well.

Conses-with-pos doesn't exist, hence doesn't work.  I doubt it can work.
Can you create it?

To create scratch/accurate-warning-pos took a fair amount of work, but
not that much.  You can work much faster than I can, so why don't you
hack up a version of the byte-compiler which uses positions of conses to
track source positions?  Bring it to the point where it can successfully
bootstrap Emacs.  Then we can compare the two approaches.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

On 11/29/18 2:05 PM, Alan Mackenzie wrote:
> We're not
> talking about macros generating messages for invalid input.  we're
> talking about generating message positions for the source being
> compiled.

I'm not sure what you mean here. The point of this exercise, as I 
understand it, is to generate better diagnostics for Lisp code that has 
problems in it (or is "invalid input" in some sense). Such code may 
define macros, and it may use those macros or may use macros defined 
elsewhere. If you're saying that we should talk only about problems in 
the part of the code that uses macros, and not about problems in the 
part of the code that defines macros, then that sounds like a 
conversation that would be too narrow.

>
>> I'm afraid we'll have to disagree here, as I can see examples where
>> symbols-with-pos fails ....
> I can't.  Please give an example, so I can see what you're talking about.

I attempted to give one in 
<https://lists.gnu.org/r/emacs-devel/2018-11/msg00530.html>, but you 
dismissed it as not relevant to the discussion. As mentioned above, I 
don't see why it's irrelevant.


> Symbols-with-pos exists, works, and works well.
> Conses-with-pos doesn't exist, hence doesn't work.  I doubt it can work.
> Can you create it?

I think I could, with time. I hope someone else will do it, though, as 
the problem does not seem that urgent. I would suggest that whoever does 
it, looks into Stefan's ideas in this area, e.g., 
<https://lists.gnu.org/r/emacs-devel/2018-11/msg00532.html>.

> is the idea of the two instances of code
> still live?
It's live only in the sense that it's better than 
scratch/accurate-warning-pos because of performance issues. I'd rather 
that we did something better than either.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Paul.

On Fri, Nov 30, 2018 at 09:50:09 -0800, Paul Eggert wrote:
> On 11/29/18 2:05 PM, Alan Mackenzie wrote:
> > We're not talking about macros generating messages for invalid
> > input.  we're talking about generating message positions for the
> > source being compiled.

> I'm not sure what you mean here. The point of this exercise, as I 
> understand it, is to generate better diagnostics for Lisp code that has 
> problems in it (or is "invalid input" in some sense).

Then you've completely misunderstood.

The point of the exercise is specific, not vague as you have
misformulated it.  It is to print the correct source locations in
compiler warnings which are currently being output as wrong locations.
For example, read bug #9109.

> Such code may define macros, and it may use those macros or may use
> macros defined elsewhere. If you're saying that we should talk only
> about problems in the part of the code that uses macros, and not about
> problems in the part of the code that defines macros, then that sounds
> like a conversation that would be too narrow.

Feel free to have that wider conversation, somewhere.  I might even join
in.  It is orthogonal to the current set of bugs.  

> >
> >> I'm afraid we'll have to disagree here, as I can see examples where
> >> symbols-with-pos fails ....
> > I can't.  Please give an example, so I can see what you're talking about.

> I attempted to give one in 
> <https://lists.gnu.org/r/emacs-devel/2018-11/msg00530.html>, but you 
> dismissed it as not relevant to the discussion. As mentioned above, I 
> don't see why it's irrelevant.

I hope you now see why it is irrelevant, and will take back the
insinuation that symbols-with-pos fails, in the sense of bug #9101.

> > Symbols-with-pos exists, works, and works well.  Conses-with-pos
> > doesn't exist, hence doesn't work.  I doubt it can work.  Can you
> > create it?

> I think I could, with time.

But you're not going to?

> I hope someone else will do it, though, as the problem does not seem
> that urgent. I would suggest that whoever does it, looks into Stefan's
> ideas in this area, e.g.,
> <https://lists.gnu.org/r/emacs-devel/2018-11/msg00532.html>.

Nobody else is going to do it, and why should they?  I've already
explained to you why the approach you suggest won't work, you being
about the only person insisting that it will.  If you actually tried it,
as I have, you'd soon find out that I'm right, and why.

And if we only fixed "urgent" bugs, not a lot would get done on Emacs.
These bugs have been hanging around Emacs for many, many years, and it's
about high time they finally got fixed.  I have fixed them.

> > is the idea of the two instances of code still live?
> It's live only in the sense that it's better than
> scratch/accurate-warning-pos because of performance issues. I'd rather
> that we did something better than either.

There are no performance "issues".  Emacs with this bug fix is minutely
slower than without it, not enough to matter.  If you insist there are
"issues", then please perform some timings and report them here.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

On 11/30/18 10:55 AM, Alan Mackenzie wrote:
> There are no performance "issues".  Emacs with this bug fix is minutely
> slower than without it, not enough to matter.

I'm afraid we'll have to disagree about that. It's not worth slowing all 
Emacs computations by 3.5% - 10% or whatever, merely to get 
more-accurate locations in byte-compiler diagnostics. That is a 
significant slowdown, and it's too much of a price to pay for such a 
small benefit.

I too have played with the idea to redefine eq, for a different purpose 
(so that bignums behave more like fixnums), and I gave it up because of 
similar slowdowns. It wasn't worth the cost there either. Sometimes 
ideas just don't work out.

> The point of the exercise is specific, not vague as you have
> misformulated it.  It is to print the correct source locations in
> compiler warnings which are currently being output as wrong locations.
> For example, read bug #9109.
Bug #9109 could be fixed by a read variant that returns 
symbols-with-positions, by changing the byte-compiler to understand the 
output of this read variant, and by having the byte compiler remove 
position information before passing a subexpression to a macro. It could 
also be fixed by a read variant that returns conses-with-positions. 
Neither fix would slow down ordinary code. However, as I understand it, 
we don't want either fix because these fixes would still generate subpar 
locations in diagnostics when byte-compiling other code that calls 
macros. This is why I have been thinking about other code errors that 
involve macros, errors that you say are out of scope.

When trying to consider alternatives to scratch/accurate-warning-pos, I 
was trying to get an understanding of problem scope that is independent 
of the proposed solution. Unfortunately I failed to do that, and this 
makes it difficult to consider alternatives.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Paul.

On Fri, Nov 30, 2018 at 12:14:59 -0800, Paul Eggert wrote:
> On 11/30/18 10:55 AM, Alan Mackenzie wrote:
> > There are no performance "issues".  Emacs with this bug fix is minutely
> > slower than without it, not enough to matter.

> I'm afraid we'll have to disagree about that. It's not worth slowing all 
> Emacs computations by 3.5% - 10% or whatever, merely to get 
> more-accurate locations in byte-compiler diagnostics.

Please stop being so disparaging.  There's nothing "mere" about accurate
compiler messages, and the current bug fix delivers CORRECT locations,
not "more-accurate" ones.

> That is a significant slowdown, .... 

No it isn't.  We could sit you down in front of each version of Emacs,
get you to play with them, and you would be unable to distinguish them
without using a machine to do timings.

> .... and it's too much of a price to pay for such a small benefit.

You think having compiler messages whose locations simulate a random
number generator is acceptable?  Remember, in a bootstrap, only 25% of
these locations are correct.  People notice, and it makes us maintainers
look incompetent and stupid.

> I too have played with the idea to redefine eq, for a different purpose 
> (so that bignums behave more like fixnums), and I gave it up because of 
> similar slowdowns. It wasn't worth the cost there either.

No doubt, you found some alternative.

What you're saying is that being fast is more important than working
properly.  Taking that attitude to extremes, Emacs could be considerably
speeded up by removing some safety and sanity checks.

> Sometimes ideas just don't work out.

You're not willing to work on an alternative, though.  You weren't even
in on the conversation when we were discussing alternatives (or, more
precisely, the lack of alternatives).

> > The point of the exercise is specific, not vague as you have
> > misformulated it.  It is to print the correct source locations in
> > compiler warnings which are currently being output as wrong locations.
> > For example, read bug #9109.

> Bug #9109 could be fixed by a read variant that returns
> symbols-with-positions, by changing the byte-compiler to understand
> the output of this read variant, and by having the byte compiler
> remove position information before passing a subexpression to a macro.

That wouldn't work.  Macro expansion is done at an early stage of
compilation.  You can't remove arbitrary position information and still
expect to get accurate (or indeed any) positions in compiler messages.

> It could also be fixed by a read variant that returns
> conses-with-positions.

Only by rewriting the compiler such that that information is somehow
preserved as compilation progresses.  As long as you don't have any
macros which massage the code, that is.

> Neither fix would slow down ordinary code.

These were all considered earlier on, and dismissed as impractical or
non-working.

> However, as I understand it, we don't want either fix because these
> fixes would still generate subpar locations in diagnostics when
> byte-compiling other code that calls macros.

If by "subpar" you mean no location at all (beyond, perhaps, the name of
the top-level function), then subpar describes it well.  And you're
talking about most code, typical code, here.  What proportion of defuns
doesn't contain a `when'?  Even `defun' itself is a macro.  ;-(

> This is why I have been thinking about other code errors that involve
> macros, errors that you say are out of scope.

They are out of scope, yes.  Expanding the scope into vague abstract
territory is an excellent way of postponing a solution for ever.

> When trying to consider alternatives to scratch/accurate-warning-pos, I 
> was trying to get an understanding of problem scope that is independent 
> of the proposed solution. Unfortunately I failed to do that, and this 
> makes it difficult to consider alternatives.

Eli, Stefan and I had a wide ranging discussion about the possibilities.
If there had been alternatives, likely we would have stumbled across
them.  Maybe we missed something.  Feel free to take up that other
thread again if you've got anything to add.

But at the moment, scratch/accurate-warning-pos is looking like the only
pracaticable solution to these many bugs.  It is a bad solution, yes,
but there are no good ones, and it is the least bad from what's
available.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

On 11/30/18 2:02 PM, Alan Mackenzie wrote:
>> I'm afraid we'll have to disagree about that. It's not worth slowing all
>> Emacs computations by 3.5% - 10% or whatever, merely to get
>> more-accurate locations in byte-compiler diagnostics.
> Please stop being so disparaging.  There's nothing "mere" about accurate
> compiler messages, and the current bug fix delivers CORRECT locations,
> not "more-accurate" ones.

I value performance and the error message location issues don't bother 
me much in practice, perhaps partly because I tend to break Lisp code 
into smaller pieces so the exact location of the error doesn't matter 
that much. Evidently your preferences differ: the location issues bother 
you, and your machine is faster and/or you don't mind the performance 
hit. That doesn't make you right and me wrong, or vice versa. It's a 
matter of balancing priorities.


> Remember, in a bootstrap, only 25% of
> these locations are correct.  People notice, and it makes us maintainers
> look incompetent and stupid.

My impression is more that it's the presence of the diagnostics, not the 
line numbers in the diagnostics, that are makes us developers look 
"incompetent and stupid" (I would say "spread too thin" but maybe that's 
just me :-).

And even if this were a public-relations issue that could be fixed by 
proper line numbers (an assertion I'm skeptical of), I proposed a 
solution that will work for this particular issue and won't involve 
slowing down Emacs significantly in ordinary use, namely, use a separate 
Emacs binary for batch byte compiling when bootstrapping.

>> I too have played with the idea to redefine eq, for a different purpose
>> (so that bignums behave more like fixnums), and I gave it up because of
>> similar slowdowns. It wasn't worth the cost there either.
> No doubt, you found some alternative.

Sure, and my alternative then was to live with Emacs as it is. That's 
preferable to making Emacs significantly slower for a relatively minor 
benefit.


>
>> Bug #9109 could be fixed by a read variant that returns
>> symbols-with-positions, by changing the byte-compiler to understand
>> the output of this read variant, and by having the byte compiler
>> remove position information before passing a subexpression to a macro.
> That wouldn't work.

It would work for the test case in Bug#9101. It no doubt would not work 
for some other examples. The point is that it's not clear (to me, at 
least) what the scope of the problem is, and we don't seem to be making 
progress on delineating the scope, and without knowing the scope it's 
hard to suggest or evaluate alternative solutions.


> scratch/accurate-warning-pos is looking like the only
> pracaticable solution to these many bugs.  It is a bad solution, yes,
> but there are no good ones, and it is the least bad from what's
> available.
>
All in all the status quo is preferable to scratch/accurate-warning-pos. 
Although I don't expect you to agree with me on this, I don't think I'm 
alone in having concerns about the significant performance issues here.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Gemini Lasswell]

Alan Mackenzie <acm@muc.de> writes:

> Hello, Paul.
>
> On Fri, Nov 30, 2018 at 09:50:09 -0800, Paul Eggert wrote:
>> On 11/29/18 2:05 PM, Alan Mackenzie wrote:
>> > Symbols-with-pos exists, works, and works well.  Conses-with-pos
>> > doesn't exist, hence doesn't work.  I doubt it can work.  Can you
>> > create it?
>
>> I think I could, with time.
>
> But you're not going to?
>
>> I hope someone else will do it, though, as the problem does not seem
>> that urgent. I would suggest that whoever does it, looks into Stefan's
>> ideas in this area, e.g.,
>> <https://lists.gnu.org/r/emacs-devel/2018-11/msg00532.html>.
>
> Nobody else is going to do it, and why should they?  I've already
> explained to you why the approach you suggest won't work, you being
> about the only person insisting that it will.  If you actually tried it,
> as I have, you'd soon find out that I'm right, and why.

Saying that nobody will do it is a bit strong.  I like Stefan's ideas,
understand edebug specs, and would like to know more about how the byte
compiler works.  This seems like a perfect project for that.  But it
wouldn't happen immediately, as I'd like to get my half-finished
benchmarking tool to a state where it's usable by other people, before I
start any other projects.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[martin rudalics]

 > All in all the status quo is preferable to
 > scratch/accurate-warning-pos. Although I don't expect you to agree
 > with me on this, I don't think I'm alone in having concerns about
 > the significant performance issues here.

There are around 50 issues in current Emacs that annoy me at least as
much as wrong positions reported by the byte compiler.  If, on the
average, solving any such issue took away just 2% of the performance
of my Emacs, I'd experience an overall slowdown of 100% when solving
all of them.  So please note that Paul is not alone with his concerns.

martin

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Eli Zaretskii]

> Date: Sat, 01 Dec 2018 08:35:54 +0100
> From: martin rudalics <rudalics@gmx.at>
> Cc: michael_heerdegen@web.de, Eli Zaretskii <eliz@gnu.org>,
> 	cpitclaudel@gmail.com, monnier@IRO.UMontreal.CA, emacs-devel@gnu.org
> 
> So please note that Paul is not alone with his concerns.

No, he certainly isn't.

In fact, there's an overwhelming majority here that considers any
slowdown above 1% unacceptable.  The fact that CPUs get faster, albeit
not at the same rate as 10 years ago, is not perceived as an
opportunity to perform more expensive processing in newer Emacs
versions, it is perceived as an opportunity to do the same tasks
faster.

If that were not the majority view, we would have had Emacs without
dumping many moons ago.  So the community clearly prefers performance
over features, even very important features.

Actually, I'm surprised how 10 years ago was I given a go-ahead to
make the Emacs display engine bidi-aware: that definitely slows down
redisplay, sometimes by as much as 50%.  Maybe I was just lucky.

Alan, unless your changes cause more than 1% slowdown, there's no hope
of them being accepted.  And even for 1%, expect to hear quite a few
negative opinions.  You *have* been warned.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Eli.

On Sat, Dec 01, 2018 at 10:25:53 +0200, Eli Zaretskii wrote:
> > Date: Sat, 01 Dec 2018 08:35:54 +0100
> > From: martin rudalics <rudalics@gmx.at>
> > Cc: michael_heerdegen@web.de, Eli Zaretskii <eliz@gnu.org>,
> > 	cpitclaudel@gmail.com, monnier@IRO.UMontreal.CA, emacs-devel@gnu.org

> > So please note that Paul is not alone with his concerns.

> No, he certainly isn't.

> In fact, there's an overwhelming majority here that considers any
> slowdown above 1% unacceptable.  The fact that CPUs get faster, albeit
> not at the same rate as 10 years ago, is not perceived as an
> opportunity to perform more expensive processing in newer Emacs
> versions, it is perceived as an opportunity to do the same tasks
> faster.

The bit you're glossing over is that we're not talking about adding some
marginal feature, we're talking about fixing a bug which has been
reported by users many times, a bug which is in a central part of
Emacs's functionality.

I'm shocked over people's preference for buggy software.

> If that were not the majority view, we would have had Emacs without
> dumping many moons ago.  So the community clearly prefers performance
> over features, even very important features.

We're not talking about features, we're talking about a bug.

> Actually, I'm surprised how 10 years ago was I given a go-ahead to
> make the Emacs display engine bidi-aware: that definitely slows down
> redisplay, sometimes by as much as 50%.  Maybe I was just lucky.

> Alan, unless your changes cause more than 1% slowdown, there's no hope
> of them being accepted.

Eli, I think it's up to you to accept or reject this bug fix.  It's your
personal responsibility.  I'd be grateful if you would make the decision
and explicitly state it.  I will accept and respect it.  If you reject
the fix, the bug is highly unlikely ever to be fixed.

> And even for 1%, expect to hear quite a few negative opinions.  You
> *have* been warned.

And there's no indication that any of these people complaining, with the
exception of Gemini Lasswell, have actually fired up the software and
tried it out.  There's no indication any of these people have even
understood the bug and the attempts to fix it, let alone tried out the
fix to evaluate it.  The views of those who took the trouble to report
the bug haven't been heard.

Suppose we don't fix this bug.  Then we should close all the pertinent
bug reports as "won't fix".  Additionally, the documentation of the byte
compiler needs to be amended to match the reality.  I propose the
following patch to .../doc/lispref/compile.texi:



diff --git a/doc/lispref/compile.texi b/doc/lispref/compile.texi
index 6d21ca3e6a..c4b47dbfdc 100644
--- a/doc/lispref/compile.texi
+++ b/doc/lispref/compile.texi
@@ -438,8 +438,12 @@ Compiler Errors
 
   Error and warning messages from byte compilation are printed in a
 buffer named @file{*Compile-Log*}.  These messages include file names
-and line numbers identifying the location of the problem.  The usual
-Emacs commands for operating on compiler output can be used on these
+and line numbers which try to identify the location of the problem.
+Frequently a wrong line number is indicated, but it is rarely very far
+away from the actual problem@footnote{A fix for this problem was
+worked out in 2018, but since it would have caused a slight slowdown
+in Emacs, it was decided not to incorporate it.}.  The usual Emacs
+commands for operating on compiler output can be used on these
 messages.
 
   When an error is due to invalid syntax in the program, the byte


-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Eli Zaretskii]

> Date: Sat, 1 Dec 2018 11:08:17 +0000
> Cc: martin rudalics <rudalics@gmx.at>, eggert@cs.ucla.edu,
>   michael_heerdegen@web.de, cpitclaudel@gmail.com, monnier@IRO.UMontreal.CA,
>   emacs-devel@gnu.org
> From: Alan Mackenzie <acm@muc.de>
> 
> Eli, I think it's up to you to accept or reject this bug fix.

Sorry, no.  My personal preference should be quite clear from what I
wrote, but last time I tried something like that, I got a s**tstorm.
Evidently, my prestige is not high enough to get something like that
through.  So I'm not going to try doing that again, not here anyway.

> It's your personal responsibility.

There cannot be a responsibility without the authority to go with it.

> Suppose we don't fix this bug.  Then we should close all the pertinent
> bug reports as "won't fix".  Additionally, the documentation of the byte
> compiler needs to be amended to match the reality.  I propose the
> following patch to .../doc/lispref/compile.texi:

Fine with me, thanks.  Although I'd recommend amending the footnote to
be less specific to when the fix was proposed.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Gemini.

On Fri, Nov 30, 2018 at 16:26:58 -0800, Gemini Lasswell wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > Hello, Paul.

> > On Fri, Nov 30, 2018 at 09:50:09 -0800, Paul Eggert wrote:
> >> On 11/29/18 2:05 PM, Alan Mackenzie wrote:
> >> > Symbols-with-pos exists, works, and works well.  Conses-with-pos
> >> > doesn't exist, hence doesn't work.  I doubt it can work.  Can you
> >> > create it?

> >> I think I could, with time.

> > But you're not going to?

> >> I hope someone else will do it, though, as the problem does not seem
> >> that urgent. I would suggest that whoever does it, looks into Stefan's
> >> ideas in this area, e.g.,
> >> <https://lists.gnu.org/r/emacs-devel/2018-11/msg00532.html>.

> > Nobody else is going to do it, and why should they?  I've already
> > explained to you why the approach you suggest won't work, you being
> > about the only person insisting that it will.  If you actually tried it,
> > as I have, you'd soon find out that I'm right, and why.

> Saying that nobody will do it is a bit strong.

OK.

> I like Stefan's ideas, understand edebug specs, and would like to know
> more about how the byte compiler works.

I'll admit I didn't follow these ideas completely.  But I don't think
they addressed the central problem, namely how to associate source
buffer positions with the form being compiled as it flows through sets
of conses on its journey from source to byte code.

> This seems like a perfect project for that.

It's a frustrating and difficult project.  I've been trying to fix it,
on and off, since 2016.  ;-)

> But it wouldn't happen immediately, as I'd like to get my
> half-finished benchmarking tool to a state where it's usable by other
> people, before I start any other projects.

By the way, if I haven't already said it, I like this tool, despite the
timings it gave my software.  Thanks!

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Martin.

On Sat, Dec 01, 2018 at 08:35:54 +0100, martin rudalics wrote:
>  > All in all the status quo is preferable to
>  > scratch/accurate-warning-pos. Although I don't expect you to agree
>  > with me on this, I don't think I'm alone in having concerns about
>  > the significant performance issues here.

> There are around 50 issues in current Emacs that annoy me at least as
> much as wrong positions reported by the byte compiler.

Likely, none of them are as difficult to address as the byte compiler
bug.  But who's going to address them, if they can expect the reaction
I'm getting at the moment?

> If, on the average, solving any such issue took away just 2% of the
> performance of my Emacs, I'd experience an overall slowdown of 100%
> when solving all of them.  So please note that Paul is not alone with
> his concerns.

On average, solving these other bugs will cost 0% in performance.
You're positing a completely unrealistic scenario.

Have you even tried scratch/accurate-warning-pos?  You've said in the
past that you have a slow machine, so if that's still true, you would
probably be the person to notice perceptible slowdown, should there be
any.  Do you notice any slowdown with it?

The byte compiler bug is extremely unusual, possibly unique, in its
resistance to being resolved.  Just about every possible approach has
been tried (along with several which are not possible), and only one
approach, the one in scratch/accurate-warning-pos, has got anywhere at
all.

If you still object to this fix, even after trying it, what you are
saying is that you prefer fast buggy software over slightly slower
functional software.

> martin

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[martin rudalics]

Hello, Alan

 >> There are around 50 issues in current Emacs that annoy me at least as
 >> much as wrong positions reported by the byte compiler.
 >
 > Likely, none of them are as difficult to address as the byte compiler
 > bug.

Running an incremental copying collector in a separate thread is, for
example.

 > But who's going to address them, if they can expect the reaction
 > I'm getting at the moment?

I've been following this thread since its beginning and I've not seen
an unjustified reaction so far, including mine.  I've kept quiet when
the 'open-paren-in-column-0-is-defun-start' convention was abolished
recently.  And I would have kept quiet in the case at hand as well.
But Eli once said that people should complain _before_ a change is
made and so I did.

 >> If, on the average, solving any such issue took away just 2% of the
 >> performance of my Emacs, I'd experience an overall slowdown of 100%
 >> when solving all of them.  So please note that Paul is not alone with
 >> his concerns.
 >
 > On average, solving these other bugs will cost 0% in performance.
 > You're positing a completely unrealistic scenario.

Right.  But mainly so because many of these 50 problems should have a
solution in the opposite direction.  They aim at improving performance
rather than degrading it.  For example, I spent quite some time on
reducing the overhead associated with creating Emacs tooltip frames -
largely unnoticed, but it saves me an entire GC cycle for every
tooltip displayed here.  Why should I spend my time on such things if
any gains are annihilated by solutions like yours.

 > Have you even tried scratch/accurate-warning-pos?  You've said in the
 > past that you have a slow machine, so if that's still true, you would
 > probably be the person to notice perceptible slowdown, should there be
 > any.  Do you notice any slowdown with it?

I haven't tried it.  As a rule, I bootstrap each of my Emacs instances
once a year, after Glenn changed the file headers.  Here a bootstrap
means that I can't use my machine for more than an hour.  And that I
have to either make a separate copy of my Emacs tree or do another
bootstrap to return to the previous version.  This year I already made
one extra bootstrap for the portable dumper and that consumed enough
of my remaining energy for such endeavours.  You have to live with the
fact that people like me live in a different world.

 > The byte compiler bug is extremely unusual, possibly unique, in its
 > resistance to being resolved.  Just about every possible approach has
 > been tried (along with several which are not possible), and only one
 > approach, the one in scratch/accurate-warning-pos, has got anywhere at
 > all.
 >
 > If you still object to this fix, even after trying it, what you are
 > saying is that you prefer fast buggy software over slightly slower
 > functional software.

I don't say that.  IMO the "byte compiler bug" has a simple fix which,
however, nobody but me would accept: When there is the slightest doubt
do not print position indications.  Also I don't understand why it
should be the byte compiler's task to check the syntax of expressions
for correctness.  If the byte compiler raises an error or new Lisp
code should be compiled, run a separate syntax checker that gets as
close as possible to the source of any error in the code.  And it can
spend as much time on that as it needs.  But enough heresy here.

martin

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Stefan Monnier]

[ FWIW, my Helvetic origins want me to stay neutral on this issue.  ]

> Also I don't understand why it should be the byte compiler's task to
> check the syntax of expressions for correctness.

I can answer this part: it's simply because most of the work needed to
perform such checks is already done by the compiler for other reasons,
so it's easy to do it within the compiler.

If you want to do it elsewhere, you'll need to either duplicate some of
the compiler's code, or refactor it so it can be used for both.
Refactoring is the ideal solution since it makes it possible to do yet
more things (e.g. think of the Clang project as an example, compared to
having it buried within LLVM), but it can be a lot of work.

In any case, such a refactoring wouldn't make a difference to the
problem at hand: the problem is not "the byte-compiler needs line
numbers" but "the code that emits the warnings needs line numbers" and
then how to make the line-information available to it and make this
information survive through macro-expansion (even though the macros are
not prepared to manipulate annotated source code).


        Stefan

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Eli Zaretskii]

> Date: Sat, 01 Dec 2018 15:09:48 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: cpitclaudel@gmail.com, Paul Eggert <eggert@cs.ucla.edu>, 
>  michael_heerdegen@web.de, emacs-devel@gnu.org, monnier@IRO.UMontreal.CA, 
>  Eli Zaretskii <eliz@gnu.org>
> 
> IMO the "byte compiler bug" has a simple fix which, however, nobody
> but me would accept: When there is the slightest doubt do not print
> position indications.

IMO, that would be a change for the worse: currently we at least have
a position near the locus of the problem, which does limit the search
to some extent.

> Also I don't understand why it should be the byte compiler's task to
> check the syntax of expressions for correctness.

Because that's what any other compiler out there does?

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[martin rudalics]

 > the problem is not "the byte-compiler needs line
 > numbers" but "the code that emits the warnings needs line numbers" and

If "the code that emits the warnings needs line numbers" and getting
that code right is hard, then why burden the byte compiler with it and
indirectly the code it produces?  And if we do insist on burdening the
byte compiler, then why don't we make that at least optional with the
help of some flags?

I do byte compile code to make Emacs run faster.  If I do not need
fast code, I run it interpreted.  But probably I'm missing something
all too obvious here.

martin

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[martin rudalics]

 >> IMO the "byte compiler bug" has a simple fix which, however, nobody
 >> but me would accept: When there is the slightest doubt do not print
 >> position indications.
 >
 > IMO, that would be a change for the worse: currently we at least have
 > a position near the locus of the problem, which does limit the search
 > to some extent.

Yes and I know that everybody else agrees with you.  But I replied to
Alan's remark that I "prefer fast buggy software over slightly slower
functional software".

 >> Also I don't understand why it should be the byte compiler's task to
 >> check the syntax of expressions for correctness.
 >
 > Because that's what any other compiler out there does?

Doesn't "any other compiler out there" also leave me the choice what I
want to check?

martin

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Eli Zaretskii]

> Date: Sat, 01 Dec 2018 17:31:26 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: cpitclaudel@gmail.com, eggert@cs.ucla.edu, michael_heerdegen@web.de, 
>  emacs-devel@gnu.org, monnier@IRO.UMontreal.CA, acm@muc.de
> 
>  >> Also I don't understand why it should be the byte compiler's task to
>  >> check the syntax of expressions for correctness.
>  >
>  > Because that's what any other compiler out there does?
> 
> Doesn't "any other compiler out there" also leave me the choice what I
> want to check?

You can ask for more warnings, yes.  You can usually also ask for no
warnings at all, but that's not the default nowadays, not usually.

And in any case, do we have line numbers with errors right?  I thought
error messages also display inaccurate line numbers, for the same
basic reason.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Martin.

On Sat, Dec 01, 2018 at 15:09:48 +0100, martin rudalics wrote:
> Hello, Alan

>  >> There are around 50 issues in current Emacs that annoy me at least as
>  >> much as wrong positions reported by the byte compiler.

>  > Likely, none of them are as difficult to address as the byte compiler
>  > bug.

> Running an incremental copying collector in a separate thread is, for
> example.

Possibly.  Has anybody raised a bug report concerning (the lack of) an
i.c.c.?

>  > But who's going to address them, if they can expect the reaction
>  > I'm getting at the moment?

> I've been following this thread since its beginning and I've not seen
> an unjustified reaction so far, including mine.

The reaction to my work on this bug, AFTER spending many many days
hacking it has been uniformly hostile.  Not one reply has said "thanks
for fixing this bug", or anything like it.  I'm trying to think of a
reply which even acknowledged that a bug, a difficult bug, has been
fixed.

Not one reply has offered to try to help optimise the new code, rather
than discarding it.

I'm not entirely sure either that those criticizing entirely understand
what has been fixed.  Paul E. admitted as much in his latest post.

> I've kept quiet when the 'open-paren-in-column-0-is-defun-start'
> convention was abolished recently.  And I would have kept quiet in the
> case at hand as well.  But Eli once said that people should complain
> _before_ a change is made and so I did.

Nobody complained before I put in 50 - 100 hours of work fixing the bug.
Or if they did, I wasn't listening.  Perhaps if they had, I might have
not bothered fixing it, given that people don't really seem to want it
fixed anyway.

>  >> If, on the average, solving any such issue took away just 2% of the
>  >> performance of my Emacs, I'd experience an overall slowdown of 100%
>  >> when solving all of them.  So please note that Paul is not alone with
>  >> his concerns.

>  > On average, solving these other bugs will cost 0% in performance.
>  > You're positing a completely unrealistic scenario.

> Right.  But mainly so because many of these 50 problems should have a
> solution in the opposite direction.  They aim at improving performance
> rather than degrading it.  For example, I spent quite some time on
> reducing the overhead associated with creating Emacs tooltip frames -
> largely unnoticed, but it saves me an entire GC cycle for every
> tooltip displayed here.  Why should I spend my time on such things if
> any gains are annihilated by solutions like yours.

The solution to this bug would be the same regardless of who implemented
it.  The improvement in performance from your changes will be the same
regardless of whether my bug fix is merged in or not.

>  > Have you even tried scratch/accurate-warning-pos?  You've said in the
>  > past that you have a slow machine, so if that's still true, you would
>  > probably be the person to notice perceptible slowdown, should there be
>  > any.  Do you notice any slowdown with it?

> I haven't tried it.  As a rule, I bootstrap each of my Emacs instances
> once a year, after Glenn changed the file headers.  Here a bootstrap
> means that I can't use my machine for more than an hour.

And after my bug fix a bootstrap would still take "more than an hour".
You'd scarcely notice the difference.

> And that I have to either make a separate copy of my Emacs tree or do
> another bootstrap to return to the previous version.  This year I
> already made one extra bootstrap for the portable dumper and that
> consumed enough of my remaining energy for such endeavours.  You have
> to live with the fact that people like me live in a different world.

Yes.  I haven't forgotten doing overnight builds on Emacs myself.  But
you also have to live with acknowledging most people's typical setup, as
I'm sure you do.

>  > The byte compiler bug is extremely unusual, possibly unique, in its
>  > resistance to being resolved.  Just about every possible approach has
>  > been tried (along with several which are not possible), and only one
>  > approach, the one in scratch/accurate-warning-pos, has got anywhere at
>  > all.

>  > If you still object to this fix, even after trying it, what you are
>  > saying is that you prefer fast buggy software over slightly slower
>  > functional software.

> I don't say that.

You are saying that.  You're saying that the 7% - 8% or whatever slowdown
is more important to you than a bug free byte compiler.

> IMO the "byte compiler bug" has a simple fix which, however, nobody but
> me would accept: When there is the slightest doubt do not print
> position indications.

This would mean printing no positions at all.  In a bootstrap (which I've
done rather a lot of, recently) only 25% of the positions output are
correct.  The other three quarters are junk.  Likely the pattern is
repeated in compilations of users' programs.  Apparently, nobody but me
thinks this is a Bad Thing.

> Also I don't understand why it should be the byte compiler's task to
> check the syntax of expressions for correctness.  If the byte compiler
> raises an error or new Lisp code should be compiled, run a separate
> syntax checker that gets as close as possible to the source of any
> error in the code.

That wouldn't help.  The difficulty is associating a source code position
with the (read) code, which is being steadily transformed.  This would be
the same regardless of whether the byte compiler itself or some separate
bit of code does the checking.

> And it can spend as much time on that as it needs.  But enough heresy
> here.

What takes the extra time is that EQ, NILP, SYMBOLP, and XSYMBOL have
been amended to recognize, when a flag is set, that #<symbol foo at 666>
is the same as foo.  It is the testing of that flag which takes the time,
even when it is not set.

> martin

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Michael Heerdegen]

Alan Mackenzie <acm@muc.de> writes:

> The reaction to my work on this bug, AFTER spending many many days
> hacking it has been uniformly hostile.  Not one reply has said "thanks
> for fixing this bug", or anything like it.  I'm trying to think of a
> reply which even acknowledged that a bug, a difficult bug, has been
> fixed.

FWIW I really appreciate your efforts, and I hope the final state will
not be to leave the bug unfixed.

But I must also admit that the slowdown is a high price, a price that is
in a similar region of annoyance than the bug itself (depending on
personal preferences).  I really wished we could find an acceptable way
to soften the slowdown.


Michael.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Eli Zaretskii]

> Date: Sat, 1 Dec 2018 17:21:27 +0000
> Cc: cpitclaudel@gmail.com, Paul Eggert <eggert@cs.ucla.edu>,
>   michael_heerdegen@web.de, emacs-devel@gnu.org, monnier@IRO.UMontreal.CA,
>   Eli Zaretskii <eliz@gnu.org>
> From: Alan Mackenzie <acm@muc.de>
> 
> This would mean printing no positions at all.  In a bootstrap (which I've
> done rather a lot of, recently) only 25% of the positions output are
> correct.  The other three quarters are junk.  Likely the pattern is
> repeated in compilations of users' programs.  Apparently, nobody but me
> thinks this is a Bad Thing.

I do.

And if I didn't thank you for your work on this, I apologize and thank
you now.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Clément Pit-Claudel]

On 01/12/2018 12.21, Alan Mackenzie wrote:
> Not one reply has offered to try to help optimise the new code, rather
> than discarding it.

Hey Alan,

I'm very thankful for your efforts to fix this longstanding bug (incorrectly-located warnings are a continuing source of confusion for me, and positions in backtraces, which I think would be a natural extension of your work, would make a lot of debugging work easier).

On the other hand, I understand worries about performance.  Besides vim, Emacs is roughly the only full-featured editor that runs on my raspberry pi at a reasonable speed.  I'm sure there's no malice in the worries expressed by others on this list; rather, it's easy to worry that somewhat costly changes will accumulate until we get performance as bad as many other more recent IDEs.

Here's one technical question about the branch:

IIUC, your code's strategy is to attach location info to symbols as they are read; this is a clever trick, because macroexpansion tends to preserve symbols more than it preserves conses (interestingly, Racket macros are given syntax objects that include position information instead of raw SEXPs for precisely this reason).  But, again IIUC, attaching position information on symbols means that they cannot be compared using pointer equality anymore, which means that EQ must get slower.  I wonder: after byte-compiling and reporting accurately-positioned warnings, couldn't we strip position information from symbols, so that usual Emacs execution isn't affected? The idea would be to start by reading in symbols with position information, preserve this position information throughout byte-compilat
 ion, and then strip it so that the rest of the code can use raw pointer equality.

If that is still too slow, could we have two version of read, one acting just like today and one returning sexps whose symbols contain position information?  Then we would change the byte-compiler to use the latter, and use eq-with-position-information instead of eq throughout the file.  Wouldn't such a scheme make use of the code you wrote, while restricting the performance impact to the byte-compiler and leaving the rest of Emacs unchanged?

Cheers and thanks again for your work,
Clément.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Yuri Khan]

On Sun, Dec 2, 2018 at 12:51 AM Clément Pit-Claudel
<cpitclaudel@gmail.com> wrote:

> IIUC, your code's strategy is to attach location info to symbols as they are read; this is a clever trick, because macroexpansion tends to preserve symbols more than it preserves conses (interestingly, Racket macros are given syntax objects that include position information instead of raw SEXPs for precisely this reason).  But, again IIUC, attaching position information on symbols means that they cannot be compared using pointer equality anymore, which means that EQ must get slower.  I wonder: after byte-compiling and reporting accurately-positioned warnings, couldn't we strip position information from symbols, so that usual Emacs execution isn't affected? The idea would be to start by reading in symbols with position information, preserve this position information throughout byte-compilation, and then strip it so that the rest of the code can use raw pointer equality.

I am wondering about the same thing. The simplified view of the
process would be:

* Parse text into position-annotated AST
* Analyze the position-annotated AST for errors and warnings,
outputting accurate positions
* Byte-compile position-annotated AST, stripping annotations in the process

The complication is that parsing may invoke macros, some of which use
‘eq’ and cannot be recompiled to use ‘eq-with-position-information’.
*If* that latter assumption is true, this complication could be solved
by an opt-in property: A macro that has a 'position-aware property on
it can be called directly on the position-annotated AST and expected
to use ‘eq-with-position-information’; while a legacy macro will be
called on a de-annotated AST, may use plain ‘eq’, and will return a
similarly position-oblivious AST. The parser will then slap its best
guess of position on the root of the macro’s output and say “well, I
tried”.

This way, when people say “I’m getting an incorrect warning position”,
the first question will be “which macros do you use?”, and the second,
“have they been updated to support source positions?” Then people will
go ask those macros’ authors to update them, and/or write new
position-aware implementations.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello, Michael.

On Sat, Dec 01, 2018 at 18:44:27 +0100, Michael Heerdegen wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > The reaction to my work on this bug, AFTER spending many many days
> > hacking it has been uniformly hostile.  Not one reply has said "thanks
> > for fixing this bug", or anything like it.  I'm trying to think of a
> > reply which even acknowledged that a bug, a difficult bug, has been
> > fixed.

> FWIW I really appreciate your efforts, and I hope the final state will
> not be to leave the bug unfixed.

Thanks!

> But I must also admit that the slowdown is a high price, a price that is
> in a similar region of annoyance than the bug itself (depending on
> personal preferences).  I really wished we could find an acceptable way
> to soften the slowdown.

Going back to Paul's idea of a "double interpreter", how about the
following strategy (I don't know enough about the subject even to know
if it's a reasonable idea)?

We build "bulky" Emacs using the scratch/accurate-warning-pos
definitions of EQ, NILP, etc.  We then build "slim" Emacs using the
traditional definitions of EQ, NILP, etc., such that functions'
addresses are the same in both versions.

At runtime, when we want to byte compile, get the operating system to
switch the mapping of code virtual addresses to the "bulky" code
segment.  When this is complete, we switch back again.

Or something like that.

Is this feasible?

> Michael.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[martin rudalics]

 > And in any case, do we have line numbers with errors right?  I thought
 > error messages also display inaccurate line numbers, for the same
 > basic reason.

I miss you here.  Which errors and which error messages?

martin

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[martin rudalics]

 > Possibly.  Has anybody raised a bug report concerning (the lack of) an
 > i.c.c.?

I've been talking about annoyances not about bugs.  If, at all, people
would file a feature request disguised as a bug report for that.

 > The reaction to my work on this bug, AFTER spending many many days
 > hacking it has been uniformly hostile.  Not one reply has said "thanks
 > for fixing this bug", or anything like it.  I'm trying to think of a
 > reply which even acknowledged that a bug, a difficult bug, has been
 > fixed.

As someone who has been bitten by this bug occasionally I express my
thanks to you for your efforts.  I even have to admit that due to the
fact that line numbers are often reported incorrectly, I now never
bother to consult them in the first place but try to retrieve the
necessary information from any other context available.

But please do not judge my replies as "hostile".  I expressed my
concerns and I tried to do that at a sufficiently early stage in
response to similar concerns raised by Paul before.  And I tried to
provide a context in the sense that for me there are many annoyances
that bother me to the same extent (or even more) and where I never
would accept a slowdown as your reported.

For some years already I cannot run Emacs regularly from a debug build
because it can't keep up with my typing speed (which ironically also
gets slower day by day).  I can't yet imagine what to do when my
optimized builds slow down to a similar degree.

 > Not one reply has offered to try to help optimise the new code, rather
 > than discarding it.
 >
 > I'm not entirely sure either that those criticizing entirely understand
 > what has been fixed.  Paul E. admitted as much in his latest post.

I do _not_ criticize you and have no intentions to do that.  My
knowledge of the byte compiler is much too limited for that.

 >> I've kept quiet when the 'open-paren-in-column-0-is-defun-start'
 >> convention was abolished recently.  And I would have kept quiet in the
 >> case at hand as well.  But Eli once said that people should complain
 >> _before_ a change is made and so I did.
 >
 > Nobody complained before I put in 50 - 100 hours of work fixing the bug.
 > Or if they did, I wasn't listening.  Perhaps if they had, I might have
 > not bothered fixing it, given that people don't really seem to want it
 > fixed anyway.

You did not say in advance what people would have to pay for the fix.
And IIUC even you didn't know the price when you started coding.

 > The solution to this bug would be the same regardless of who implemented
 > it.  The improvement in performance from your changes will be the same
 > regardless of whether my bug fix is merged in or not.

Here I disagree.  If bug fixes are allowed to cause a significant
slowdown of Emacs, people will sooner or later stop improving its
performance or stop using it at all.  This is IMO reality and I hope
you don't dismiss it as yet another criticism.

 > And after my bug fix a bootstrap would still take "more than an hour".
 > You'd scarcely notice the difference.

Noticing differences and their cause is a process that may take
months.  Remember our ups and downs with xdisp.c.

 > Yes.  I haven't forgotten doing overnight builds on Emacs myself.  But
 > you also have to live with acknowledging most people's typical setup, as
 > I'm sure you do.

I do.  Although I'm probably too old to understand the philosophy of
using a retina display with reduced resolution when running Emacs.

 >>   > If you still object to this fix, even after trying it, what you are
 >>   > saying is that you prefer fast buggy software over slightly slower
 >>   > functional software.
 >
 >> I don't say that.
 >
 > You are saying that.  You're saying that the 7% - 8% or whatever slowdown
 > is more important to you than a bug free byte compiler.

If I can make the byte compiler bug free by not printing line numbers
when they are buggy with a probability of ...

 >> IMO the "byte compiler bug" has a simple fix which, however, nobody but
 >> me would accept: When there is the slightest doubt do not print
 >> position indications.
 >
 > This would mean printing no positions at all.  In a bootstrap (which I've
 > done rather a lot of, recently) only 25% of the positions output are
 > correct.  The other three quarters are junk.  Likely the pattern is
 > repeated in compilations of users' programs.  Apparently, nobody but me
 > thinks this is a Bad Thing.

... 75% then the first solution that comes to my mind is to not print
line numbers at all.  Then we have a "bug free compiler" (we all know
that such a thing does not exist, in reality).

martin

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Clément Pit-Claudel]

On 01/12/2018 13.26, Yuri Khan wrote:
> The complication is that parsing may invoke macros, some of which use
> ‘eq’ and cannot be recompiled to use ‘eq-with-position-information’.

Ah, that's a good point.  But maybe we could we rebind the lisp-level `eq` to `eq-with-position-information` for the duration of the byte-compilation process?

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Eli Zaretskii]

> Date: Sat, 01 Dec 2018 20:04:31 +0100
> From: martin rudalics <rudalics@gmx.at>
> CC: cpitclaudel@gmail.com, eggert@cs.ucla.edu, michael_heerdegen@web.de, 
>  emacs-devel@gnu.org, monnier@IRO.UMontreal.CA, acm@muc.de
> 
>  > And in any case, do we have line numbers with errors right?  I thought
>  > error messages also display inaccurate line numbers, for the same
>  > basic reason.
> 
> I miss you here.  Which errors and which error messages?

AFAIK, line numbers displayed with byte-compilation errors are
inaccurate like line numbers displayed with warnings.  So this is not
just about warnings, which one could disable, this is also about
errors, which definitely cannot and should not be disabled.

IOW, the issue is not so overrated.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Stefan Monnier]

>> the problem is not "the byte-compiler needs line
>> numbers" but "the code that emits the warnings needs line numbers" and
> If "the code that emits the warnings needs line numbers" and getting
> that code right is hard, then why burden the byte compiler with it and
> indirectly the code it produces?

Alan's solution doesn't burden the byte-compiler (or any other code that
would emit the warnings): it slows down execution of every other code
instead.  That's its main/sole problem (in other respects, it's
"perfect" in that it is very nicely localized).


        Stefan

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Alan Mackenzie]

Hello again, Michael.

On Sat, Dec 01, 2018 at 18:58:08 +0000, Alan Mackenzie wrote:
> On Sat, Dec 01, 2018 at 18:44:27 +0100, Michael Heerdegen wrote:
> > Alan Mackenzie <acm@muc.de> writes:

> > > The reaction to my work on this bug, AFTER spending many many days
> > > hacking it has been uniformly hostile.  Not one reply has said "thanks
> > > for fixing this bug", or anything like it.  I'm trying to think of a
> > > reply which even acknowledged that a bug, a difficult bug, has been
> > > fixed.

> > FWIW I really appreciate your efforts, and I hope the final state will
> > not be to leave the bug unfixed.

> Thanks!

> > But I must also admit that the slowdown is a high price, a price that is
> > in a similar region of annoyance than the bug itself (depending on
> > personal preferences).  I really wished we could find an acceptable way
> > to soften the slowdown.

> Going back to Paul's idea of a "double interpreter", how about the
> following strategy (I don't know enough about the subject even to know
> if it's a reasonable idea)?

[ .... ]

No, I don't think that idea (snipped) was at all good.  If it could
work, it would be a security nightmare.

Instead, how about this:

We build (part of) Emacs using the scratch/accurate-warning-pos versions
of EQ, NILP, ....  Using some object code utility, we systematically
change the names of variables, constants, ......  We note the addresses
of the subrs (of which there are a mere 1450).  Call this the "bulky"
version.  We build the "slim" Emacs, then link them together into
temacs.

At run time, when we do byte compilation, we first set the function
addresses of (most of?) the subrs to the "bulky" subrs, perform the
compilation, then restore them again.

The main disadvantage would be the increased size of temacs,
approximately doubled.  The advantage would be that "normal" Emacs would
run at "full speed".

> > Michael.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

Eli Zaretskii wrote:
>> Apparently, nobody but me
>> thinks this is a Bad Thing.
> I do.
> 
> And if I didn't thank you for your work on this, I apologize and thank
> you now.

Likewise. It's a real bug, Emacs needs to be improved in this area, and I 
appreciate Alan's looking into the problem.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Paul Eggert]

Yuri Khan wrote:
> A macro that has a 'position-aware property on
> it can be called directly on the position-annotated AST and expected
> to use ‘eq-with-position-information’; while a legacy macro will be
> called on a de-annotated AST, may use plain ‘eq’, and will return a
> similarly position-oblivious AST.

Thanks, this is a good suggestion, perhaps the best idea I've heard so far in 
this thread. Here are a few more suggestions on top of it.

1. We could have (equal A B) ignore position information in A and B; this would 
make it more convenient for macros to use existing code based on 'equal', and 
it'd mean we wouldn't need a new predicate eq-with-position-information. In one 
form of this idea, when A and B are symbols then (equal A B) acts like 
(string-equal (symbol-name A) (symbol-name B)), thus ignoring the distinction 
between interned and uninterned symbols. In another form of this idea, (equal A 
B) merely ignores position information.

2. Instead of having a bulky notation for new, position-aware macros, we could 
use a syntax that's shorter or simpler than what we have now, to make new-style 
macros more appealing. For example, we could use "defmac" to define new-style 
macros, and keep "defmacro" for old ones (we can say that the trailing "ro" 
stands for "region oblivious" :-).

3. If (2) is too drastic for you, we could have something like "(require 
'position-macros)" (or some other syntax) that tells the byte-compiler that all 
macros in this file are smart about positions; this would be more convenient 
than annotating each macro individually. Also, we could warn people that this is 
a transition approach, and that eventually the idea will be that position-macros 
will be the default and that if you want an old-style macro you need to 
explicitly annotate it as old-style.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Yuri Khan]

On Sun, Dec 2, 2018 at 4:26 AM Paul Eggert <eggert@cs.ucla.edu> wrote:

> Yuri Khan wrote:
> > A macro that has a 'position-aware property on
> > it can be called directly on the position-annotated AST and expected
> > to use ‘eq-with-position-information’; while a legacy macro will be
> > called on a de-annotated AST, may use plain ‘eq’, and will return a
> > similarly position-oblivious AST.
>
> 2. Instead of having a bulky notation for new, position-aware macros, we could
> use a syntax that's shorter or simpler than what we have now, to make new-style
> macros more appealing. For example, we could use "defmac" to define new-style
> macros, and keep "defmacro" for old ones (we can say that the trailing "ro"
> stands for "region oblivious" :-).

Yes, I was implying that the property would be set by a new
macro-defining macro or some such. The file-cookie-based idea is also
good and IIRC has been successfully used to (start to) introduce
lexical scoping.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Gemini Lasswell]

Paul Eggert <eggert@cs.ucla.edu> writes:

> Yuri Khan wrote:
>> A macro that has a 'position-aware property on
>> it can be called directly on the position-annotated AST and expected
>> to use ‘eq-with-position-information’; while a legacy macro will be
>> called on a de-annotated AST, may use plain ‘eq’, and will return a
>> similarly position-oblivious AST.
>
> Thanks, this is a good suggestion, perhaps the best idea I've heard so
> far in this thread. Here are a few more suggestions on top of it.

We don't need to invent a new property or rewrite every macro to make
macros work on position-annotated ASTs, because we already have this
feature.  It's (declare (debug ...)), aka Edebug specifications, and it
is how Edebug gets exact source positions when stepping through code
which contains macros.

Paul, you said earlier in this thread that you weren't familiar with
Edebug, so I will attempt to explain how its position annotation works.

In an Edebug specification, a macro author can specify which of the
arguments to a macro (and which structures inside those arguments) are
forms which will be evaluated, and therefore should be
position-annotated, and which are not.  For example in the form

  (dolist (VAR FORM) BODY)

FORM and BODY will be evaluated, but VAR will not be, instead it is
expected to be a symbol which will be bound in the code generated by
'dolist'.  So the Edebug spec for 'dolist' instructs Edebug to annotate
FORM and BODY and leave VAR alone.

If the macro author doesn't provide an Edebug spec, then Edebug doesn't
add position annotations to any of its arguments, and then when
debugging, Edebug will skip over those arguments to the next point of
execution which has a position annotation.

When an Edebug spec instructs it to, Edebug wraps macro arguments with
position annotations, which look just like normal function calls
(because they are) and can be treated by the macro just like any other
Lisp form destined for evaluation.  Edebug's annotated forms look like:

  (edebug-after (edebug-before POS1) POS2 FORM)

where POS1 and POS2 are source code positions for the beginning and end
of FORM (to be precise, they are indices into a vector of source
positions built by the Edebug reader).  FORM could be recursively
annotated if it contains other forms.

For those of you who may have missed it earlier, here are Stefan's
thoughts on how Edebug specs could be used by the byte compiler:
http://lists.gnu.org/archive/html/emacs-devel/2018-11/msg00526.html.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Dmitry Gutov]

On 01.12.2018 19:21, Alan Mackenzie wrote:
> The reaction to my work on this bug, AFTER spending many many days
> hacking it has been uniformly hostile.  Not one reply has said "thanks
> for fixing this bug", or anything like it.  I'm trying to think of a
> reply which even acknowledged that a bug, a difficult bug, has been
> fixed.

Alan, thanks a lot for working on this bug. Even if this particular 
approach ends up not being adopted, it's surely a step forward toward 
fixing it.

Re: scratch/accurate-warning-pos: Solid progress: the branch now bootstraps.

[Stefan Monnier]

> We don't need to invent a new property or rewrite every macro to make
> macros work on position-annotated ASTs, because we already have this
> feature.  It's (declare (debug ...)), aka Edebug specifications, and it
> is how Edebug gets exact source positions when stepping through code
> which contains macros.

This isn't quite right: e.g. relying only on the debug spec for the
`defun` macro would require to strip out the source-position annotation
from the function name, args, and docstring, which would prevent precise
position information for errors in those parts.  It's probably not
terribly important (we'd likely fallback on the position of the
open-paren that starts the `defun` which isn't too bad), but it's not as
good as what we could get by changing the macro to be annotation-aware.


        Stefan