Incorrect byte compiler error/warning message positions. A possible fix.

Incorrect byte compiler error/warning message positions. A possible fix.

[Alan Mackenzie]

Hello, Emacs.

The byte compiler reports wrong source positions in its error and
warning messages; not every time, but perhaps most of the time.  There
are currently at least six open bugs for this problem, e.g. bug #22288.

I fixed the problem at the end of 2018 by introducing a new Emacs type
"symbol with position".  The position component of these symbols was
the position in the source code, generated by the reader.  Such a
position gave a correct source position for error/warning messages.

Unfortunately, this fix slowed Emacs down a little, and for this reason
it was rejected by the people here.  The bug remains unfixed.

The reason for the slowdown was the need for EQ continually to check a
flag which stated whether or not symbols with position were currently
active, which they weren't apart from when compiling.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

I've recently returned to the bug and believe the symbols with position
idea can be made to work without the slowdown in EQ.

In the new scheme it is no longer true that (eq #<foo at 20081> 'foo).
The Lisp form for a function being compiled is manipulated as at
present, only its symbols will have positions.  These will get stripped
out of the symbols before final code generation.

The biggest problem is with macros, and I think I can solve this.
Arguments to macros are used in two ways:
(i) They are tested to see what code to generate.  What I call "USE"
(ii) They are returned as generated code, e.g. ",@body".  Call this
  "MENTION".
Clearly (i) won't work if the symbols have positions, (ii) needs the
positions for possible messages.

The answer to this is, whilst compiling the macro, to keep track of
whether a use of an argument is for USE or for MENTION and to generate
code to extract the bare symbol for USE.  This will involve little more
than having two or three dynamically bound variables to be tested
throughout the byte compiler.

There are mild complications, e.g. when a function contributes towards
the MENTION.  But these can be overcome.

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

This proposed fix is going to be more complicated than the one from
three years ago.  It will be able to reuse a lot of code from that
previous fix, though.

I intend to work on this in the coming weeks.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Incorrect byte compiler error/warning message positions. A possible fix.

[Lars Ingebrigtsen]

Alan Mackenzie <acm@muc.de> writes:

> The byte compiler reports wrong source positions in its error and
> warning messages; not every time, but perhaps most of the time.  There
> are currently at least six open bugs for this problem, e.g. bug #22288.

In my experience, the vast majority of the warning messages point to the
correct position.  But, yes, it does sometimes give the wrong position.

> In the new scheme it is no longer true that (eq #<foo at 20081> 'foo).
> The Lisp form for a function being compiled is manipulated as at
> present, only its symbols will have positions.  These will get stripped
> out of the symbols before final code generation.

Hm...  interesting.  But couldn't the same stripping be done even if the
symbols are eq?  That way eq would be slower during byte compilation,
but not otherwise?

> The biggest problem is with macros, and I think I can solve this.

There's also an issue with byte-hunk-handlers and byte-optimizer forms
(etc), I'd guess?  They inspect the forms and do operations based on the
symbols.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Incorrect byte compiler error/warning message positions. A possible fix.

[Po Lu]

Lars Ingebrigtsen <larsi@gnus.org> writes:

>> The byte compiler reports wrong source positions in its error and
>> warning messages; not every time, but perhaps most of the time.  There
>> are currently at least six open bugs for this problem, e.g. bug #22288.

> In my experience, the vast majority of the warning messages point to the
> correct position.  But, yes, it does sometimes give the wrong position.

I wouldn't say the "correct position", but somewhere close to the
correct position.

It could be, for example, a few lines above where it should actually be,
when giving diagnostics for unused lexical bindings.

But it is usually close enough for people to quickly locate and solve
the error.

Re: Incorrect byte compiler error/warning message positions. A possible fix.

[Alan Mackenzie]

Hello, Lars.

On Mon, Nov 15, 2021 at 06:22:50 +0100, Lars Ingebrigtsen wrote:
> Alan Mackenzie <acm@muc.de> writes:

> > The byte compiler reports wrong source positions in its error and
> > warning messages; not every time, but perhaps most of the time.  There
> > are currently at least six open bugs for this problem, e.g. bug #22288.

> In my experience, the vast majority of the warning messages point to the
> correct position.  But, yes, it does sometimes give the wrong position.

On 2018-11-22 (before you destroyed my test dataset by fixing all the
warnings in Emacs ;-) there were 335 warnings.  81 gave the correct
location, 254 a wrong one.

> > In the new scheme it is no longer true that (eq #<foo at 20081> 'foo).
> > The Lisp form for a function being compiled is manipulated as at
> > present, only its symbols will have positions.  These will get stripped
> > out of the symbols before final code generation.

> Hm...  interesting.  But couldn't the same stripping be done even if the
> symbols are eq?  That way eq would be slower during byte compilation,
> but not otherwise?

That's what I tried three years ago.  `eq' works mainly with the EQ
macro in src/lisp.h.  It works with a straight comparison of two 64 (or
32) bit words.  When that was augmented by a test of symbol_with_pos_p
(which was always zero outside of byte compilation), the performance of
Emacs dropped that 8% - 15% that people objected to so much.

> > The biggest problem is with macros, and I think I can solve this.

> There's also an issue with byte-hunk-handlers and byte-optimizer forms
> (etc), I'd guess?  They inspect the forms and do operations based on the
> symbols.

They should be fairly easy (if, perhaps, tedious) to solve, because
everything is under our control.  It's macros where people outside of
our control do wierd and wonderful things.  I think I know how to
compile macros so that they both work, yet preserve the symbols with
position on the code they generate.  These compiled macros won't work on
earlier versions of Emacs, but that's a bridge to cross when we come to
it.

> -- 
> (domestic pets only, the antidote for overdose, milk.)
>    bloggy blog: http://lars.ingebrigtsen.no

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Incorrect byte compiler error/warning message positions. A possible fix.

[Alan Mackenzie]

Hello, Po.

On Mon, Nov 15, 2021 at 13:29:23 +0800, Po Lu wrote:
> Lars Ingebrigtsen <larsi@gnus.org> writes:

> >> The byte compiler reports wrong source positions in its error and
> >> warning messages; not every time, but perhaps most of the time.  There
> >> are currently at least six open bugs for this problem, e.g. bug #22288.

> > In my experience, the vast majority of the warning messages point to the
> > correct position.  But, yes, it does sometimes give the wrong position.

> I wouldn't say the "correct position", but somewhere close to the
> correct position.

> It could be, for example, a few lines above where it should actually be,
> when giving diagnostics for unused lexical bindings.

> But it is usually close enough for people to quickly locate and solve
> the error.

But not always.  As I said, there are at least six bug reports about
this problem.

My expectations are somewhat higher.  ;-)  How can we call ourselves
hackers, when we can't even produce a fully functional byte compiler?
I'm hoping to close this gap.

-- 
Alan Mackenzie (Nuremberg, Germany).

Re: Incorrect byte compiler error/warning message positions. A possible fix.

[Stefan Monnier]

>> In my experience, the vast majority of the warning messages point to the
>> correct position.  But, yes, it does sometimes give the wrong position.
> On 2018-11-22 (before you destroyed my test dataset by fixing all the
> warnings in Emacs ;-) there were 335 warnings.  81 gave the correct
> location, 254 a wrong one.

FWIW, I think the current code gives (statistically) slightly less bad
positions, because of changes I've made to `cconv` and `bytecomp`.
That may explain Lars's impression.

Our positions are still too often poor.  Some of us have just grown used
to missing or poor location info and don't notice them any more :-(

> They should be fairly easy (if, perhaps, tedious) to solve, because
> everything is under our control.

Agreed.

> It's macros where people outside of our control do wierd and wonderful
> things.  I think I know how to compile macros so that they both work,
> yet preserve the symbols with position on the code they generate.
> These compiled macros won't work on earlier versions of Emacs, but
> that's a bridge to cross when we come to it.

I'm curious know how you intend to make it work,


        Stefan

Re: Incorrect byte compiler error/warning message positions. A possible fix.

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Our positions are still too often poor.  Some of us have just grown used
> to missing or poor location info and don't notice them any more :-(

Yes, that might be the explanation for my impression that it generally
works.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Incorrect byte compiler error/warning message positions. A possible fix.

[Alan Mackenzie]

Hello, Stefan.

On Mon, Nov 15, 2021 at 16:19:47 -0500, Stefan Monnier wrote:
> >> In my experience, the vast majority of the warning messages point to the
> >> correct position.  But, yes, it does sometimes give the wrong position.
> > On 2018-11-22 (before you destroyed my test dataset by fixing all the
> > warnings in Emacs ;-) there were 335 warnings.  81 gave the correct
> > location, 254 a wrong one.

> FWIW, I think the current code gives (statistically) slightly less bad
> positions, because of changes I've made to `cconv` and `bytecomp`.
> That may explain Lars's impression.

> Our positions are still too often poor.  Some of us have just grown used
> to missing or poor location info and don't notice them any more :-(

I'm now fundamentally opposed to the use of terms like "poor" and
"inaccurate" here.  I prefer "wrong".  ;-)

> > They should be fairly easy (if, perhaps, tedious) to solve, because
> > everything is under our control.

> Agreed.

> > It's macros where people outside of our control do wierd and wonderful
> > things.  I think I know how to compile macros so that they both work,
> > yet preserve the symbols with position on the code they generate.
> > These compiled macros won't work on earlier versions of Emacs, but
> > that's a bridge to cross when we come to it.

> I'm curious know how you intend to make it work,

While a macro is being compiled it will use a dynamically bound variable
called, say, byte-comp-use.  When non-nil, it indicates the macro
may/should return executable code (i.e. without symbols with position).
When nil, byte-comp-use instructs the macro to return "code" containing
symbols with position, enabling the calling byte compiler to output
these positions in warning messages.

As macros call other macros, special forms in a macro will be enhanced
to bind byte-comp-use appropriately.  For example, in a progn (whose
last form is the result code), byte-comp-use will be t for all subforms
except the last.  In an `if' whose two arms are alternative result
codes, the condition form would have b-c-use t, each of the arms would
have nil.

Or something like that.

An alternative/additional idea is to have things like the evaluator
handle a symbol with position natively.  This would not slow Emacs down
the way amending EQ did three years ago; it would merely slow down the
generation of code from macros, probably not by very much.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).