Thoughts on getting correct line numbers in the byte compiler's warning messages

[Alan Mackenzie <acm@muc.de>]

Hello, Emacs.

Most of the time, the byte compiler identifies the correct place of error
in its warning messages.  This is remarkable, given the crude hack which
it uses.

However, it sometimes fails, and this has given rise to a number of bug
reports, e.g., 22288, and several others which have been merged with it.
In bug #22288:

    (defun test ()
      (let (a))
      a)

, the byte compiler correctly reports "reference to free variable 'a',
but wrongly gives the source position as L2 C9 rather than L3 C3.

The problem is that the Emacs Lisp source code being compiled is first
read, and this discards line/column numbers of the constructs created.  I
believe that, somehow, accurate source position information must be
preserved.  But how?  It is not easy.

The forms created by the reader go through several (?many) transformative
phases where they get replaced by successor forms.  This makes things
more difficult.

My first idea to track position information was for the reader to create
a hash table of conses (the key) and positions (the value), so that the
position could be found simply by accessing the entry corresponding with
the current form.  This doesn't work so easily, because of the previous
paragraph.

Then I tried duplicating a hash table entry when a transformation was
effected.  This was just too tedious and error prone, and was also slow.

Second idea was still to maintain this hash table, but on each
transformation to write the result back to the same cons cell as the
original.  I actually put quite a lot of work into this approach, but in
the end didn't get very far.  It was just too much detailed work, too
fiddly.

The third idea is to amend the reader so that whereas it now produces a
form, in a byte compiler special mode, it would produce the cons (form .
offset).  So, for example, the text "(not a)" currently gets read into
the form (not . (a . nil)).  The amended reader would produce (((not . 1)
. ((a . 5) . (nil . 6))) . 0) (where 0, 1, 5, and 6 are the textual
offsets of the elements coded).  Such forms would require special
versions of `cons', `car', `cdr', `cond', ...., `mapcar', .... to be
easily manipulable.  These versions would be macros to begin with, but
probably primitives ultimately.  Assuming appropriate design, it should
be possibly to substitute these new macros/primitives for the existing
cons/car/cdr/...s in the byte compiler without too much related change.
I'm still exploring this scheme.

I feel that this bug is not intractable, though it will take quite a lot
of work to fix.

Comments?

-- 
Alan Mackenzie (Nuremberg, Germany).