A couple of things that I think should be in byte bytecode meta comments

A couple of things that I think should be in byte bytecode meta comments

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 353 bytes --]

The file from which the bytecode was compiled from.

A SHA of the source code that the compilation came from. That way one can
be more confident that this bytecode file goes with this particular source
code and not another file with the same name.  If the SHA is too
computationally expensive, then a checksum or even size of the file might
be helpful.

[-- Attachment #2: Type: text/html, Size: 399 bytes --]

Re: A couple of things that I think should be in byte bytecode meta comments

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

I mean file path or file name, not the file contents.

On Fri, Dec 22, 2017 at 12:54 PM, Rocky Bernstein <rocky@gnu.org> wrote:

> The file from which the bytecode was compiled from.
>
> A SHA of the source code that the compilation came from. That way one can
> be more confident that this bytecode file goes with this particular source
> code and not another file with the same name.  If the SHA is too
> computationally expensive, then a checksum or even size of the file might
> be helpful.
>

[-- Attachment #2: Type: text/html, Size: 820 bytes --]

Re: A couple of things that I think should be in byte bytecode meta comments

[Eli Zaretskii]

> From: Rocky Bernstein <rocky@gnu.org>
> Date: Fri, 22 Dec 2017 12:55:15 -0500
> 
> I mean file path or file name, not the file contents.

You mean, the absolute file name of the source file?  But then the
byte-compiled file will be unportable, as the file name will be
different on another system.

Re: A couple of things that I think should be in byte bytecode meta comments

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 1888 bytes --]

There is a bit of flexibility in how run-time systems handle denoting the
file name. One thing that is often done is give the path as it was
specified in the compile command. This could be ./byte-comp.el or
./emacs-lisp/byte-comp.el  (And from this you might see where I'm going
with this).

One run-time system for a programming language I am familiar with started
with the what I described (use path as given in invocation) and then and
switched to turning everything into an absolute path. Personally, I like
giving both when the starting point was indeed a relative path.

Now as to the portability. Yes, if the file is run on another system, the
path isn't exact. But it does give some idea of what we are talking as you
git closer to the bottom of the path and that may be helpful.

Consider cases where I have a stable and development branch and then
install into say /usr/local/share/emacs/lisp. Even though the top-level
directories are not the same, it still is useful to know where in the
source code tree (whether on my system or not).

And I recall one Emacs package that concatenated a bunch of bytecode files
xx-a.elc xx-b.elc and called the result xx.elc. Although unusual, with the
filenames in the bytecode, you can unscramble this .

And finally there will be cases where the path is exact.

In sum, just because sometimes it doesn't work out, doesn't mean it
will be *totally
*meaningless all the time. And I prefer "sometimes useful" to no
information, however accurate that is.


On Fri, Dec 22, 2017 at 1:30 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Rocky Bernstein <rocky@gnu.org>
> > Date: Fri, 22 Dec 2017 12:55:15 -0500
> >
> > I mean file path or file name, not the file contents.
>
> You mean, the absolute file name of the source file?  But then the
> byte-compiled file will be unportable, as the file name will be
> different on another system.
>

[-- Attachment #2: Type: text/html, Size: 2509 bytes --]

Re: A couple of things that I think should be in byte bytecode meta comments

[Eli Zaretskii]

> From: Rocky Bernstein <rocky@gnu.org>
> Date: Fri, 22 Dec 2017 13:49:06 -0500
> Cc: emacs-devel@gnu.org
> 
> Now as to the portability. Yes, if the file is run on another system, the path isn't exact. But it does give some
> idea of what we are talking as you git closer to the bottom of the path and that may be helpful.   
> 
> Consider cases where I have a stable and development branch and then install into say
> /usr/local/share/emacs/lisp. Even though the top-level directories are not the same, it still is useful to know
> where in the source code tree (whether on my system or not). 

I guess I'm not following you.  On my system, there are 60 files whose
absolute file names end in lisp/emacs-lisp/bytecomp.el.  (And my
equivalent of your /usr/local/share/emacs is populated with files that
came from a tree which is not the stable nor the development branches.)
Some of these 60 files come from the same versions of Emacs, some from
different versions.  How can a signature that records the absolute
file name help in distinguishing between bytecomp.elc's that were
produced from the same or identical files, and those which were
produced from different, i.e. "incompatible" files?  What am I missing
here?

> And finally there will be cases where the path is exact. 

Too few to rely on, what with today's practice of installing pre-built
packages instead of building from sources on each end-user system.

> In sum, just because sometimes it doesn't work out, doesn't mean it will be totally meaningless all the time.
> And I prefer "sometimes useful" to no information, however accurate that is. 

I'm saying that the minuscule amount of times it will work will drown
in the sea of times it won't.  Worse, when it "doesn't work", it will
many times produce a false alarm: the file name is different, but the
contents was identical.  How will you distinguish between true
negatives and false negatives?  Without a means to distinguish between
them, this feature will be worse then useless: it will be an absolute
nuisance.

Re: A couple of things that I think should be in byte bytecode meta comments

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 2495 bytes --]

On Fri, Dec 22, 2017 at 3:25 PM, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Rocky Bernstein <rocky@gnu.org>
> > Date: Fri, 22 Dec 2017 13:49:06 -0500
> > Cc: emacs-devel@gnu.org
> >
> > Now as to the portability. Yes, if the file is run on another system,
> the path isn't exact. But it does give some
> > idea of what we are talking as you git closer to the bottom of the path
> and that may be helpful.
> >
> > Consider cases where I have a stable and development branch and then
> install into say
> > /usr/local/share/emacs/lisp. Even though the top-level directories are
> not the same, it still is useful to know
> > where in the source code tree (whether on my system or not).
>
> I guess I'm not following you.  On my system, there are 60 files whose
> absolute file names end in lisp/emacs-lisp/bytecomp.el.  (And my
> equivalent of your /usr/local/share/emacs is populated with files that
> came from a tree which is not the stable nor the development branches.)
> Some of these 60 files come from the same versions of Emacs, some from
> different versions.  How can a signature that records the absolute
> file name help in distinguishing between bytecomp.elc's that were
> produced from the same or identical files, and those which were
> produced from different, i.e. "incompatible" files?  What am I missing
> here?
>

That there is also a SHA of  the text. If the text in any of those 60 files
is identical it doesn't matter for purposes of debugging and error location
determination which one
in the set you decide to call the source.



> > And finally there will be cases where the path is exact.
>
> Too few to rely on, what with today's practice of installing pre-built
> packages instead of building from sources on each end-user system.
>
> > In sum, just because sometimes it doesn't work out, doesn't mean it will
> be totally meaningless all the time.
> > And I prefer "sometimes useful" to no information, however accurate that
> is.
>
> I'm saying that the minuscule amount of times it will work will drown
> in the sea of times it won't.  Worse, when it "doesn't work", it will
> many times produce a false alarm: the file name is different, but the
> contents was identical.


If that's the case, then how is this different than what we have now?

How will you distinguish between true
> negatives and false negatives?  Without a means to distinguish between
> them, this feature will be worse then useless: it will be an absolute
> nuisance.
>

Again, the SHA.

[-- Attachment #2: Type: text/html, Size: 3542 bytes --]

Re: A couple of things that I think should be in byte bytecode meta comments

[Eli Zaretskii]

> From: Rocky Bernstein <rocky@gnu.org>
> Date: Fri, 22 Dec 2017 15:55:06 -0500
> Cc: emacs-devel@gnu.org
> 
> That there is also a SHA of  the text. If the text in any of those 60 files is identical it doesn't matter for purposes
> of debugging and error location determination which one
> in the set you decide to call the source. 

If there's a SHA, why do we also need a file name?

>  I'm saying that the minuscule amount of times it will work will drown
>  in the sea of times it won't.  Worse, when it "doesn't work", it will
>  many times produce a false alarm: the file name is different, but the
>  contents was identical. 
> 
> If that's the case, then how is this different than what we have now? 

If it isn't different, why add the recording of file names?  It does
nothing to improve the situation.

Re: A couple of things that I think should be in byte bytecode meta comments

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 1160 bytes --]

On Sat, Dec 23, 2017 at 3:25 AM, Eli Zaretskii <eliz@gnu.org> wrote:

> > From: Rocky Bernstein <rocky@gnu.org>
> > Date: Fri, 22 Dec 2017 15:55:06 -0500
> > Cc: emacs-devel@gnu.org
> >
> > That there is also a SHA of  the text. If the text in any of those 60
> files is identical it doesn't matter for purposes
> > of debugging and error location determination which one
> > in the set you decide to call the source.
>
> If there's a SHA, why do we also need a file name?
>
> >  I'm saying that the minuscule amount of times it will work will drown
> >  in the sea of times it won't.  Worse, when it "doesn't work", it will
> >  many times produce a false alarm: the file name is different, but the
> >  contents was identical.
> >
> > If that's the case, then how is this different than what we have now?
>
> If it isn't different, why add the recording of file names?  It does
> nothing to improve the situation.
>



Path names give you good places to start looking for the file.

And often they can quickly give information as to what's up, e.g. I am
running  from the stable or development branch. Or running from an Ubuntu
build or a source-code build.

[-- Attachment #2: Type: text/html, Size: 1813 bytes --]

Re: A couple of things that I think should be in byte bytecode meta comments

[Robert Weiner]

[-- Attachment #1: Type: text/plain, Size: 1078 bytes --]

On Sat, Dec 23, 2017 at 3:44 AM, Rocky Bernstein <rocky@gnu.org> wrote:

>
> Path names give you good places to start looking for the file.
>
> And often they can quickly give information as to what's up, e.g. I am
> running  from the stable or development branch. Or running from an Ubuntu
> build or a source-code build.
>

​If you are actually running a branch of Emacs and using a .elc file
included therein, then ​your load-path should be configured to take you to
the proper associated source file with the find-library command, unless
filenames are duplicated within the lisp tree.  Only uses outside of that,
i.e. working on files outside of the branch you are running, would this be
an issue, I would think, or if there ever is a bundled format of .elc that
combines the byte-compiled output of multiple files.

If anything of this nature is ever done, it should be based on the source
file's default installed location relative to the Emacs root directory for
portability.  Any reasonable function/tool could then find the source file.

Bob

[-- Attachment #2: Type: text/html, Size: 2127 bytes --]

Re: A couple of things that I think should be in byte bytecode meta comments

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 2840 bytes --]

On Sat, Dec 23, 2017 at 10:32 AM, Robert Weiner <rsw@gnu.org> wrote:

> On Sat, Dec 23, 2017 at 3:44 AM, Rocky Bernstein <rocky@gnu.org> wrote:
>
>>
>> Path names give you good places to start looking for the file.
>>
>> And often they can quickly give information as to what's up, e.g. I am
>> running  from the stable or development branch. Or running from an Ubuntu
>> build or a source-code build.
>>
>
> ​If you are actually running a branch of Emacs and using a .elc file
> included therein, then ​your load-path should be configured to take you to
> the proper associated source file with the find-library command, unless
> filenames are duplicated within the lisp tree.  Only uses outside of that,
> i.e. working on files outside of the branch you are running, would this be
> an issue, I would think, or if there ever is a bundled format of .elc that
> combines the byte-compiled output of multiple files.
>
> If anything of this nature is ever done, it should be based on the source
> file's default installed location relative to the Emacs root directory for
> portability.  Any reasonable function/tool could then find the source file.
>
> Bob
>
>

Let's back up a little. A function like byte-compile-file which is what
would store this information into the bytecode file only has the path it is
given. There are only two possible kinds of paths I know of: relative or
absolute.

You have most forcefully convinced yourself that the only sane way that
programmers or that  programs written by programmers work would be to use
relative paths. It matters not because, as I said, the function has only
what is given it.

My suggestion was that when a relative path is given, (which is always in
your world), also turn that into an absolute path and store in the bytecode
file as well.  In my experience in working in debugging, debuggers and in
showing where errors are, sometimes the absolute path can be useful in
addition to the relative path, for informative purposes.

I know others will not believe that, and furthermore claim most
emphatically that were these paths included as meta comments  in the
bytecode file, that would be either useless, confusing and harmful and who
knows what other bad things could happen.  (By the way, as something
similar, when I enter Emacs, I am shown a build string for a system is not
mine, which so far has not caused such havoc, and I even find mildly
interesting).

Ok. this was a just suggestion. That it has caused such negative reaction,
I've seen before in other venues, and I'm sorry.

For my part, I know what I can do to handle my concerns when or if I decide
to improve the error reporting and debugging situation on Emacs. Yes, I am
sorry I didn't say at the outset that this is were I envision this getting
used.

[-- Attachment #2: Type: text/html, Size: 4429 bytes --]

Re: A couple of things that I think should be in byte bytecode meta comments

[Robert Weiner]

[-- Attachment #1: Type: text/plain, Size: 2785 bytes --]

On Sat, Dec 23, 2017 at 12:16 PM, Rocky Bernstein <rocky@gnu.org> wrote:

>
> My suggestion was that when a relative path is given, (which is always in
> your world), also turn that into an absolute path and store in the bytecode
> file as well.
>

​People are just sharing their thoughts on whether that appears to be
helpful or not.  You have acknowledged that at runtime, the absolute path
may be invalid, but you have not provided your use case to show where and
when it might be helpful.

​​
>   In my experience in working in debugging, debuggers and in showing where
> errors are, sometimes the absolute path can be useful in addition to the
> relative path, for informative purposes.
>
​​
​Only when it is accurate.  But you haven't yet made a suggestion about how
to make absolute paths in .elc files accurate during runtime, as far as I
recall.
Maybe a static absolute path would provide a heuristic for finding the
run-time absolute path, maybe not.  You haven't shown anything either way
yet.

> ​​
> ​I know others will not believe that, and furthermore claim most
> emphatically that were these paths included as meta comments  in the
> bytecode file, that would be either useless, confusing and harmful and who
> knows what other bad things could happen.  (By the way, as something
> similar, when I enter Emacs, I am shown a build string for a system is not
> mine, which so far has not caused such havoc, and I even find mildly
> interesting).
>

​It is a matter of utility, not of havoc.
​

> ​Ok. this was a just suggestion. That it has caused such negative
> reaction, I've seen before in other venues, and I'm sorry.
> ​​
> For my part, I know what I can do to handle my concerns when or if I
> decide to improve the error reporting and debugging situation on Emacs.
> Yes, I am sorry I didn't say at the outset that this is were I envision
> this getting used.
>

​Yes, an easy to follow use case would be helpful.  I am very much in favor
of improving Emacs' error messages, especially anything that leads to the
source of an error when a backtrace is not produced.  If you can explain
how something you envision could make that happen, then you'll likely see
feedback change.

BTW, I think there might be a good idea in there about using hash codes to
verify valid use of a file, though my personal experience is that
incompatible byte codes are well reported by Emacs and have not caused me
any problems to date.  The much bigger problem is tracing an error raised
from C code back to the source function that raised it when running without
a C debugger active.  Without that, users can't provide much of a bug
report except possibly how to trigger the problem.

Bob

[-- Attachment #2: Type: text/html, Size: 5468 bytes --]

Re: A couple of things that I think should be in byte bytecode meta comments

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 3540 bytes --]

On Sat, Dec 23, 2017 at 1:30 PM, Robert Weiner <rsw@gnu.org> wrote:

> ​... It is a matter of utility, not of havoc.
>

Reminds me of the discussions I recall having decades ago with colleagues
who maintained that undo wasn't necessary, since they always got in the
habit of saving the file before making important changes.

​
>
>> ​Ok. this was a just suggestion. That it has caused such negative
>> reaction, I've seen before in other venues, and I'm sorry.
>> ​​
>> For my part, I know what I can do to handle my concerns when or if I
>> decide to improve the error reporting and debugging situation on Emacs.
>> Yes, I am sorry I didn't say at the outset that this is were I envision
>> this getting used.
>>
>
> ​Yes, an easy to follow use case would be helpful.
>

I've spent too much time in this thread at the expense of working in things
that will have a more direct benefit to society.  The easy-to-follow stuff
will have to be deferred.



> I am very much in favor of improving Emacs' error messages, especially
> anything that leads to the source of an error when a backtrace is not
> produced.
>

Great -  think about it it and discuss it with your colleagues. I am eager
to hear what you have to offer.

  If you can explain how something you envision could make that happen,
> then you'll likely see feedback change.
>

I made an inadvertent mistake when I made the suggestion assuming that
others' experience and background here was similar to mine. I'm sure you
are right, but as I said, right now I don't think this is an effective use
of either my or emacs-devs' limited time. So again, rather than explain, I
will defer on this.

None of the feedback here so far has been particularly helpful in getting
things done. So right now having feedback change through more verbiage
isn't important and doesn't feel like it is going to be a winner either.

Here is an analogous situation that might explain my position. Rather than
explain "undo" to a community where there are some that are more vocally
disposed to opine that frequent saves are good enough, a better approach
(and more pleasing too)  would just to create "undo" and let the feature
sell itself - or not. (I know some who are still "I don't need undo"
believers)

There are some communities where one suggests something and people respond
with the default response: is why? And there are other communities where
the response is; why not?

Because Emacs is critically used, by necessity the overwhelming and more
vocal group is in the "why?" the camp. There are other venues that have the
luxury to be more in the "why not?" camp for things like this.


> BTW, I think there might be a good idea in there about using hash codes to
> verify valid use of a file,
>

Glad to have introduced you to this idea.  I am eager to see what you do
with it.

though my personal experience is that incompatible byte codes are well
> reported by Emacs and have not caused me any problems to date.
>

This has nothing to do with incompatible byte codes. I think you are
conflating this with another thread.

The much bigger problem is tracing an error raised from C code back to the
> source function that raised it when running without a C debugger active.
> Without that, users can't provide much of a bug report except possibly how
> to trigger the problem.
>


Ok.  Think about and discuss. Again, I look forward to hearing what you
come up with.



> Bob
>
>
>

[-- Attachment #2: Type: text/html, Size: 6969 bytes --]

Re: A couple of things that I think should be in byte bytecode meta comments

[Thien-Thi Nguyen]

[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]


() Rocky Bernstein <rocky@gnu.org>
() Sat, 23 Dec 2017 21:44:34 -0500

   Because Emacs is critically used, by necessity the
   overwhelming and more vocal group is in the "why?" the camp.
   There are other venues that have the luxury to be more in the
   "why not?" camp for things like this.

I think many people are also trying to relate to you as a fellow
hacker, on a "what could possibly go wrong" level, w/ varying
bits of experience and familiarity (or lack thereof) of Emacs
conventions, that inform their instincts.

Personally, i think the ‘load’ conventions are well-entrenched
and so there is no real need to embed filename (can be reliably
inferred for "normal" usage -- people wandering into the muck
probably know what they're after, anyway, and can make do).

An argument explicitly against embedding absolute filename would
be the recent trend towards reproducible builds.  Everyone is
trying to catch up to Emacs' historical ignorance^Wbliss.  :-D

-- 
Thien-Thi Nguyen -----------------------------------------------
 (defun responsep (query)
   (pcase (context query)
     (`(technical ,ml) (correctp ml))
     ...))                              748E A0E8 1CB8 A748 9BFA
--------------------------------------- 6CE4 6703 2224 4C80 7502


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 197 bytes --]

Re: A couple of things that I think should be in byte bytecode meta comments

[Andreas Schwab]

On Dez 22 2017, Rocky Bernstein <rocky@gnu.org> wrote:

> The file from which the bytecode was compiled from.

It's always the name with 'c' stripped off.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: A couple of things that I think should be in byte bytecode meta comments

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 390 bytes --]

On Fri, Dec 22, 2017 at 4:27 PM, Andreas Schwab <schwab@linux-m68k.org>
wrote:

> On Dez 22 2017, Rocky Bernstein <rocky@gnu.org> wrote:
>
> > The file from which the bytecode was compiled from.
>
> It's always the name with 'c' stripped off.
>

Except when a project concatenates a bunch byecode files together. Also the
parent path information or lack of it can sometimes also be useful.

[-- Attachment #2: Type: text/html, Size: 811 bytes --]

Re: A couple of things that I think should be in byte bytecode meta comments

[Andreas Schwab]

On Dez 22 2017, Rocky Bernstein <rocky@gnu.org> wrote:

> Except when a project concatenates a bunch byecode files together. Also the
> parent path information or lack of it can sometimes also be useful.

That won't help if the files are moved.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: A couple of things that I think should be in byte bytecode meta comments

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 951 bytes --]

It is clear from both email messages you haven't been paying attention as
this has been discussed before, same as in your other email.

(I'll admit though that there has been far too many grammar mistakes in
what I wrote.)

I'm not interested in carrying the discussion further. I've been through
these kind of discussions on the same topics a couple of times before. And
I know what the solution is should I want to pursue it.

On Sat, Dec 23, 2017 at 3:25 AM, Andreas Schwab <schwab@linux-m68k.org>
wrote:

> On Dez 22 2017, Rocky Bernstein <rocky@gnu.org> wrote:
>
> > Except when a project concatenates a bunch byecode files together. Also
> the
> > parent path information or lack of it can sometimes also be useful.
>
> That won't help if the files are moved.
>
> Andreas.
>
> --
> Andreas Schwab, schwab@linux-m68k.org
> GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
> "And now for something completely different."
>

[-- Attachment #2: Type: text/html, Size: 1539 bytes --]