Prickliness of the "invalid byte code" stuff

Prickliness of the "invalid byte code" stuff

[Lars Ingebrigtsen]

One thing that I've always wondered about when building Emacs is that
running Emacs instances then start to fail in odd ways:

url-http-parse-headers: Invalid byte code in /home/larsi/src/emacs/trunk/lisp/emacs-lisp/cl-seq.elc [8 times]

And it's almost always with this "invalid byte code" stuff.

What does it mean, really?  Is there some sanity check to ensure that
you're not running an .elc file designed for a different Emacs build,
or...  something else?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Prickliness of the "invalid byte code" stuff

[Andreas Schwab]

On Jun 15 2019, Lars Ingebrigtsen <larsi@gnus.org> wrote:

> One thing that I've always wondered about when building Emacs is that
> running Emacs instances then start to fail in odd ways:
>
> url-http-parse-headers: Invalid byte code in /home/larsi/src/emacs/trunk/lisp/emacs-lisp/cl-seq.elc [8 times]
>
> And it's almost always with this "invalid byte code" stuff.
>
> What does it mean, really?

emacs-lisp/cl-seq.elc was modified after being loaded.  You need to
reload it.

> Is there some sanity check to ensure that you're not running an .elc
> file designed for a different Emacs build, or...  something else?

See byte-compile-dynamic.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: Prickliness of the "invalid byte code" stuff

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > url-http-parse-headers: Invalid byte code in /home/larsi/src/emacs/trunk/lisp/emacs-lisp/cl-seq.elc [8 times]
  > >
  > > And it's almost always with this "invalid byte code" stuff.
  > >
  > > What does it mean, really?

  > emacs-lisp/cl-seq.elc was modified after being loaded.  You need to
  > reload it.

I wonder if we could modify that error message so as to inform users
better about the likely cause.

The code which reports that error could check the date of the file,
determine whether the problem is due to changing the file, and
if so say so clearly.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Prickliness of the "invalid byte code" stuff

[Lars Ingebrigtsen]

Richard Stallman <rms@gnu.org> writes:

> I wonder if we could modify that error message so as to inform users
> better about the likely cause.
>
> The code which reports that error could check the date of the file,
> determine whether the problem is due to changing the file, and
> if so say so clearly.

Yes, that would be nice.

I just did a very simple test: I deleted all the .elc files and then
said `C-x v g' on a file and got the familiar "invalid byte code"
message.  At least in this instance it would be more helpful to say that
the .elc file no longer exists...  or perhaps not backtrace at all, but
just load .el file?

Debugger entered--Lisp error: (error "Invalid byte code in /home/larsi/src/emacs/trunk/lisp/emacs-lisp/cl-macs.elc")
  #f(compiled-function (arg1 &rest rest) #<bytecode 0x1563e82520e1>)((>= (point) bol))
  (cl-assert (>= (point) bol))
  (let ((bol (point)) (date (vc-call-backend vc-annotate-backend 'annotate-time)) (inhibit-read-only t)) (cl-assert (>= (point) bol)) (put-text-property bol (point) 'invisible 'vc-annotate-annotation) date)
  vc-annotate-get-time-set-line-props()
  (setq date (vc-annotate-get-time-set-line-props))
[...]

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Prickliness of the "invalid byte code" stuff

[Stefan Monnier]

> I just did a very simple test: I deleted all the .elc files and then
> said `C-x v g' on a file and got the familiar "invalid byte code"
> message.  At least in this instance it would be more helpful to say that
> the .elc file no longer exists...  or perhaps not backtrace at all, but
> just load .el file?

IIRC while some details can be improved, it's pretty hard to make it
work really well in all cases without making non-trivial changes.

But there's an easy fix for the specific problem you mention (see
attached).

Maybe this fix should be applied to all the relevant files.
[ The lack of smileys is because I'm suggesting this quite seriously:
  AFAIK this is a micro-optimization which doesn't make much sense
  nowadays anyway.  Has anyone bothered to actually measure the speed
  up it can offer?  ]


        Stefan


diff --git a/lisp/emacs-lisp/cl-macs.el b/lisp/emacs-lisp/cl-macs.el
index 24e79acfa5..7e11dd2e4a 100644
--- a/lisp/emacs-lisp/cl-macs.el
+++ b/lisp/emacs-lisp/cl-macs.el
@@ -3397,7 +3397,6 @@ cl-struct-slot-value
 (run-hooks 'cl-macs-load-hook)
 
 ;; Local variables:
-;; byte-compile-dynamic: t
 ;; generated-autoload-file: "cl-loaddefs.el"
 ;; End:
 

Re: Prickliness of the "invalid byte code" stuff

[T.V Raman]

Richard Stallman <rms@gnu.org> writes:


And if  one is developing a large package that one also uses all the
time, it would be even nicer if it just went ahead and reloaded the
file, > [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > url-http-parse-headers: Invalid byte code in /home/larsi/src/emacs/trunk/lisp/emacs-lisp/cl-seq.elc [8 times]
>   > >
>   > > And it's almost always with this "invalid byte code" stuff.
>   > >
>   > > What does it mean, really?
>
>   > emacs-lisp/cl-seq.elc was modified after being loaded.  You need to
>   > reload it.
>
> I wonder if we could modify that error message so as to inform users
> better about the likely cause.
>
> The code which reports that error could check the date of the file,
> determine whether the problem is due to changing the file, and
> if so say so clearly.

-- 

Re: Prickliness of the "invalid byte code" stuff

[Stefan Monnier]

> And if  one is developing a large package that one also uses all the
> time, it would be even nicer if it just went ahead and reloaded the
> file,

At the time when we realize that we need to load the bytecode, we could
indeed reload the file, but we'd still be in the middle of trying to run
the bytecode of a function from the old file and we wouldn't know which
function this is (we only have at hand the byte-code-function-p object,
not the corresponding symbol) so we wouldn't know how to redirect the
call to the newly loaded version of the function.


        Stefan

Re: Prickliness of the "invalid byte code" stuff

[T.V Raman]

Not quite sure I understood all that --- though I am sure you know what
you're saying.

From my experience, if you byte-compile an already loaded file, then
calling functions from that file raises the afore-mentioned error ---
because of byte-compile-dynamic. If you then do M-x load-library of the
recompiled file, then life continues as normal in the running
session. My thought was that if we  could detect this specific case,
then we could as well have emacs load that file, rather than having the
user reload it by hand.
-- 

Re: Prickliness of the "invalid byte code" stuff

[Stefan Monnier]

> From my experience, if you byte-compile an already loaded file, then
> calling functions from that file raises the afore-mentioned error ---
> because of byte-compile-dynamic. If you then do M-x load-library of the
> recompiled file, then life continues as normal in the running
> session. My thought was that if we  could detect this specific case,
> then we could as well have emacs load that file, rather than having the
> user reload it by hand.

Yes, that usually works, but unless we reorganize the current code, we
end up having to solve a slightly different problem similar to:

    (setq sm-foo (indirect-function '<pkg>-<fun>))
    (load "<pkg>")
    (funcall sm-foo <args>)

this will still burp because reloading <pkg> redefined <pkg>-<fun> but
it did not update `sm-foo` accordingly.


        Stefan

Re: Prickliness of the "invalid byte code" stuff

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >  My thought was that if we  could detect this specific case,
  > then we could as well have emacs load that file, rather than having the
  > user reload it by hand.

That would indeed be better, if we can DTRT _reliably_.
I don't know how hard that will turn out to be.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Prickliness of the "invalid byte code" stuff

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > session. My thought was that if we  could detect this specific case,
  > > then we could as well have emacs load that file, rather than having the
  > > user reload it by hand.

  > Yes, that usually works, but unless we reorganize the current code, we
  > end up having to solve a slightly different problem similar to:

  >     (setq sm-foo (indirect-function '<pkg>-<fun>))
  >     (load "<pkg>")
  >     (funcall sm-foo <args>)

That is true -- but since indirect-function is called in only around
40 places, problems of this kind will be far less frequent than the
problems we have now.

And maybe we could fix some of those 40 places to avoid the problems,
in their own way.

We're talking about problems that result from an anomalous situation
in which we don't do the really right thing.  In such a case, a change
that reduces the frequency of problems can be a good thing, even if it
doesn'tmake things 100% correct.

The main criterion is that the change should never cause problems
in any non-anomalous situation.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Prickliness of the "invalid byte code" stuff

[Stefan Monnier]

>   > > session. My thought was that if we  could detect this specific case,
>   > > then we could as well have emacs load that file, rather than having the
>   > > user reload it by hand.
>
>   > Yes, that usually works, but unless we reorganize the current code, we
>   > end up having to solve a slightly different problem similar to:
>
>   >     (setq sm-foo (indirect-function '<pkg>-<fun>))
>   >     (load "<pkg>")
>   >     (funcall sm-foo <args>)
>
> That is true -- but since indirect-function is called in only around
> 40 places, problems of this kind will be far less frequent than the
> problems we have now.

No, the problem we have now is exactly this one, because at the place where
we try to fetch the bytecode, all we have is a byte-code-function-p
object (i.e. what indirect-function returned above).  In many cases,
this object was found in the symbol-function cell of a symbol, but we
don't even currently try and propagate this symbol to the fetch-bytecode
function, so while in theory it's stashed somewhere up in the stack we
don't currently have any way to recover it.

In any case, I think deprecating byte-compile-dynamic is a better
solution than trying to fix its many shortcomings, since I don't know of
any concrete evidence that it has any real benefit.


        Stefan

Re: Prickliness of the "invalid byte code" stuff

[Paul Eggert]

On 6/17/19 11:10 AM, Stefan Monnier wrote:
> In any case, I think deprecating byte-compile-dynamic is a better
> solution than trying to fix its many shortcomings, since I don't know of
> any concrete evidence that it has any real benefit.

This sounds good to me as well. Should I write up a patch or is this a 
controversial point?

Re: Prickliness of the "invalid byte code" stuff

[Stefan Monnier]

>> In any case, I think deprecating byte-compile-dynamic is a better
>> solution than trying to fix its many shortcomings, since I don't know of
>> any concrete evidence that it has any real benefit.
> This sounds good to me as well.  Should I write up a patch or is this
> a controversial point?

I'd like to have confirmation from Eli that he's OK with it.


        Stefan

Re: Prickliness of the "invalid byte code" stuff

[T.V Raman]

When it was originally introduced, I remember byte-compile-dynamic made
a significant difference when loading large modules from which one
initially used only a few functions, it enabled lazy-loading.

Would that still matter in today's emacs, or have things moved along far
enough that it wont matter?
-- 

Re: Prickliness of the "invalid byte code" stuff

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > In any case, I think deprecating byte-compile-dynamic is a better
  > solution than trying to fix its many shortcomings, since I don't know of
  > any concrete evidence that it has any real benefit.

Maybe you are right.  I think byte-compile-dynamic was meant to reduce the
memory size of Emacs, and that is less important now than it was.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Prickliness of the "invalid byte code" stuff

[Stefan Monnier]

> When it was originally introduced, I remember byte-compile-dynamic made
> a significant difference when loading large modules from which one
> initially used only a few functions, it enabled lazy-loading.

That was the idea, yes.

> Would that still matter in today's emacs, or have things moved along far
> enough that it wont matter?

My impression is that the gain is negligible on today's hardware (and
I consider my 2007-era Thinkpad to be part of "today's hardware).

Maybe I could be convinced otherwise by actual benchmarks, of course.


        Stefan

Re: Prickliness of the "invalid byte code" stuff

[T.V Raman]

emacspeak is a fairly large package that at the time  showed a
significant win with respect to start-up time. I'll recompile it when I
get a chance with byte-compile-dynamic set to nil and do some timings.
-- 

Re: Prickliness of the "invalid byte code" stuff

[T.V Raman]

Hi Stefan 

Did a quick test, emacspeak startup feels just as fast with or without
byte-compile-dynamic -- will continue running the rebuilt session for a
few hours to see if  autoloaded functions feel slow.

I remember 2 things impacted when I originally did the switch -- likely
10 or 15 years ago: 

1. Startup got faster
2. If installed on a slower filesystem, then calling autoloaded
functions  from large modules felt slow.

Laptop has a solid-state   drive, guess I'll need to compare it on both
desktop and laptop 
-- 

Re: Prickliness of the "invalid byte code" stuff

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 17 Jun 2019 18:53:57 -0400
> Cc: Richard Stallman <rms@gnu.org>, emacs-devel@gnu.org
> 
> >> In any case, I think deprecating byte-compile-dynamic is a better
> >> solution than trying to fix its many shortcomings, since I don't know of
> >> any concrete evidence that it has any real benefit.
> > This sounds good to me as well.  Should I write up a patch or is this
> > a controversial point?
> 
> I'd like to have confirmation from Eli that he's OK with it.

I know nothing about this variable.  If my opinion is required, I will
need to read up on that, or maybe someone could just tell more details
about it.

Re: Prickliness of the "invalid byte code" stuff

[Stefan Monnier]

> I know nothing about this variable.  If my opinion is required, I will
> need to read up on that, or maybe someone could just tell more details
> about it.

OK, here's my take on it:
byte-compile-dynamic is a compilation option which makes the .elc file
have the actual bytes of the bytecoded functions lazy-loaded (in the
same way as the docstrings are lazy-loaded: the byte-code-function-p
objects contain in their bytecode field a "pointer" to the
file-and-position so the bytecodes are fecthed on-demand when the
function is called, via `fetch-bytecode`).

This was introduced at the same time as the lazy loading of docstrings.

The files using this option are expected to be faster to load (because
we load a bit less into heap).

I'm claiming that this benefit is not worthwhile.

The downsides are:
- it doesn't work when the .elc files are compressed.
- it doesn't work when the .elc files are loaded via Tramp.
- If the .elc files is re-generated after the .elc file was loaded,
  calling functions from that file typically signal an error (such as
  "invalid byte code").

A `grep byte-compile-dynamic: **/*.el` seems to indicate it's currently
used in 12 files bundled with Emacs.


        Stefan

Re: Prickliness of the "invalid byte code" stuff

[T.V Raman]

So after a few hours of running Emacs/emacspeak with
byte-compile-dynamic set to nil during compilation:

emacspeak has over 240 elisp files, all of which were using
byte-compile-dynamic set to t -- they're now set to nil.
1. No appreciable slowdown in startup -- either laptop or desktop 

2. No appreciable slowdown in callign autoloaded functions/commands 
-- 
Id: kg:/m/0285kf1 

Re: Prickliness of the "invalid byte code" stuff

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: eggert@cs.ucla.edu,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Tue, 18 Jun 2019 14:11:01 -0400
> 
> byte-compile-dynamic is a compilation option which makes the .elc file
> have the actual bytes of the bytecoded functions lazy-loaded (in the
> same way as the docstrings are lazy-loaded: the byte-code-function-p
> objects contain in their bytecode field a "pointer" to the
> file-and-position so the bytecodes are fecthed on-demand when the
> function is called, via `fetch-bytecode`).
> 
> This was introduced at the same time as the lazy loading of docstrings.
> 
> The files using this option are expected to be faster to load (because
> we load a bit less into heap).

There's also the effect on Emacs memory footprint, right?

> The downsides are:
> - it doesn't work when the .elc files are compressed.
> - it doesn't work when the .elc files are loaded via Tramp.
> - If the .elc files is re-generated after the .elc file was loaded,
>   calling functions from that file typically signal an error (such as
>   "invalid byte code").

The same problems happen with lazily loading doc strings, right?  Or
is that different in some way?

> A `grep byte-compile-dynamic: **/*.el` seems to indicate it's currently
> used in 12 files bundled with Emacs.

Any idea why those 12 use it?

Thanks.

Re: Prickliness of the "invalid byte code" stuff

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  eggert@cs.ucla.edu,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Tue, 18 Jun 2019 11:23:13 -0700
> 
> emacspeak has over 240 elisp files, all of which were using
> byte-compile-dynamic set to t -- they're now set to nil.
> 1. No appreciable slowdown in startup -- either laptop or desktop 
> 
> 2. No appreciable slowdown in callign autoloaded functions/commands 

What about the memory footprint of the Emacs session -- does this
variable have any effect on it?

Thanks.

Re: Prickliness of the "invalid byte code" stuff

[Stefan Monnier]

>> The files using this option are expected to be faster to load (because
>> we load a bit less into heap).
> There's also the effect on Emacs memory footprint, right?

Yes.

>> The downsides are:
>> - it doesn't work when the .elc files are compressed.
>> - it doesn't work when the .elc files are loaded via Tramp.
>> - If the .elc files is re-generated after the .elc file was loaded,
>>   calling functions from that file typically signal an error (such as
>>   "invalid byte code").
> The same problems happen with lazily loading doc strings, right?

Yes (at least for the third point (reload), not sure how the first two
points are affected by compression and access via Tramp).

> Or is that different in some way?

There are some qualitative differences:

- docstrings are only loaded in fairly specific circumstances (mostly
  C-h <foo>), whereas lazy loading of code can happen in the middle of
  arbitrary code execution, so an error there can be more troublesome.

- when lazy loading a docstring and we discover that the file has
  changed (which we admittedly don't always notice), the code currently
  is able to reload the .elc file to update the offset and try again.
  In contrast, for lazy loading of code, the code is currently unable to
  do that (this is the origin on this discussion: it could be fixed in
  the common cases, but last I looked at it, it'd require changes to the
  code which could be detrimental to the common case).

- when we don't notice that the file was modified (i.e. the offset is
  wrong, but it happens to point to a valid place), displaying the wrong
  docstring is just annoying, whereas loading the wrong bytecode can
  lead to much more severe problems.

>> A `grep byte-compile-dynamic: **/*.el` seems to indicate it's currently
>> used in 12 files bundled with Emacs.
> Any idea why those 12 use it?

I think in the 20th century, the performance difference could be
measured (at least in benchmarks, tho maybe also in actual use), but
I seriously doubt it makes a noticeable difference on machines of this
century (not sure if it's because of changes in hardware such as
available RAM or CPU speed, or because the growth of the rest of
Emacs dwarfs those effects, or what).


        Stefan

Re: Prickliness of the "invalid byte code" stuff

[Stefan Monnier]

> Yes (at least for the third point (reload), not sure how the first two
> points are affected by compression and access via Tramp).

I meant: ..., not sure how lazy docstrings are affected by the first two
points (compression and access via Tramp).


        Stefan

Re: Prickliness of the "invalid byte code" stuff

[T.V Raman]

It does to an extent, but doesn't appear to affect performance --
either on my laptop or desktop 
Eli Zaretskii writes:
 > > From: "T.V Raman" <raman@google.com>
 > > Cc: Eli Zaretskii <eliz@gnu.org>,  eggert@cs.ucla.edu,  rms@gnu.org,  emacs-devel@gnu.org
 > > Date: Tue, 18 Jun 2019 11:23:13 -0700
 > > 
 > > emacspeak has over 240 elisp files, all of which were using
 > > byte-compile-dynamic set to t -- they're now set to nil.
 > > 1. No appreciable slowdown in startup -- either laptop or desktop 
 > > 
 > > 2. No appreciable slowdown in callign autoloaded functions/commands 
 > 
 > What about the memory footprint of the Emacs session -- does this
 > variable have any effect on it?
 > 
 > Thanks.

-- 
Id: kg:/m/0285kf1 

-- 
Id: kg:/m/0285kf1 

Re: Prickliness of the "invalid byte code" stuff

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Date: Tue, 18 Jun 2019 12:54:28 -0700
> Cc: raman@google.com,
>     monnier@iro.umontreal.ca,
>     eggert@cs.ucla.edu,
>     rms@gnu.org,
>     emacs-devel@gnu.org
> 
> It does to an extent, but doesn't appear to affect performance --
> either on my laptop or desktop 

Can you tell any numbers, like the memory growth in percents and in
KB?

Thanks.

Re: Prickliness of the "invalid byte code" stuff

[T.V Raman]

Eli Zaretskii <eliz@gnu.org> writes:


this will take some time --- emacspeak has a lot of modules that are
loaded on demand -- and the memory footprint is a function of how long
the session is running -- I consider killing emacs a mortal crime.

Will let you know if I feel the running image  getting bloated over
time.

>> From: "T.V Raman" <raman@google.com>
>> Date: Tue, 18 Jun 2019 12:54:28 -0700
>> Cc: raman@google.com,
>>     monnier@iro.umontreal.ca,
>>     eggert@cs.ucla.edu,
>>     rms@gnu.org,
>>     emacs-devel@gnu.org
>> 
>> It does to an extent, but doesn't appear to affect performance --
>> either on my laptop or desktop 
>
> Can you tell any numbers, like the memory growth in percents and in
> KB?
>
> Thanks.
>

-- 

Re: Prickliness of the "invalid byte code" stuff

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Cc: eggert@cs.ucla.edu,  emacs-devel@gnu.org,  monnier@iro.umontreal.ca,  rms@gnu.org
> Date: Wed, 19 Jun 2019 06:54:48 -0700
> 
> this will take some time --- emacspeak has a lot of modules that are
> loaded on demand -- and the memory footprint is a function of how long
> the session is running -- I consider killing emacs a mortal crime.

There's no rush.  It's just that emacspeak uses this variable in a lot
of Lisp files, so I think it's important to see how such an extreme
situation will react to deletion of this variable, before we make the
decision.

> Will let you know if I feel the running image  getting bloated over
> time.

Thank you.

Re: Prickliness of the "invalid byte code" stuff

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: eggert@cs.ucla.edu,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Tue, 18 Jun 2019 15:48:30 -0400
> 
> >> A `grep byte-compile-dynamic: **/*.el` seems to indicate it's currently
> >> used in 12 files bundled with Emacs.
> > Any idea why those 12 use it?
> 
> I think in the 20th century, the performance difference could be
> measured (at least in benchmarks, tho maybe also in actual use), but
> I seriously doubt it makes a noticeable difference on machines of this
> century (not sure if it's because of changes in hardware such as
> available RAM or CPU speed, or because the growth of the rest of
> Emacs dwarfs those effects, or what).

I understand, but I was curious why these 12 files do use the
variable, while all the rest don't.  IOW, what is (or was) so special
about these files?

I think I'd like to wait with the decision until we hear the
emacsspeak results.  I expect that the effect on the memory will be
minor, even in a package _all_ of whose files use this variable.  If
that is indeed so, I think we can disable the variable.

Re: Prickliness of the "invalid byte code" stuff

[Stefan Monnier]

> I understand, but I was curious why these 12 files do use the
> variable, while all the rest don't.  IOW, what is (or was) so special
> about these files?

I'm to blame for the pcvs-util.el, I think, but I must say I can't
remember nor figure out why I did that.

For the others, I believe the expectation is that it's common to use only
a small part of those files (like a handful of functions from CL or one
particular command from dired-aux), which is the most favorable
situation for byte-compile-dynamic.


        Stefan

Re: Prickliness of the "invalid byte code" stuff

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: eggert@cs.ucla.edu,  rms@gnu.org,  emacs-devel@gnu.org
> Date: Wed, 19 Jun 2019 12:25:19 -0400
> 
> For the others, I believe the expectation is that it's common to use only
> a small part of those files (like a handful of functions from CL or one
> particular command from dired-aux), which is the most favorable
> situation for byte-compile-dynamic.

Makes sense, thanks.

Re: Prickliness of the "invalid byte code" stuff

[Stefan Monnier]

> I think I'd like to wait with the decision until we hear the
> emacsspeak results.

There's no hurry on this, indeed.


        Stefan

Re: Prickliness of the "invalid byte code" stuff

[T.V Raman]

Stefan Monnier <monnier@iro.umontreal.ca> writes:


After 22 hours, 55 minutes, 40 seconds of Emacs uptime, 
      RSS: 1738828
      VSize: 2483972
      %Mem:  5.3
      %CPU:  0.4

which look fairly normal to what I was seeing in the past with emacspeak
compiled using byte-compile-dynamic.

I've also not noticed any perceivable lag (desktop and laptop) when
calling autoloaded functions from  large modules for the first time, e.g.
        https://github.com/tvraman/emacspeak/blob/7957d9c0e6effc9749d027779c36a6cd82c05ebd/lisp/emacspeak-wizards.el
This module has over 4200 lines of code, and functions from it are often
used for the first time many hours after Emacs has been started.
>> I think I'd like to wait with the decision until we hear the
>> emacsspeak results.
>
> There's no hurry on this, indeed.
>
>
>         Stefan
>
>

-- 
Id: kg:/m/0285kf1 

Re: Prickliness of the "invalid byte code" stuff

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Date: Wed, 19 Jun 2019 14:35:18 -0700
> Cc: emacs-devel@gnu.org
> 
> Stefan Monnier <monnier@iro.umontreal.ca> writes:
> 
> 
> After 22 hours, 55 minutes, 40 seconds of Emacs uptime, 
>       RSS: 1738828
>       VSize: 2483972
>       %Mem:  5.3
>       %CPU:  0.4
> 
> which look fairly normal to what I was seeing in the past with emacspeak
> compiled using byte-compile-dynamic.
> 
> I've also not noticed any perceivable lag (desktop and laptop) when
> calling autoloaded functions from  large modules for the first time, e.g.
>         https://github.com/tvraman/emacspeak/blob/7957d9c0e6effc9749d027779c36a6cd82c05ebd/lisp/emacspeak-wizards.el
> This module has over 4200 lines of code, and functions from it are often
> used for the first time many hours after Emacs has been started.
> >> I think I'd like to wait with the decision until we hear the
> >> emacsspeak results.
> >
> > There's no hurry on this, indeed.

OK, thanks.  I did some more tests, and I think it's indeed OK to let
this variable do nothing.  But I guess we still need to accept its
specification in file-local variables without any errors, at least for
some time?

Re: Prickliness of the "invalid byte code" stuff

[T.V Raman]

I'll keep emacs running on both desktop and laptop and send in some
numbers next week -- likely around mid-week. At this point, at least
from an emacspeak runtime performance perspective, I  think it's fine to
delete that variable --- 
-- 

Re: Prickliness of the "invalid byte code" stuff

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Cc: monnier@iro.umontreal.ca,  emacs-devel@gnu.org
> Date: Fri, 21 Jun 2019 07:10:07 -0700
> 
> I'll keep emacs running on both desktop and laptop and send in some
> numbers next week -- likely around mid-week. At this point, at least
> from an emacspeak runtime performance perspective, I  think it's fine to
> delete that variable --- 

Thanks.

Re: Prickliness of the "invalid byte code" stuff

[Stefan Monnier]

> OK, thanks.  I did some more tests, and I think it's indeed OK to let
> this variable do nothing.

My original suggestion was just to declare it obsolete (and, as
a consequence, to stop using it in our own files).  So I just pushed that.

We can additionally change bytecomp to simply ignore the variable,
indeed, that's fine by me, but I think we can wait a bit before doing
that (even until Emacs-28, if you ask me).

> But I guess we still need to accept its specification in file-local
> variables without any errors, at least for some time?

Yes, of course.  Note that when compiling the file, any unknown
fie-local specifications are simply ignored, so if/when we remove the

    (put 'byte-compile-dynamic 'safe-local-variable 'booleanp)
    
it will not cause an error, but it will just prompt the user whether to
obey this "risky" setting instead of accepting it silently.


        Stefan

Re: Prickliness of the "invalid byte code" stuff

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: "T.V Raman" <raman@google.com>,  emacs-devel@gnu.org
> Date: Fri, 21 Jun 2019 10:34:46 -0400
> 
> > OK, thanks.  I did some more tests, and I think it's indeed OK to let
> > this variable do nothing.
> 
> My original suggestion was just to declare it obsolete (and, as
> a consequence, to stop using it in our own files).  So I just pushed that.
> 
> We can additionally change bytecomp to simply ignore the variable,
> indeed, that's fine by me, but I think we can wait a bit before doing
> that (even until Emacs-28, if you ask me).

Fine with me,

> > But I guess we still need to accept its specification in file-local
> > variables without any errors, at least for some time?
> 
> Yes, of course.  Note that when compiling the file, any unknown
> fie-local specifications are simply ignored, so if/when we remove the
> 
>     (put 'byte-compile-dynamic 'safe-local-variable 'booleanp)
>     
> it will not cause an error, but it will just prompt the user whether to
> obey this "risky" setting instead of accepting it silently.

The prompts are annoying, so I think we should do something to avoid
them in this specific case.