bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[German Pacenza]

Hi, every time I start emacs cl-loaddefs.el gets native-compiled.

Compiling /usr/share/emacs/29.0.50/lisp/emacs-lisp/cl-loaddefs.el...

I think it is related to 6c11214dc1124bcb459088e89334e16e46127e16

Thanks


In GNU Emacs 29.0.50 (build 1, x86_64-pc-linux-gnu, GTK+ Version
 3.24.34, cairo version 1.17.6) of 2022-09-06 built on KRONOS
Repository revision: 015fb4ac1c84485c563934087884f8a7dfe51955
Repository branch: master
System Description: Manjaro Linux

Configured using:
 'configure --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib
 --localstatedir=/var --mandir=/usr/share/man --with-gameuser=:games
 --with-modules --without-libotf --without-m17n-flt --without-gconf
 --with-native-compilation --with-xinput2 --with-pgtk --without-xaw3d
 --with-sound=no --without-gpm --without-compress-install
 '--program-transform-name=s/\([ec]tags\)/\1.emacs/'
 'CFLAGS=-march=native -mtune=native -O2 -pipe -fno-plt -fexceptions
 -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security
 -fstack-clash-protection -fcf-protection'
 LDFLAGS=-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now'

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GSETTINGS HARFBUZZ JPEG JSON
LCMS2 LIBSYSTEMD LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PGTK
PNG RSVG SECCOMP SQLITE3 THREADS TIFF TOOLKIT_SCROLL_BARS WEBP XIM GTK3
ZLIB

Important settings:
  value of $LC_MONETARY: es_AR.UTF-8
  value of $LC_NUMERIC: es_AR.UTF-8
  value of $LC_TIME: es_AR.UTF-8
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Major mode: Fundamental

Minor modes in effect:
  savehist-mode: t
  vertico-multiform-mode: t
  vertico-mode: t
  popper-mode: t
  minibuffer-depth-indicate-mode: t
  delete-selection-mode: t
  mood-line-mode: t
  straight-use-package-mode: t
  straight-package-neutering-mode: t
  straight-live-modifications-mode: t
  global-so-long-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  buffer-read-only: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/german/.emacs.d/straight/build/transient/transient hides /usr/share/emacs/29.0.50/lisp/transient

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util time-date mm-decode mm-bodies mm-encode mail-parse
rfc2231 mailabbrev gmm-utils mailheader sendmail rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mail-utils orderless savehist vc-hg vc-git
diff-mode vc vc-dispatcher project byte-opt consult-vertico consult
compat-28 compat compat-macs recentf tree-widget wid-edit kmacro
bookmark text-property-search pp comp comp-cstr warnings icons rx
elec-pair vertico-flat vertico-multiform pcase vertico helpful-autoloads
elisp-refs-autoloads f-autoloads s-autoloads denote-autoloads popper
popper-autoloads magit-autoloads magit-section-autoloads
git-commit-autoloads with-editor-autoloads transient-autoloads
dash-autoloads elfeed-autoloads vertico-autoloads embark-autoloads
consult-autoloads compat-autoloads mb-depth orderless-autoloads delsel
xah-fly-keys xah-fly-keys-autoloads rainbow-mode-autoloads easy-mmode
g3r-dark-theme ef-themes-autoloads info mood-line mood-line-autoloads
straight-autoloads cl-seq cl-extra help-mode straight subr-x cl-macs gv
bytecomp byte-compile cconv so-long cl-loaddefs cl-lib rmc iso-transl
tooltip eldoc paren electric uniquify ediff-hook vc-hooks
lisp-float-type elisp-mode mwheel term/pgtk-win pgtk-win term/common-win
pgtk-dnd tool-bar dnd fontset image regexp-opt fringe tabulated-list
replace newcomment text-mode lisp-mode prog-mode register page tab-bar
menu-bar rfn-eshadow isearch easymenu timer select scroll-bar mouse
jit-lock font-lock syntax font-core term/tty-colors frame minibuffer
nadvice seq simple cl-generic indonesian philippine cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech european
ethiopic indian cyrillic chinese composite emoji-zwj charscript charprop
case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs faces cus-face macroexp files window
text-properties overlay sha1 md5 base64 format env code-pages mule
custom widget keymap hashtable-print-readable backquote threads dbusbind
inotify dynamic-setting system-font-setting font-render-setting cairo
gtk pgtk lcms2 multi-tty make-network-process native-compile emacs)

Memory information:
((conses 16 158819 71652)
 (symbols 48 11974 2)
 (strings 32 43101 34802)
 (string-bytes 1 1641507)
 (vectors 16 26765)
 (vector-slots 8 465341 172223)
 (floats 8 83 459)
 (intervals 56 357 144)
 (buffers 1000 11))

-- 
German Pacenza

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

German Pacenza <germanp82@hotmail.com> writes:

> Hi, every time I start emacs cl-loaddefs.el gets native-compiled.
>
> Compiling /usr/share/emacs/29.0.50/lisp/emacs-lisp/cl-loaddefs.el...
>
> I think it is related to 6c11214dc1124bcb459088e89334e16e46127e16

Is there a 

;; no-native-compile: t

cookie in your cl-loaddefs.el file?  If not, try saying "make
bootstrap".

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[German Pacenza]

Lars Ingebrigtsen <larsi@gnus.org> writes:
>
> Is there a 
>
> ;; no-native-compile: t
>
> cookie in your cl-loaddefs.el file?  If not, try saying "make
> bootstrap".

Yes at the end of the file, make bootstrap didn't help

;; Local Variables:
;; version-control: never
;; no-update-autoloads: t
;; no-native-compile: t
;; coding: utf-8-emacs-unix
;; End:

-- 
German Pacenza

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

German Pacenza <germanp82@hotmail.com> writes:

> Yes at the end of the file, make bootstrap didn't help

Hm, yes -- I can reproduce the problem here, too.

So the

;; no-native-compile: t

cookie doesn't seem to actually work?  I've added Andrea to the CCs;
perhaps he has some comments.

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> German Pacenza <germanp82@hotmail.com> writes:
>
>> Yes at the end of the file, make bootstrap didn't help
>
> Hm, yes -- I can reproduce the problem here, too.
>
> So the
>
> ;; no-native-compile: t
>
> cookie doesn't seem to actually work?  I've added Andrea to the CCs;
> perhaps he has some comments.

Hi all,

AFAICS it does work, the issue is that we print the "Compiling
blablabla.el" message before we giving up because we realize
no-native-compile is set... :/

I'll think about how to fix this.

  Andrea

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Eli Zaretskii]

> Cc: German Pacenza <germanp82@hotmail.com>, 57627@debbugs.gnu.org
> From: Andrea Corallo <akrl@sdf.org>
> Date: Tue, 06 Sep 2022 16:33:54 +0000
> 
> > So the
> >
> > ;; no-native-compile: t
> >
> > cookie doesn't seem to actually work?  I've added Andrea to the CCs;
> > perhaps he has some comments.
> 
> Hi all,
> 
> AFAICS it does work, the issue is that we print the "Compiling
> blablabla.el" message before we giving up because we realize
> no-native-compile is set... :/

Are you sure?  If I do "M-x list-processes" right after starting
Emacs, I see this in the list:

  Compiling: /ho… 5370    run     *Async-native-compile-lo… /dev/pts/2   Main    /home/eliz/git/emacs/native-comp/src/emacs --batch -l /tmp/emacs-async-comp-cl-loaddefs-UlziS9.el

which tells me we really do native-compile cl-loaddefs.el.

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: German Pacenza <germanp82@hotmail.com>, 57627@debbugs.gnu.org
>> From: Andrea Corallo <akrl@sdf.org>
>> Date: Tue, 06 Sep 2022 16:33:54 +0000
>> 
>> > So the
>> >
>> > ;; no-native-compile: t
>> >
>> > cookie doesn't seem to actually work?  I've added Andrea to the CCs;
>> > perhaps he has some comments.
>> 
>> Hi all,
>> 
>> AFAICS it does work, the issue is that we print the "Compiling
>> blablabla.el" message before we giving up because we realize
>> no-native-compile is set... :/
>
> Are you sure?  If I do "M-x list-processes" right after starting
> Emacs, I see this in the list:
>
>   Compiling: /ho… 5370 run *Async-native-compile-lo… /dev/pts/2 Main
> /home/eliz/git/emacs/native-comp/src/emacs --batch -l
> /tmp/emacs-async-comp-cl-loaddefs-UlziS9.el
>
> which tells me we really do native-compile cl-loaddefs.el.

I think so, we emit the "Compiling" message already in the subprocess.
So yeah, if the requirement is not to spawn a process,
`no-native-compile' is not up to the job.

  Andrea

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> I think so, we emit the "Compiling" message already in the subprocess.
> So yeah, if the requirement is not to spawn a process,
> `no-native-compile' is not up to the job.

It'd be better to detect this earlier.

And...  since the .eln file is never written, it'll fork these Emacsen
every time you start Emacs?  That seems to be the case -- I get

Compiling /home/larsi/src/emacs/nativecomp/lisp/emacs-lisp/cl-loaddefs.el...

in the async buffer on every Emacs restart.

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Eli Zaretskii <eliz@gnu.org>,  germanp82@hotmail.com,
>   57627@debbugs.gnu.org
> Date: Tue, 06 Sep 2022 22:40:47 +0200
> 
> And...  since the .eln file is never written, it'll fork these Emacsen
> every time you start Emacs?  That seems to be the case -- I get
> 
> Compiling /home/larsi/src/emacs/nativecomp/lisp/emacs-lisp/cl-loaddefs.el...
> 
> in the async buffer on every Emacs restart.

If you let it finish, i.e. wait until list-processes shows an empty
buffer, it won't start these compilations in the next invocations.  At
least that's what happens to me.

Btw, is that a GUI session or a -nw session?  I see this in a -nw
session, which I can explain: we load the terminal-specific file from
lisp/term/, and that requires compilation, so we load comp.el to start
compilation, and that then loads all the dependencies of comp.el and
compiles them, which of course includes cl-lib, cl-macs, cl-loaddefs,
etc.

In a GUI session I don't expect all this to happen, so if it does, we
should investigate why we load something at startup in that case.

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> If you let it finish, i.e. wait until list-processes shows an empty
> buffer, it won't start these compilations in the next invocations.  At
> least that's what happens to me.

It's not what happens to me.  Every time I start Emacs (not -Q), it
says:

---
Compiling /home/larsi/src/emacs/nativecomp/lisp/emacs-lisp/cl-loaddefs.el...
Compiling /home/larsi/src/emacs/nativecomp/lisp/net/tramp-loaddefs.el...
Compilation finished.
---

> Btw, is that a GUI session or a -nw session?

GUI.

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[German Pacenza]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> It's not what happens to me.  Every time I start Emacs (not -Q), it
> says:
>
> ---
> Compiling /home/larsi/src/emacs/nativecomp/lisp/emacs-lisp/cl-loaddefs.el...
> Compiling /home/larsi/src/emacs/nativecomp/lisp/net/tramp-loaddefs.el...
> Compilation finished.
> ---
>

For me it happens in both GUI and -nw . If I add:
(setq native-comp-deferred-compilation-deny-list '("cl-loaddefs.el"))

The *Async-native-compile-log* buffer is still created but it only says:
Compilation finished.

-- 
German Pacenza

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

German Pacenza <germanp82@hotmail.com> writes:

> For me it happens in both GUI and -nw . If I add:
> (setq native-comp-deferred-compilation-deny-list '("cl-loaddefs.el"))

I think perhaps the easiest solution here would be to just pre-populate
that defcustom with '("loaddefs.el\\'").

> The *Async-native-compile-log* buffer is still created but it only says:
> Compilation finished.

Hm, that sounds like a bug -- it shouldn't create that buffer at all if
there's nothing to compile.  But perhaps it does the
native-comp-deferred-compilation-deny-list filtering at a too-late time
now?

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> I think perhaps the easiest solution here would be to just pre-populate
> that defcustom with '("loaddefs.el\\'").

Or perhaps remove the cookies again and arrange to have
lisp/loaddefs.elc native-compiled.  That would be even simpler, I guess.

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: akrl@sdf.org,  germanp82@hotmail.com,  57627@debbugs.gnu.org
> Date: Wed, 07 Sep 2022 14:47:37 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > If you let it finish, i.e. wait until list-processes shows an empty
> > buffer, it won't start these compilations in the next invocations.  At
> > least that's what happens to me.
> 
> It's not what happens to me.  Every time I start Emacs (not -Q), it
> says:
> 
> ---
> Compiling /home/larsi/src/emacs/nativecomp/lisp/emacs-lisp/cl-loaddefs.el...
> Compiling /home/larsi/src/emacs/nativecomp/lisp/net/tramp-loaddefs.el...
> Compilation finished.
> ---
> 
> > Btw, is that a GUI session or a -nw session?
> 
> GUI.

Then maybe it's a different issue than what I see here.  Since this is
not a -Q session, I don't know what you load there.  Suggest to start
Emacs under GDB with a breakpoint in Fload, and see what are we
loading that leads to those two.

In general, if you let Emacs native-compile everything it loads at
startup, the next startup it will not need to compile anything, since
the *.eln files are already available.  So if anything still tries to
load and thus native-compile those *-loaddefs files should be the
stuff you load itself, not the compilation infrastructure.

OTOH, I don't see a problem in Emacs starting an async compilation
that ends up not compiling anything.  It runs in the background, so I
don't think it interferes with anything?

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> In general, if you let Emacs native-compile everything it loads at
> startup, the next startup it will not need to compile anything, since
> the *.eln files are already available.

The loaddefs files aren't compiled, since they have a cookie stopping
that from happening.  So no .eln file is made.

> OTOH, I don't see a problem in Emacs starting an async compilation
> that ends up not compiling anything.  It runs in the background, so I
> don't think it interferes with anything?

It's inefficient and confusing.

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> I think so, we emit the "Compiling" message already in the subprocess.
>> So yeah, if the requirement is not to spawn a process,
>> `no-native-compile' is not up to the job.
>
> It'd be better to detect this earlier.

Yeah I think we should probably check directly in 'maybe_swap_for_eln'
and if the cookie is present just return (if that works).

I can try to prepare and test a patch but not before Friday sorry.

Best Regards

  Andrea

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Yeah I think we should probably check directly in 'maybe_swap_for_eln'
> and if the cookie is present just return (if that works).

It would require reading the .el file in the main Emacs before forking
off the sub-Emacs, I guess?  So it'd be slightly slower in general...

But I guess there's no way around that.

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Yeah I think we should probably check directly in 'maybe_swap_for_eln'
>> and if the cookie is present just return (if that works).
>
> It would require reading the .el file in the main Emacs before forking
> off the sub-Emacs, I guess?  So it'd be slightly slower in general...
>
> But I guess there's no way around that.

Hi Lars,

I'm not sure, shouldn't the cookie already be present in the
bytecode?

  Andrea

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> I'm not sure, shouldn't the cookie already be present in the
> bytecode?

It's not at present -- it's just a comment in the .el file, after all.

But perhaps we could propagate that to the .elc file somehow?

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> I'm not sure, shouldn't the cookie already be present in the
>> bytecode?
>
> It's not at present -- it's just a comment in the .el file, after all.
>
> But perhaps we could propagate that to the .elc file somehow?

Uh you are right!  Dunno why I was convinced cookies are present in the
.elc :/

Yes I think we should propapgate them, well at least `no-native-compile'
(I'm not sure of the others).  Opinions are very welcome :)

  Andrea

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Yes I think we should propapgate them, well at least `no-native-compile'
> (I'm not sure of the others).  Opinions are very welcome :)

Perhaps we should make the no-native-compile thing into code instead of
having it as a comment?  I.e., we could put

(push #$ native-comp-deferred-compilation-inhibited-list)

into the files instead?  (And then use that variable in addition to the
native-comp-deferred-compilation-deny-list defcustom.)

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> (push #$ native-comp-deferred-compilation-inhibited-list)

(Or not actually #$, since that byte-compiles away, but

  (push "foo-loaddefs.el" native-comp-deferred-compilation-inhibited-list)

and so on, I guess.

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Perhaps we should make the no-native-compile thing into code instead of
> having it as a comment? 

I started futzing around with this again, but then wondered whether we
should have a more general language mechanism for stuff like this.  So
I've added Stefan to the CCs; perhaps he's thought about this stuff
before.

So to recap the practical problem we have today and would like to fix:

We put

;; no-native-compile: t

into .el files to signal that we don't want them to be native-compiled.
However, the machinery today doesn't see that when we want it to.  That
is, we load the .elc file, the nativecomp machinery then kicks in and
adds the file to the async .eln list, which then forks off an Emacs
which then loads the .el file and sees that cookie, and then does
nothing.  (And this will happen on every Emacs run.)

So the machinery either has to inspect the .el file in addition to the
.elc file (which is inefficient), or we need to put something into the
.elc file to tell the machinery not to bother with generating the .eln.

This is simple enough, of course -- we could just introduce a
(no-native-compile) function that hooks into the machinery at the right
point.

But this reminds me of other "file-wide directives" that have been
previously discussed over the years, and makes me wonder whether we
should introduce something more general.

As an example of an ad-hoc one that already exists, we have `defgroup'
that makes all subsequent `defcustom' forms use that group in the same
file.  We've previously discussed (in i18n contexts) whether it should
be possible to specify the translation domain of the file.  And we have
different Emacs Lisp dialects, and we have shorthands, which use
comments to change the behaviour of the code, which is unsatisfactory.

So...  is it time to introduce something like `pragma'?

That is, in this case, we'd say

(pragma 'no-native-compile)

somewhere in the file.  We could have

(pragma 'dynamic-binding)

for future versions of Emacs when lexical binding is default but we want
to allow people files to still be dynamic.

And we could have

(pragma 'shorthands "snu-" "some-nice-string-utils-")

And, of course, for backwards/forwards compat, any unknown pragmas would
be ignored.

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

> ;; no-native-compile: t
>
> into .el files to signal that we don't want them to be native-compiled.
> However, the machinery today doesn't see that when we want it to.  That
> is, we load the .elc file, the nativecomp machinery then kicks in and
> adds the file to the async .eln list, which then forks off an Emacs
> which then loads the .el file and sees that cookie, and then does
> nothing.  (And this will happen on every Emacs run.)
>
> So the machinery either has to inspect the .el file in addition to the
> .elc file (which is inefficient), or we need to put something into the
> .elc file to tell the machinery not to bother with generating the .eln.

After spending many milliseconds thinking about it, my conclusion is
that the bytecompiler should add a little code snippet like
(puthash load-file-name t comp--no-native-compile) in the
file if `no-native-compile` was specified.  So it then be easy for the
lazy native compilation to detect that it should skip this file (since
lazy native compilation is triggered after loading the file) by just
consulting `comp--no-native-compile`.

For that, there's no need to change the way `no-native-compile` is specified.

> So...  is it time to introduce something like `pragma'?
>
> That is, in this case, we'd say
>
> (pragma 'no-native-compile)
>
> somewhere in the file.

I guess that could work for `no-native-compile`, indeed.  But if you ask
to native compile this file and the pragma is halfway down the file,
what happens?

> We could have
>
> (pragma 'dynamic-binding)

I guess this one could work (of course, it'd have to be at top-level),
and we could switch back&forth within the same file (yuck!).

But if we allow such `pragma` to be output by macros, then it becomes
tricky for `eval-region` to reliably decide which dialect to use.

> And we could have
>
> (pragma 'shorthands "snu-" "some-nice-string-utils-")

Same question: the tooling will often want to have access to that
information but without necessarily wanting to run (or macroexpand) all
the code first.

So we could allow such `pragma`, but we'd likely end up restricting its
syntax so we're able to find it with something like a regexp search, so
in the end it's not clear what's the advantage over
file-local variables.


        Stefan

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> After spending many milliseconds thinking about it, my conclusion is
> that the bytecompiler should add a little code snippet like
> (puthash load-file-name t comp--no-native-compile) in the
> file if `no-native-compile` was specified.  So it then be easy for the
> lazy native compilation to detect that it should skip this file (since
> lazy native compilation is triggered after loading the file) by just
> consulting `comp--no-native-compile`.
>
> For that, there's no need to change the way `no-native-compile` is specified.

True, but it's kinda hacky, and if possible, I'd like to avoid adding
more hacks in this area...

>> That is, in this case, we'd say
>>
>> (pragma 'no-native-compile)
>>
>> somewhere in the file.
>
> I guess that could work for `no-native-compile`, indeed.  But if you ask
> to native compile this file and the pragma is halfway down the file,
> what happens?

Yes, that's no good.  Uhm...  we could make a rule that all `pragma's
have to appear as the first form(s) in a Lisp file?  And error out if
somebody tries to add a `pragma' later in the file.

I think that would make sense in general -- we're trying to express
something about the file as a whole, after all.

>> We could have
>>
>> (pragma 'dynamic-binding)
>
> I guess this one could work (of course, it'd have to be at top-level),
> and we could switch back&forth within the same file (yuck!).
>
> But if we allow such `pragma` to be output by macros, then it becomes
> tricky for `eval-region` to reliably decide which dialect to use.

Hm, yes...  But we have the same issue today, don't we?  That is

(progn
  (pop-to-buffer "*foo*")
  (emacs-lisp-mode)
  (insert ";;;  -*- lexical-binding: t -*-\n(message \"Lex: %s\" lexical-binding)")
  (eval-region (pos-bol) (point)))

Well, OK, that's the same result with point-min, but...

> So we could allow such `pragma`, but we'd likely end up restricting its
> syntax so we're able to find it with something like a regexp search, so
> in the end it's not clear what's the advantage over
> file-local variables.

If they have to be the first forms, there may be some advantages...

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Stefan Monnier via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Re-reading this discussion what I see is two different options, neither
of which is significantly better than the other.
So the motivation to change is not very high.


        Stefan


Lars Ingebrigtsen [2022-10-15 12:13:32] wrote:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> After spending many milliseconds thinking about it, my conclusion is
>> that the bytecompiler should add a little code snippet like
>> (puthash load-file-name t comp--no-native-compile) in the
>> file if `no-native-compile` was specified.  So it then be easy for the
>> lazy native compilation to detect that it should skip this file (since
>> lazy native compilation is triggered after loading the file) by just
>> consulting `comp--no-native-compile`.
>>
>> For that, there's no need to change the way `no-native-compile` is specified.
>
> True, but it's kinda hacky, and if possible, I'd like to avoid adding
> more hacks in this area...
>
>>> That is, in this case, we'd say
>>>
>>> (pragma 'no-native-compile)
>>>
>>> somewhere in the file.
>>
>> I guess that could work for `no-native-compile`, indeed.  But if you ask
>> to native compile this file and the pragma is halfway down the file,
>> what happens?
>
> Yes, that's no good.  Uhm...  we could make a rule that all `pragma's
> have to appear as the first form(s) in a Lisp file?  And error out if
> somebody tries to add a `pragma' later in the file.
>
> I think that would make sense in general -- we're trying to express
> something about the file as a whole, after all.
>
>>> We could have
>>>
>>> (pragma 'dynamic-binding)
>>
>> I guess this one could work (of course, it'd have to be at top-level),
>> and we could switch back&forth within the same file (yuck!).
>>
>> But if we allow such `pragma` to be output by macros, then it becomes
>> tricky for `eval-region` to reliably decide which dialect to use.
>
> Hm, yes...  But we have the same issue today, don't we?  That is
>
> (progn
>   (pop-to-buffer "*foo*")
>   (emacs-lisp-mode)
>   (insert ";;;  -*- lexical-binding: t -*-\n(message \"Lex: %s\" lexical-binding)")
>   (eval-region (pos-bol) (point)))
>
> Well, OK, that's the same result with point-min, but...
>
>> So we could allow such `pragma`, but we'd likely end up restricting its
>> syntax so we're able to find it with something like a regexp search, so
>> in the end it's not clear what's the advantage over
>> file-local variables.
>
> If they have to be the first forms, there may be some advantages...

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> Re-reading this discussion what I see is two different options, neither
> of which is significantly better than the other.
> So the motivation to change is not very high.

True.

So I've now implemented Stefan's suggestion (i.e., just putting a
puthash into the .elc file if the file-local variable says so), and it
seems to work fine.

Andrea, does this look OK to you?

diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index 74ba8984f2..1dc5442716 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -2323,9 +2323,15 @@ byte-compile-from-buffer
        (setq case-fold-search nil))
      (displaying-byte-compile-warnings
       (with-current-buffer inbuffer
-	(and byte-compile-current-file
-	     (byte-compile-insert-header byte-compile-current-file
-                                         byte-compile--outbuffer))
+	(when byte-compile-current-file
+	  (byte-compile-insert-header byte-compile-current-file
+                                      byte-compile--outbuffer)
+          ;; Instruct native-comp to ignore this file.
+          (when (bound-and-true-p no-native-compile)
+            (with-current-buffer byte-compile--outbuffer
+              (insert
+               "(when (boundp 'native-comp--no-native-compile)
+  (puthash load-file-name t native-comp--no-native-compile))\n\n"))))
 	(goto-char (point-min))
 	;; Should we always do this?  When calling multiple files, it
 	;; would be useful to delay this warning until all have been
diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
index 889bffa3f5..3c64f472a0 100644
--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -4119,6 +4119,8 @@ native-compile-async-skip-p
 LOAD and SELECTOR work as described in `native--compile-async'."
   ;; Make sure we are not already compiling `file' (bug#40838).
   (or (gethash file comp-async-compilations)
+      (gethash (file-name-with-extension file "elc")
+               native-comp--no-native-compile)
       (cond
        ((null selector) nil)
        ((functionp selector) (not (funcall selector file)))
diff --git a/lisp/loadup.el b/lisp/loadup.el
index c01c827a75..95bfbc502e 100644
--- a/lisp/loadup.el
+++ b/lisp/loadup.el
@@ -501,7 +501,10 @@
                                          bin-dest-dir)
                      ;; Relative filename from the built uninstalled binary.
                      (file-relative-name file invocation-directory)))))
-	       comp-loaded-comp-units-h))))
+	       comp-loaded-comp-units-h)))
+  ;; Set up the mechanism to allow inhibiting native-comp via
+  ;; file-local variables.
+  (defvar native-comp--no-native-compile (make-hash-table :test #'equal)))
 
 (when (hash-table-p purify-flag)
   (let ((strings 0)

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Stefan Monnier <monnier@iro.umontreal.ca> writes:
>
>> Re-reading this discussion what I see is two different options, neither
>> of which is significantly better than the other.
>> So the motivation to change is not very high.
>
> True.
>
> So I've now implemented Stefan's suggestion (i.e., just putting a
> puthash into the .elc file if the file-local variable says so), and it
> seems to work fine.
>
> Andrea, does this look OK to you?

In principle absolutely, but I've two questions:

Why not to use `native-comp-deferred-compilation-deny-list' instead of
adding `native-comp--no-native-compile'?

If we can't use `native-comp-deferred-compilation-deny-list' maybe we
should rename `native-comp--no-native-compile' to
`comp--no-native-compile'?  This is the scheme we used for variables
that are not supposed to be manipulated by the user.

Thanks

  Andrea

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Andrea Corallo <akrl@sdf.org> writes:

> Why not to use `native-comp-deferred-compilation-deny-list' instead of
> adding `native-comp--no-native-compile'?

`native-comp-deferred-compilation-deny-list' is a user option, so it
shouldn't be used for programmatic updates like this.  (I.e., a user may
overwrite the automatic updates.)

> If we can't use `native-comp-deferred-compilation-deny-list' maybe we
> should rename `native-comp--no-native-compile' to
> `comp--no-native-compile'?  This is the scheme we used for variables
> that are not supposed to be manipulated by the user.

Right; now done, and change pushed to master.

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Andrea Corallo]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Why not to use `native-comp-deferred-compilation-deny-list' instead of
>> adding `native-comp--no-native-compile'?
>
> `native-comp-deferred-compilation-deny-list' is a user option, so it
> shouldn't be used for programmatic updates like this.  (I.e., a user may
> overwrite the automatic updates.)

Right

>> If we can't use `native-comp-deferred-compilation-deny-list' maybe we
>> should rename `native-comp--no-native-compile' to
>> `comp--no-native-compile'?  This is the scheme we used for variables
>> that are not supposed to be manipulated by the user.
>
> Right; now done, and change pushed to master.

Thanks

  Andrea

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  germanp82@hotmail.com,
>   57627@debbugs.gnu.org,  Eli Zaretskii <eliz@gnu.org>
> Date: Mon, 17 Oct 2022 10:49:23 +0200
> 
> > If we can't use `native-comp-deferred-compilation-deny-list' maybe we
> > should rename `native-comp--no-native-compile' to
> > `comp--no-native-compile'?  This is the scheme we used for variables
> > that are not supposed to be manipulated by the user.
> 
> Right; now done, and change pushed to master.

I suggest some minimally suitable doc string for that variable.  At
least something which says "Internal use only.".

And why doesn't "C-h v" say in what file was the variable defined?  is
that a bug?

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Eli Zaretskii <eliz@gnu.org> writes:

> I suggest some minimally suitable doc string for that variable.  At
> least something which says "Internal use only.".

We commonly don't add doc strings to internal variables (except when
their usage has to be explained to other developers, which I don't think
is the case here).

> And why doesn't "C-h v" say in what file was the variable defined?  is
> that a bug?

Good question.  Could it be because it's not defined at top-level?
(Which I'm not sure is the correct solution anyway.)

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Arash Esbati]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> So I've now implemented Stefan's suggestion (i.e., just putting a
> puthash into the .elc file if the file-local variable says so), and it
> seems to work fine.

Thanks for looking at this.  When I re-start Emacs (after initial
native-compiling all .elc files I'm using), it creates a
*Async-native-compile-log* buffer with this single line in it:

  Compilation finished.

Is this the intended behavior?  I.e., will there always be the
*Async-native-compile-log* at start up telling me that no new .elc file
was compiled?

Best, Arash

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Arash Esbati <arash@gnu.org> writes:

> Thanks for looking at this.  When I re-start Emacs (after initial
> native-compiling all .elc files I'm using), it creates a
> *Async-native-compile-log* buffer with this single line in it:
>
>   Compilation finished.
>
> Is this the intended behavior?  I.e., will there always be the
> *Async-native-compile-log* at start up telling me that no new .elc file
> was compiled?

No, I think this means that you haven't regenerated all the relevant
loaddef*elc files.  Try saying "make bootstrap" and see whether the
problem goes away.

If it doesn't, it a bug somewhere.

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Arash Esbati]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Arash Esbati <arash@gnu.org> writes:
>
>> Thanks for looking at this.  When I re-start Emacs (after initial
>> native-compiling all .elc files I'm using), it creates a
>> *Async-native-compile-log* buffer with this single line in it:
>>
>>   Compilation finished.
>>
>> Is this the intended behavior?  I.e., will there always be the
>> *Async-native-compile-log* at start up telling me that no new .elc file
>> was compiled?
>
> No, I think this means that you haven't regenerated all the relevant
> loaddef*elc files.  Try saying "make bootstrap" and see whether the
> problem goes away.
>
> If it doesn't, it a bug somewhere.

My build script does

  git clean -fdx --exclude=ChangeLog
  ./autogen.sh
  ...

So I think I'm on the safe side reg. regenerated files.

Best, Arash

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Arash Esbati <arash@gnu.org> writes:

> My build script does
>
>   git clean -fdx --exclude=ChangeLog
>   ./autogen.sh
>   ...
>
> So I think I'm on the safe side reg. regenerated files.

Hm...  Oh, I think I see the problem...

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Hm...  Oh, I think I see the problem...

Now fixed, I think.  (I've tested a full new built to see whether this
tweak leads to any problems, but looks OK so far.)

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[Arash Esbati]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Now fixed, I think.  (I've tested a full new built to see whether this
> tweak leads to any problems, but looks OK so far.)

Thanks for the quick fix, the issue is gone.

Best, Arash

bug#57627: 29.0.50; [native-compilation] cl-loaddefs.el recompiled on startup

[German Pacenza]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
> Now fixed, I think.  (I've tested a full new built to see whether this
> tweak leads to any problems, but looks OK so far.)

It looks OK on my machine. No problems detected. Thanks.

-- 
German Pacenza