bug#73303: 30.0.91; Native compiler repeatedly interrupts at random moments

bug#73303: 30.0.91; Native compiler repeatedly interrupts at random moments

[N. Jackson]

Since building Emacs 30.0.91, I have repeatedly been interrupted in
my work by warnings from the native compiler.

A few examples of the warnings are:

  Warning (native-compiler): ~/.config/emacs/modules/cdlatex.el:1025:26: Warning: the function ‘reftex-what-environment’ is not known to be defined.
  Warning (native-compiler): python-el-fgallina-expansions.el:161:10: Warning: the function ‘python-nav-end-of-block’ is not known to be defined.
  Warning (native-compiler): python-el-fgallina-expansions.el:138:8: Warning: the function ‘python-util-forward-comment’ is not known to be defined.
  Warning (native-compiler): python-el-fgallina-expansions.el:94:4: Warning: the function ‘python-nav-beginning-of-statement’ is not known to be defined.
  Warning (native-compiler): python-el-fgallina-expansions.el:92:4: Warning: the function ‘python-nav-end-of-statement’ is not known to be defined.
  Warning (native-compiler): python-el-fgallina-expansions.el:62:31: Warning: the function ‘python-syntax-context’ is not known to be defined.
  Warning (native-compiler): python-el-fgallina-expansions.el:39:39: Warning: the function ‘python-indent’ is not known to be defined.
  Warning (native-compiler): python-el-fgallina-expansions.el:37:40: Warning: the function ‘python-info-ppss-context’ is not known to be defined.

[That first warning (about cdlatex) is now fixed (I think) by
upgrading to the cdlatex in Elpa, but the point is not the specific
warnings, but the fact that they pop up at inopportune moments.]

This behaviour is quite annoying and I wonder if it would not be
better if the native compiler compiled everything when Emacs is
started and reported all the errors/warnings then.

I supoose that might increase Emacs startup time which for some
users would be unacceptable, but maybe it could happen the first
time Emacs is started and after updating packages and after changing
configuration.


In GNU Emacs 30.0.91 (build 2, x86_64-pc-linux-gnu, GTK+ Version
 3.24.41, cairo version 1.18.0) of 2024-09-12 built on fedora
Windowing system distributor 'The X.Org Foundation', version 11.0.12014000
System Description: Fedora Linux 39 (Xfce)

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ
JPEG LIBOTF LIBSELINUX LIBSYSTEMD LIBXML2 M17N_FLT MODULES
NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND SQLITE3
THREADS TIFF TOOLKIT_SCROLL_BARS WEBP X11 XDBE XIM XINPUT2 XPM GTK3
ZLIB

Important settings:
  value of $LANG: en_CA.utf8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix

Major mode: Text

Minor modes in effect:
  flyspell-mode: t
  recentf-mode: t
  erc-track-mode: t
  erc-ring-mode: t
  erc-notifications-mode: t
  erc-netsplit-mode: t
  erc-menu-mode: t
  erc-match-mode: t
  erc-list-mode: t
  erc-irccontrols-mode: t
  erc-noncommands-mode: t
  erc-move-to-prompt-mode: t
  erc-readonly-mode: t
  erc-button-mode: t
  erc-fill-mode: t
  erc-stamp-mode: t
  erc-autojoin-mode: t
  yas-global-mode: t
  yas-minor-mode: t
  savehist-mode: t
  save-place-mode: t
  erc-networks-mode: t
  electric-pair-mode: t
  display-time-mode: t
  display-battery-mode: t
  desktop-save-mode: t
  delete-selection-mode: t
  cua-mode: t
  tooltip-mode: t
  global-eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  minibuffer-regexp-mode: t
  size-indication-mode: t
  column-number-mode: t
  line-number-mode: t
  global-visual-line-mode: t
  visual-line-mode: t
  transient-mark-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t
  temp-buffer-resize-mode: t
  abbrev-mode: t

Load-path shadows:
None found.

Features:
(shadow sort bbdb-message mail-extr emacsbug message puny dired
dired-loaddefs rfc822 mml mml-sec epa derived epg rfc6068 epg-config
gnus-util mm-decode mm-bodies mm-encode mail-parse rfc2231 gmm-utils
mailheader sendmail rfc2047 rfc2045 ietf-drums mm-util mail-prsvr
mail-utils view solar cal-dst holidays holiday-loaddefs ol-bbdb
org-duration cal-iso face-remap cdlatex reftex reftex-loaddefs
reftex-vars emacs-news-mode mule-util yank-media oc-basic bibtex
iso8601 org-habit display-fill-column-indicator display-line-numbers
vc-git diff-mode track-changes easy-mmode vc-dispatcher flyspell
ispell kmacro mines cookie1 gamegrid transpar expand-region
text-mode-expansions the-org-mode-expansions
python-el-fgallina-expansions er-basic-expansions expand-region-core
expand-region-custom hydra advice lv compile text-property-search
org-clock comp-run comp-common org-agenda org-element org-persist
xdg org-id org-element-ast inline avl-tree generator org-refile org
org-macro org-pcomplete org-list org-footnote org-faces org-entities
noutline outline ob-shell shell ob-R ob-python python project
ob-plantuml ob-org ob-gnuplot ob-ditaa ob-calc calc-store calc-trail
calc-ext calc calc-loaddefs rect calc-macs ob-awk ob-dot ob-maxima
ob ob-tangle org-src sh-script smie treesit executable ob-ref ob-lob
ob-table ob-exp ob-comint ob-emacs-lisp ob-core ob-eval org-cycle
org-table org-keys oc org-loaddefs thingatpt find-func ol org-fold
org-fold-core org-compat org-version org-macs bbdb-anniv diary-lib
diary-loaddefs cal-menu calendar cal-loaddefs bbdb-com crm
mailabbrev bbdb bbdb-site timezone recentf tree-widget cus-edit pp
ido erc-track erc-ring erc-desktop-notifications notifications
erc-netsplit erc-menu erc-match erc-list erc-goodies erc-pcomplete
time-date pcomplete comint ansi-osc ansi-color ring erc-button
erc-fill erc-stamp wid-edit erc-join modus-vivendi-theme
modus-themes yasnippet-classic-snippets cl-extra yasnippet help-mode
savehist saveplace company pcase erc format-spec erc-backend
erc-networks erc-common erc-compat compat erc-loaddefs elec-pair
time battery dbus xml desktop frameset delsel cua-base cus-load
ace-window-autoloads auctex-autoloads tex-site avy-autoloads
bbdb-autoloads cdlatex-autoloads company-autoloads
csv-mode-autoloads debbugs-autoloads ess-autoloads
expand-region-autoloads geiser-autoloads info orderless-autoloads rx
sql-indent-autoloads yasnippet-autoloads package browse-url url
url-proxy url-privacy url-expand url-methods url-history url-cookie
generate-lisp-file url-domsuf url-util mailcap url-handlers
url-parse auth-source cl-seq eieio eieio-core cl-macs icons
password-cache json subr-x map byte-opt gv bytecomp byte-compile
url-vars cl-loaddefs cl-lib rmc iso-transl tooltip cconv eldoc paren
electric uniquify ediff-hook vc-hooks lisp-float-type elisp-mode
mwheel term/x-win x-win term/common-win x-dnd touch-screen tool-bar
dnd fontset image regexp-opt fringe tabulated-list replace
newcomment text-mode lisp-mode prog-mode register page tab-bar
menu-bar rfn-eshadow isearch easymenu timer select scroll-bar mouse
jit-lock font-lock syntax font-core term/tty-colors frame minibuffer
nadvice seq simple cl-generic indonesian philippine cham georgian
utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao korean
japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj
charscript charprop case-table epa-hook jka-cmpr-hook help abbrev
obarray oclosure cl-preloaded button loaddefs theme-loaddefs faces
cus-face macroexp files window text-properties overlay sha1 md5
base64 format env code-pages mule custom widget keymap
hashtable-print-readable backquote threads dbusbind inotify
dynamic-setting system-font-setting font-render-setting cairo gtk
x-toolkit xinput2 x multi-tty move-toolbar make-network-process
native-compile emacs)

Memory information:
((conses 16 1087810 298689) (symbols 48 33261 3)
 (strings 32 140337 19623) (string-bytes 1 4291611) (vectors 16 87466)
 (vector-slots 8 1784398 521241) (floats 8 651 2058)
 (intervals 56 8991 6851) (buffers 984 30))

bug#73303: 30.0.91; Native compiler repeatedly interrupts at random moments

[Eli Zaretskii]

tags 73303 notabug
thanks

> From: "N. Jackson" <njackson@posteo.net>
> Date: Mon, 16 Sep 2024 18:14:46 +0000
> 
> 
> Since building Emacs 30.0.91, I have repeatedly been interrupted in
> my work by warnings from the native compiler.
> 
> A few examples of the warnings are:
> 
>   Warning (native-compiler): ~/.config/emacs/modules/cdlatex.el:1025:26: Warning: the function ‘reftex-what-environment’ is not known to be defined.
>   Warning (native-compiler): python-el-fgallina-expansions.el:161:10: Warning: the function ‘python-nav-end-of-block’ is not known to be defined.
>   Warning (native-compiler): python-el-fgallina-expansions.el:138:8: Warning: the function ‘python-util-forward-comment’ is not known to be defined.
>   Warning (native-compiler): python-el-fgallina-expansions.el:94:4: Warning: the function ‘python-nav-beginning-of-statement’ is not known to be defined.
>   Warning (native-compiler): python-el-fgallina-expansions.el:92:4: Warning: the function ‘python-nav-end-of-statement’ is not known to be defined.
>   Warning (native-compiler): python-el-fgallina-expansions.el:62:31: Warning: the function ‘python-syntax-context’ is not known to be defined.
>   Warning (native-compiler): python-el-fgallina-expansions.el:39:39: Warning: the function ‘python-indent’ is not known to be defined.
>   Warning (native-compiler): python-el-fgallina-expansions.el:37:40: Warning: the function ‘python-info-ppss-context’ is not known to be defined.

These warnings usually mean that the offending Lisp file lacks some
'require's.  JIT native compilation runs in a separate Emacs session,
which only loads the file it compiles, so any missing 'require's or
autoloading cookies trigger these warnings, whereas when the same
packages are loaded into your main Emacs session, they can benefit
from packages loaded earlier in the session.

IOW, these are minor bugs in the compiled files which need to be fixed
in those files and by their respective developers.

> This behaviour is quite annoying and I wonder if it would not be
> better if the native compiler compiled everything when Emacs is
> started and reported all the errors/warnings then.

You can disable these warnings if you are annoyed by them.  They are
just warnings, and will not usually cause any problems when using the
compiled code.  See native-comp-async-report-warnings-errors.

You can cause these compilations to happen at the beginning of your
Emacs session if you change your init files such that Emacs loads all
these files.  JIT native compilation is triggered by loading a .elc
file that doesn't yet have a corresponding .eln file, so by loading
them at the beginning, you will force Emacs to native-compile them all
at that time.

In any case, Emacs compiles each Lisp file just once, so you should
only see these when you start a new Emacs version for the first time.

> I supoose that might increase Emacs startup time which for some
> users would be unacceptable, but maybe it could happen the first
> time Emacs is started and after updating packages and after changing
> configuration.

It shouldn't increase startup time because the compilation is run in
separate processes, and those use other CPU cores.  Emacs doesn't wait
for the compilation to end before using a package; it uses the byte
code until the native compilation ends, and then loads the
native-compiled code when the compilation ends to replace the byte
code.

I see no bug in what you describe.

bug#73303: 30.0.91; Native compiler repeatedly interrupts at random moments

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

> tags 73303 notabug
> thanks
>
>> From: "N. Jackson" <njackson@posteo.net>
>> Date: Mon, 16 Sep 2024 18:14:46 +0000
>> 
>> 
>> Since building Emacs 30.0.91, I have repeatedly been interrupted in
>> my work by warnings from the native compiler.
>> 
>> A few examples of the warnings are:
>> 
>>   Warning (native-compiler): ~/.config/emacs/modules/cdlatex.el:1025:26: Warning: the function ‘reftex-what-environment’ is not known to be defined.
>>   Warning (native-compiler): python-el-fgallina-expansions.el:161:10: Warning: the function ‘python-nav-end-of-block’ is not known to be defined.
>>   Warning (native-compiler): python-el-fgallina-expansions.el:138:8: Warning: the function ‘python-util-forward-comment’ is not known to be defined.
>>   Warning (native-compiler): python-el-fgallina-expansions.el:94:4: Warning: the function ‘python-nav-beginning-of-statement’ is not known to be defined.
>>   Warning (native-compiler): python-el-fgallina-expansions.el:92:4: Warning: the function ‘python-nav-end-of-statement’ is not known to be defined.
>>   Warning (native-compiler): python-el-fgallina-expansions.el:62:31: Warning: the function ‘python-syntax-context’ is not known to be defined.
>>   Warning (native-compiler): python-el-fgallina-expansions.el:39:39: Warning: the function ‘python-indent’ is not known to be defined.
>>   Warning (native-compiler): python-el-fgallina-expansions.el:37:40: Warning: the function ‘python-info-ppss-context’ is not known to be defined.
>
> These warnings usually mean that the offending Lisp file lacks some
> 'require's.  JIT native compilation runs in a separate Emacs session,
> which only loads the file it compiles, so any missing 'require's or
> autoloading cookies trigger these warnings, whereas when the same
> packages are loaded into your main Emacs session, they can benefit
> from packages loaded earlier in the session.
>
> IOW, these are minor bugs in the compiled files which need to be fixed
> in those files and by their respective developers.
>
>> This behaviour is quite annoying and I wonder if it would not be
>> better if the native compiler compiled everything when Emacs is
>> started and reported all the errors/warnings then.
>
> You can disable these warnings if you are annoyed by them.  They are
> just warnings, and will not usually cause any problems when using the
> compiled code.  See native-comp-async-report-warnings-errors.
>
> You can cause these compilations to happen at the beginning of your
> Emacs session if you change your init files such that Emacs loads all
> these files.  JIT native compilation is triggered by loading a .elc
> file that doesn't yet have a corresponding .eln file, so by loading
> them at the beginning, you will force Emacs to native-compile them all
> at that time.
>
> In any case, Emacs compiles each Lisp file just once, so you should
> only see these when you start a new Emacs version for the first time.
>
>> I supoose that might increase Emacs startup time which for some
>> users would be unacceptable, but maybe it could happen the first
>> time Emacs is started and after updating packages and after changing
>> configuration.
>
> It shouldn't increase startup time because the compilation is run in
> separate processes, and those use other CPU cores.  Emacs doesn't wait
> for the compilation to end before using a package; it uses the byte
> code until the native compilation ends, and then loads the
> native-compiled code when the compilation ends to replace the byte
> code.
>
> I see no bug in what you describe.


Agree, that's the intended behavior.  I think the python* warnings
should be reported as bugs to
<https://github.com/magnars/expand-region.el>.

  Andrea

bug#73303: 30.0.91; Native compiler repeatedly interrupts at random moments

[N. Jackson]

Hello Magnar,

In this Emacs Bug thread[1] (about native compiler warnings popping
up at inopportune moments), it was suggested that someone report to
you some specific warnings being caused by your package
expand-region, and I am doing so by including you in this email.

[1] https://debbugs.gnu.org/cgi/bugreport.cgi?bug=73303

At 15:27 -0400 on Monday 2024-09-16, Andrea Corallo wrote:
>
>>> From: "N. Jackson" <njackson@posteo.net>
>>> Date: Mon, 16 Sep 2024 18:14:46 +0000
>>> 
>>> 
>>> Since building Emacs 30.0.91, I have repeatedly been interrupted in
>>> my work by warnings from the native compiler.
>>> 
>>> A few examples of the warnings are:
>>> 
>>>   Warning (native-compiler): ~/.config/emacs/modules/cdlatex.el:1025:26: Warning: the function ‘reftex-what-environment’ is not known to be defined.
>>>   Warning (native-compiler): python-el-fgallina-expansions.el:161:10: Warning: the function ‘python-nav-end-of-block’ is not known to be defined.
>>>   Warning (native-compiler): python-el-fgallina-expansions.el:138:8: Warning: the function ‘python-util-forward-comment’ is not known to be defined.
>>>   Warning (native-compiler): python-el-fgallina-expansions.el:94:4: Warning: the function ‘python-nav-beginning-of-statement’ is not known to be defined.
>>>   Warning (native-compiler): python-el-fgallina-expansions.el:92:4: Warning: the function ‘python-nav-end-of-statement’ is not known to be defined.
>>>   Warning (native-compiler): python-el-fgallina-expansions.el:62:31: Warning: the function ‘python-syntax-context’ is not known to be defined.
>>>   Warning (native-compiler): python-el-fgallina-expansions.el:39:39: Warning: the function ‘python-indent’ is not known to be defined.
>>>   Warning (native-compiler): python-el-fgallina-expansions.el:37:40: Warning: the function ‘python-info-ppss-context’ is not known to be defined.
>
> I think the python* warnings should be reported as bugs to
> <https://github.com/magnars/expand-region.el>.
>
>   Andrea

Thanks Andrea, that allowed me to track these ones down.  I have in
my init file

  (custom-set-variables
  ...
   '(package-archives
     '(("gnu" . "https://elpa.gnu.org/packages/")))
  ...
   '(package-selected-packages
     '(... expand-region ...))
  ...)

and indeed the offending files (for this particular set of warnings)
are in ~/.config/emacs/elpa/expand-region-1.0.0/.

I don't think I can report this to the author and maintainer of
expand-region on Github because, IIUC, that would require me to run
proprietary JavaScript, but I am including him (Magnar Sveen) in
this email.

N.

bug#73303: 30.0.91; Native compiler repeatedly interrupts at random moments

[Eli Zaretskii]

> From: "N. Jackson" <njackson@posteo.net>
> Cc: 73303@debbugs.gnu.org, Andrea Corallo <acorallo@gnu.org>
> Date: Tue, 17 Sep 2024 15:22:42 +0000
> 
> Hello Eli, thank you for your response.
> 
> At 21:42 +0300 on Monday 2024-09-16, Eli Zaretskii wrote:
> 
> > I see no bug in what you describe.
> 
> Fair enough.  The concern that prompted me to file a bug report was
> not that native compile is broken (a literal bug) but rather that
> since it's a newish feature you might welcome end-user feedback
> about it's default behaviour (or perceived behaviour) that might
> lead to to beneficial tweaks to either the behaviour or to the
> documentation.

The feedback is always welcome, of course (if my wording somehow made
an impression it wasn't, I apologize).  And what you describe is
already known: a few users posted similar experiences during the
development and testing of Emacs 29.  So these issues are not new, and
I think Emacs 30 is better equipped to deal with them, and gives users
more knobs to deal with them.

> > These warnings usually mean that the offending Lisp file lacks
> > some 'require's.  JIT native compilation runs in a separate Emacs
> > session, which only loads the file it compiles, so any missing
> > 'require's or autoloading cookies trigger these warnings, whereas
> > when the same packages are loaded into your main Emacs session,
> > they can benefit from packages loaded earlier in the session.
> ...
> > JIT native compilation is triggered by loading a .elc file that
> > doesn't yet have a corresponding .eln file
> 
> Thank you for the overview of how native compilation works.  I had
> wrongly guessed that it was deferring the compilation lazily and
> doing it later on an idle timer or something of that sort.  If I
> understand what you are saying correctly, the native compiler, not
> having a crystal ball, cannot possibly know ahead of time what .elc
> files a user might load, so it cannot compile them ahead of time.

Yes, exactly.  JIT compilation compiles only packages that you load,
when you load them for the first time.

> However, as a naïve end-user of Emacs, when I am _using_ Emacs, as a
> tool to get a job done -- writing an essay, say -- I don't want to
> think about the guts of how Emacs works.  I'm happy to save that for
> when I'm building Emacs or maintaining my configuration files and
> that is when I would want the native compilation to happen, if that
> were possible.
> 
> You write
> 
> > You can disable these warnings if you are annoyed by them.  They
> > are just warnings, and will not usually cause any problems when
> > using the compiled code.  See
> > native-comp-async-report-warnings-errors.
> 
> Sweeping warnings under the rug isn't something I'm comfortable with
> even if nine times out of ten (or 999 times in 1000) they're false
> alarms.  (I'm the sort of person that -Werror was made for.)

If you set native-comp-async-report-warnings-errors to the value
'silent', the warnings will be logged in the *Warnings* buffer, where
you can later review them, but will not be shown in a popup buffer,
which might distract you when you don't want that.

> > You can cause these compilations to happen at the beginning of your
> > Emacs session if you change your init files such that Emacs loads all
> > these files.
> ...
> > In any case, Emacs compiles each Lisp file just once, so you should
> > only see these when you start a new Emacs version for the first time.
> 
> I guess, then, that I would want to (somehow) process my init files --
> before using Emacs for the first time after upgrading to a new
> version -- to ensure native compilation of all dependencies.
> 
> Do you have any pointers on how to go about doing that? It seems
> impractical for a casual Emacs user since they don't necessary know
> what files will be loaded by the packages they use.  The warnings
> I've seen so far all refer to files I've never heard of.

If your init file arranges for many packages to load only on demand,
then I don't think there is a way, except summarily compile all the
packages under your ~/.emacs.d/ directory (assuming that's where you
install them).  Maybe we should have a native-compile-directory
function to make that easier; currently we only have
emacs-lisp-native-compile, which compiles a single file.  Andrea,
WDYT? 

> I don't suppose there's a function
> native-compile-eagerly-compile-all-dependencies-of-my-init-files-and-do-it-synchronously-right-now?!

We could do that, but before we do' we'd need to come up with a
find-all-dependecies-of-my-init-files function ;-)

> If I can deal with my init files (ensuring that they won't load any
> files that haven't yet been native compiled), it seems I would be
> left with two potential sources of files being native compiled --
> Emacs itself, and newly installed/updated packages.  Two questions
> about that, then:
> 
> 1. Does the native compiler compile all the .elc files in
> Emacs itself into .eln files ahead of time, or does it also leave
> those until they are loaded?

The preloaded files are natively compiled as part of the build, but
the rest are only natively compiled if the build uses the optional
"ahead-of-time" feature, via a switch to the configure script.

> 2. When the package system installs/updates a package, does it
> natively compile the files at the same time as it byte compiles
> them?

I'm not sure, but I think it doesn't.  Philip, can you answer that?

bug#73303: 30.0.91; Native compiler repeatedly interrupts at random moments

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> 2. When the package system installs/updates a package, does it
>> natively compile the files at the same time as it byte compiles
>> them?
>
> I'm not sure, but I think it doesn't.  Philip, can you answer that?

My understanding: Native compilation build on byte-compilation, and all
packages are byte-compiled.  If `package-native-compile' is enabled
(which is not the case by default), then the package is immediately
compiled using native-compilation as well.  This applies both to
installing packages and upgrading them.  I don't know if the native
compiler will detect .elc files automatically and compile them at some
later point.

-- 
	Philip Kaludercic on siskin

bug#73303: 30.0.91; Native compiler repeatedly interrupts at random moments

[N. Jackson]

At 18:46 +0000 on Tuesday 2024-09-17, Philip Kaludercic wrote:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> 2. When the package system installs/updates a package, does it
>>> natively compile the files at the same time as it byte compiles
>>> them?
>>
>> I'm not sure, but I think it doesn't.  Philip, can you answer that?
>
> My understanding: Native compilation build on byte-compilation,
> and all packages are byte-compiled.  If `package-native-compile'
> is enabled (which is not the case by default), then the package is
> immediately compiled using native-compilation as well.  This
> applies both to installing packages and upgrading them.

Shouldn't this be on by default now that native compilation is
enabled by default?  After all, if Emacs is going to be byte
compiling the files anyway, it might as well native compile them at
the same time and get any issues out of the way while the user's
mind is in working-with-packages mode.

> I don't know if the native compiler will detect .elc files
> automatically and compile them at some later point.

When they are loaded, presumably?

bug#73303: 30.0.91; Native compiler repeatedly interrupts at random moments

[Philip Kaludercic]

"N. Jackson" <njackson@posteo.net> writes:

> At 18:46 +0000 on Tuesday 2024-09-17, Philip Kaludercic wrote:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> 2. When the package system installs/updates a package, does it
>>>> natively compile the files at the same time as it byte compiles
>>>> them?
>>>
>>> I'm not sure, but I think it doesn't.  Philip, can you answer that?
>>
>> My understanding: Native compilation build on byte-compilation,
>> and all packages are byte-compiled.  If `package-native-compile'
>> is enabled (which is not the case by default), then the package is
>> immediately compiled using native-compilation as well.  This
>> applies both to installing packages and upgrading them.
>
> Shouldn't this be on by default now that native compilation is
> enabled by default?  After all, if Emacs is going to be byte
> compiling the files anyway, it might as well native compile them at
> the same time and get any issues out of the way while the user's
> mind is in working-with-packages mode.

I have no opinion on this, and as I don't build Emacs with native
compilation I cannot try it out either.

>> I don't know if the native compiler will detect .elc files
>> automatically and compile them at some later point.
>
> When they are loaded, presumably?

Right, Eli confirms this below.

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: "N. Jackson" <njackson@posteo.net>,  73303@debbugs.gnu.org,
>>   acorallo@gnu.org
>> Date: Tue, 17 Sep 2024 18:46:10 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> 2. When the package system installs/updates a package, does it
>> >> natively compile the files at the same time as it byte compiles
>> >> them?
>> >
>> > I'm not sure, but I think it doesn't.  Philip, can you answer that?
>> 
>> My understanding: Native compilation build on byte-compilation, and all
>> packages are byte-compiled.  If `package-native-compile' is enabled
>> (which is not the case by default), then the package is immediately
>> compiled using native-compilation as well.
>
> OK, so this means the OP will have to enable native compilation by
> customizing package-native-compile.

I haven't read through the entire thread, but it sounds like a possible
solution.

>> I don't know if the native compiler will detect .elc files
>> automatically and compile them at some later point.
>
> It will, but only when Emacs loads these .elc files.

-- 
	Philip Kaludercic on siskin

bug#73303: 30.0.91; Native compiler repeatedly interrupts at random moments

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: "N. Jackson" <njackson@posteo.net>,  73303@debbugs.gnu.org,
>   acorallo@gnu.org
> Date: Tue, 17 Sep 2024 18:46:10 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> 2. When the package system installs/updates a package, does it
> >> natively compile the files at the same time as it byte compiles
> >> them?
> >
> > I'm not sure, but I think it doesn't.  Philip, can you answer that?
> 
> My understanding: Native compilation build on byte-compilation, and all
> packages are byte-compiled.  If `package-native-compile' is enabled
> (which is not the case by default), then the package is immediately
> compiled using native-compilation as well.

OK, so this means the OP will have to enable native compilation by
customizing package-native-compile.

> I don't know if the native compiler will detect .elc files
> automatically and compile them at some later point.

It will, but only when Emacs loads these .elc files.

bug#73303: 30.0.91; Native compiler repeatedly interrupts at random moments

[N. Jackson]

At 18:59 +0300 on Tuesday 2024-09-17, Eli Zaretskii wrote:

To: Eli Zaretskii <eliz@gnu.org>
Cc: Philip Kaludercic <philipk@posteo.net>,  73303@debbugs.gnu.org,  acorallo@gnu.org
Subject: Re: bug#73303: 30.0.91; Native compiler repeatedly interrupts at random moments
From: "N. Jackson" <njackson@posteo.net>
Gcc: nnfolder+archive:sent.2024-09
--text follows this line--
At 18:59 +0300 on Tuesday 2024-09-17, Eli Zaretskii wrote:

> The feedback is always welcome, of course (if my wording somehow
> made an impression it wasn't, I apologize).

No worries.  I very much appreciate the huge amount of work you do
on Emacs and it amazes me that anyone could find time for it all.
So if your messages are sometimes somewhat terse, I can appreciate
them for being concise and I've learned to take them literally, at
face value, without imagining hidden implications.  

> So these issues are not new, and I think Emacs 30 is better
> equipped to deal with them, and gives users more knobs to deal
> with them.

More knobs is good, but probably increases the difficulty of
choosing the defaults.

> If you set native-comp-async-report-warnings-errors to the value
> 'silent', the warnings will be logged in the *Warnings* buffer, where
> you can later review them, but will not be shown in a popup buffer,
> which might distract you when you don't want that.

Thank you for that tip, I'll do that.  [I'll never remember to look
at the *Warnings* buffer, but I can set a hook that runs when Emacs
exits that logs the warnings and sets a TODO in Org to remind me to
look at them at an opportune moment.]

> If your init file arranges for many packages to load only on demand,
> then I don't think there is a way, except summarily compile all the
> packages under your ~/.emacs.d/ directory (assuming that's where you
> install them).

I don't think I do that.  Not deliberately anyway.  Almost all the
packages I use were installed with the package manager through the
list-packages interface, although there a few that comes from my
GNU/Linux distribution.

Here it's ~/.config/emacs/ nowadays, and I could compile everything
there but that wouldn't catch the files from the distro.

>> I don't suppose there's a function
>> native-compile-eagerly-compile-all-dependencies-of-my-init-files-and-do-it-synchronously-right-now?!
>
> We could do that, but before we do' we'd need to come up with a
> find-all-dependecies-of-my-init-files function ;-)

Ha!  Well I don't feel so bad now that I couldn't immediately see
exactly how to do it.

But wouldn't it be as simple as just native compiling everything on
load-path?  After all, if the native compilation that produces the
intrusive warnings is triggered by loading a .elc file, mustn't the
offending file be on the load path?  [Unless the file were loaded
explicitly, I suppose, but that case one could handle separately if
one wanted to -- after all, explicit loads should be easy to find.]
Could there be a native-compile-load-path function?

> The preloaded files are natively compiled as part of the build, but
> the rest are only natively compiled if the build uses the optional
> "ahead-of-time" feature, via a switch to the configure script.

Thank you, I'll add that to the things I turn on when I run
configure.

bug#73303: 30.0.91; Native compiler repeatedly interrupts at random moments

[Eli Zaretskii]

> From: "N. Jackson" <njackson@posteo.net>
> Cc: Philip Kaludercic <philipk@posteo.net>,  73303@debbugs.gnu.org,
>   acorallo@gnu.org
> Date: Tue, 17 Sep 2024 20:09:51 +0000
> 
> > If your init file arranges for many packages to load only on demand,
> > then I don't think there is a way, except summarily compile all the
> > packages under your ~/.emacs.d/ directory (assuming that's where you
> > install them).
> 
> I don't think I do that.  Not deliberately anyway.  Almost all the
> packages I use were installed with the package manager through the
> list-packages interface, although there a few that comes from my
> GNU/Linux distribution.

AFAIK, by default package.el installs all packages under
package-user-dir, which on my system is ~/.emacs.d/elpa/.

> >> I don't suppose there's a function
> >> native-compile-eagerly-compile-all-dependencies-of-my-init-files-and-do-it-synchronously-right-now?!
> >
> > We could do that, but before we do' we'd need to come up with a
> > find-all-dependecies-of-my-init-files function ;-)
> 
> Ha!  Well I don't feel so bad now that I couldn't immediately see
> exactly how to do it.
> 
> But wouldn't it be as simple as just native compiling everything on
> load-path?

You could do that, but that would be a huge overkill, I think.

Also, loading all of the files there might get you in trouble, because
not every Lisp package is compatible with every other.  They could
conflict.

bug#73303: 30.0.91; Native compiler repeatedly interrupts at random moments

[N. Jackson]

I'm resending this message to 73303@debbugs.gnu.org only, because
the first time my mail service provider bounced the copy to
73303@debbugs.gnu.org (because debbugs.gnu.org[209.51.188.43]
refused a TLS connection).


Hello Eli, thank you for your response.

At 21:42 +0300 on Monday 2024-09-16, Eli Zaretskii wrote:

> I see no bug in what you describe.

Fair enough.  The concern that prompted me to file a bug report was
not that native compile is broken (a literal bug) but rather that
since it's a newish feature you might welcome end-user feedback
about it's default behaviour (or perceived behaviour) that might
lead to to beneficial tweaks to either the behaviour or to the
documentation.

> These warnings usually mean that the offending Lisp file lacks
> some 'require's.  JIT native compilation runs in a separate Emacs
> session, which only loads the file it compiles, so any missing
> 'require's or autoloading cookies trigger these warnings, whereas
> when the same packages are loaded into your main Emacs session,
> they can benefit from packages loaded earlier in the session.
...
> JIT native compilation is triggered by loading a .elc file that
> doesn't yet have a corresponding .eln file

Thank you for the overview of how native compilation works.  I had
wrongly guessed that it was deferring the compilation lazily and
doing it later on an idle timer or something of that sort.  If I
understand what you are saying correctly, the native compiler, not
having a crystal ball, cannot possibly know ahead of time what .elc
files a user might load, so it cannot compile them ahead of time.

However, as a naïve end-user of Emacs, when I am _using_ Emacs, as a
tool to get a job done -- writing an essay, say -- I don't want to
think about the guts of how Emacs works.  I'm happy to save that for
when I'm building Emacs or maintaining my configuration files and
that is when I would want the native compilation to happen, if that
were possible.

You write

> You can disable these warnings if you are annoyed by them.  They
> are just warnings, and will not usually cause any problems when
> using the compiled code.  See
> native-comp-async-report-warnings-errors.

Sweeping warnings under the rug isn't something I'm comfortable with
even if nine times out of ten (or 999 times in 1000) they're false
alarms.  (I'm the sort of person that -Werror was made for.)

> You can cause these compilations to happen at the beginning of your
> Emacs session if you change your init files such that Emacs loads all
> these files.
...
> In any case, Emacs compiles each Lisp file just once, so you should
> only see these when you start a new Emacs version for the first time.

I guess, then, that I would want to (somehow) process my init files --
before using Emacs for the first time after upgrading to a new
version -- to ensure native compilation of all dependencies.

Do you have any pointers on how to go about doing that? It seems
impractical for a casual Emacs user since they don't necessary know
what files will be loaded by the packages they use.  The warnings
I've seen so far all refer to files I've never heard of.

I don't suppose there's a function
native-compile-eagerly-compile-all-dependencies-of-my-init-files-and-do-it-synchronously-right-now?!


If I can deal with my init files (ensuring that they won't load any
files that haven't yet been native compiled), it seems I would be
left with two potential sources of files being native compiled --
Emacs itself, and newly installed/updated packages.  Two questions
about that, then:

1. Does the native compiler compile all the .elc files in
Emacs itself into .eln files ahead of time, or does it also leave
those until they are loaded?

2. When the package system installs/updates a package, does it
natively compile the files at the same time as it byte compiles
them?