bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

[T.V Raman via Bug reports for GNU Emacs, the Swiss army knife of text editors]

1. Large packages that span multiple files can have build-order
   dependencies e.g. files containing macros need to be built and
   loaded when compiling the rest of the package.

   2. At present, jitted .eln files appear to be compiled from the .el
      files, and that misses these dependencies, leading to spurious
      warnings, and possibly incorrect behavior.

      3. Attempting to fix this by using -f batch-native-compile fails
         in one and possibly two ways:

         A. If the Makefile defines its build rules   using .elc and
            .el suffixes, then one needs to first  remove the .elc
            files before  running make in order to generate the .eln
            files. These .eln files  end up in .emacs.d/eln-cache;

B. however if one then rebuilds the .elc files, those are now
            newer than the .eln files, which means the .eln files get
            jitted anyway. That jit run of course produces all the
            afore-mentioned problems.
            

            Suggestion:  have batch-native-compile generate both .eln
            and .elc files -- or alternatively, have the jit compiler
            generate the .eln files from .elc files if the .elc files
            are newer. The latter solution would be nice since it
            isolates package developers from having to think about
            native compilation, and differences between .eln and .elc
            files with respect to warnings and behavior.
            

--Raman
            
            


In GNU Emacs 28.0.50 (build 2, x86_64-pc-linux-gnu, GTK+ Version 3.24.24, cairo version 1.16.0)
 of 2021-05-29 built on raman-glaptop
Repository revision: f163b9f426c6bd0bec87c0985bca4f838b9eee89
Repository branch: native-emacs
Windowing system distributor 'The X.Org Foundation', version 11.0.12011000
System Description: Debian GNU/Linux rodete

Configured using:
 'configure --enable-silent-rules --without-xwidgets --with-mailutils
 --without-compress-install --with-native-compilation
 --program-prefix=native- LDFLAGS=-O3 'CPPFLAGS=-Ofast ''

Configured features:
ACL CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS GPM GSETTINGS HARFBUZZ
JPEG JSON LCMS2 LIBOTF LIBSELINUX LIBXML2 M17N_FLT MODULES NATIVE_COMP
NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP SOUND THREADS TIFF
TOOLKIT_SCROLL_BARS X11 XDBE XIM XPM GTK3 ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  value of $XMODIFIERS: @im=none
  locale-coding-system: utf-8-unix

Major mode: Shell

Minor modes in effect:
  recentf-mode: t
  dired-omit-mode: t
  dirtrack-procfs-mode: t
  savehist-mode: t
  save-place-mode: t
  psession-mode: t
  psession-autosave-mode: t
  midnight-mode: t
  magit-wip-initial-backup-mode: t
  magit-wip-before-change-mode: t
  magit-wip-after-apply-mode: t
  magit-wip-after-save-mode: t
  magit-wip-mode: t
  global-git-commit-mode: t
  ido-ubiquitous-mode: t
  flx-ido-mode: t
  ido-everywhere: t
  display-time-mode: t
  disable-mouse-global-mode: t
  company-statistics-mode: t
  company-prescient-mode: t
  prescient-persist-mode: t
  cl-font-lock-built-in-mode: t
  auto-correct-mode: t
  global-voice-lock-mode: t
  voice-lock-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tab-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
/home/raman/emacs/lisp/emacspeak/lisp/tapestry hides /home/raman/emacs/lisp/site-lisp/vm/lisp/tapestry
/home/raman/.emacs.d/elpa/lispy-20210121.926/elpa hides /home/raman/.emacs.d/elpa/ivy-20210518.1815/elpa
/home/raman/.emacs.d/elpa/transient-20210525.1141/transient hides /home/raman/sourceforge/native-emacs/lisp/transient
/home/raman/emacs/lisp/emacspeak/lisp/tetris hides /home/raman/sourceforge/native-emacs/lisp/play/tetris

Features:
(shadow mailalias emacsbug shr-color mm-archive mule-util hl-line
finder-inf emacspeak-paradox paradox ...)

Memory information:
((conses 16 3018329 1355737)
 (symbols 48 72873 30)
 (strings 32 378570 157451)
 (string-bytes 1 15281924)
 (vectors 16 191362)
 (vector-slots 8 5234800 796305)
 (floats 8 2349 1254)
 (intervals 56 284641 71727)
 (buffers 992 45))


-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

[Eli Zaretskii]

> Date: Sun, 30 May 2021 06:57:54 -0700 (PDT)
> From:  "T.V Raman" via "Bug reports for GNU Emacs,
>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
> 
> 
> 1. Large packages that span multiple files can have build-order
>    dependencies e.g. files containing macros need to be built and
>    loaded when compiling the rest of the package.

I became stuck right here at the 1st item: what do you mean by "files
containing macros need to be built and loaded"?  For the "loaded"
part, we use 'require' and 'eval-when-compile', and these work with
*.el files exactly as they do with *.elc or *.eln.  So what exactly is
the problem you are alluding to here, and in particularly what happens
when you "build" these files with macros that requires them to be
built?

Without understanding this, I cannot follow the rest of your
description.

(Andrea, I hope you are following this.)

bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

[T.V Raman via Bug reports for GNU Emacs, the Swiss army knife of text editors]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1423 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

As a simple example, say your package uses a defstruct defined using
cl-defstruct.
(cl-defstruct foo a b c )

Other files in the package use
functions generated by defstruct such as make-foo  and foo-a

At present, when jitted, those other files produce warnings at random
such as make-foo is not known to be defined.

>> Date: Sun, 30 May 2021 06:57:54 -0700 (PDT)
>> From:  "T.V Raman" via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> 
>> 1. Large packages that span multiple files can have build-order
>>    dependencies e.g. files containing macros need to be built and
>>    loaded when compiling the rest of the package.
>
> I became stuck right here at the 1st item: what do you mean by "files
> containing macros need to be built and loaded"?  For the "loaded"
> part, we use 'require' and 'eval-when-compile', and these work with
> *.el files exactly as they do with *.elc or *.eln.  So what exactly is
> the problem you are alluding to here, and in particularly what happens
> when you "build" these files with macros that requires them to be
> built?
>
> Without understanding this, I cannot follow the rest of your
> description.
>
> (Andrea, I hope you are following this.)

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
7©4 Id: kg:/m/0285kf1  •0Ü8

bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Cc: Andrea Corallo <akrl@sdf.org>,  48743@debbugs.gnu.org
> Date: Sun, 30 May 2021 08:02:38 -0700
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> As a simple example, say your package uses a defstruct defined using
> cl-defstruct.
> (cl-defstruct foo a b c )
> 
> Other files in the package use
> functions generated by defstruct such as make-foo  and foo-a

And 'require'-ing the file with cl-defstruct doesn't solve the
problem?

bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sun, 30 May 2021 06:57:54 -0700 (PDT)
>> From:  "T.V Raman" via "Bug reports for GNU Emacs,
>>  the Swiss army knife of text editors" <bug-gnu-emacs@gnu.org>
>> 
>> 
>> 1. Large packages that span multiple files can have build-order
>>    dependencies e.g. files containing macros need to be built and
>>    loaded when compiling the rest of the package.
>
> I became stuck right here at the 1st item: what do you mean by "files
> containing macros need to be built and loaded"?  For the "loaded"
> part, we use 'require' and 'eval-when-compile', and these work with
> *.el files exactly as they do with *.elc or *.eln.  So what exactly is
> the problem you are alluding to here, and in particularly what happens
> when you "build" these files with macros that requires them to be
> built?
>
> Without understanding this, I cannot follow the rest of your
> description.
>
> (Andrea, I hope you are following this.)

Hi Eli,

yes I'm following this thread, trying ATM to make my mind on what's the
real issue.

  Andrea

bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: "T.V Raman" <raman@google.com>, 48743@debbugs.gnu.org
> Date: Mon, 31 May 2021 13:41:28 +0000
> 
> yes I'm following this thread, trying ATM to make my mind on what's the
> real issue.

Thanks.  Let me try helping you understand what I think is the issue
here.

Suppose you are maintaining a Lisp package which includes a Makefile
used to byte-compile all the Lisp files as part of the installation
procedure of the package.  This Makefile has targets and dependencies
that build *.elc files from *.el files.  How do you modify the
Makefile to produce *.eln files during the build, while maintaining
the dependencies between the Lisp files? the dependencies would mean
that when file1.el is modified, you need to recompile not only that
file1.el, but also file2.el and file3.el.  How do you get new
file2.eln and file3.eln via Makefile rules in this case?

I hope I succeeded to explain the issue.

bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: "T.V Raman" <raman@google.com>, 48743@debbugs.gnu.org
>> Date: Mon, 31 May 2021 13:41:28 +0000
>> 
>> yes I'm following this thread, trying ATM to make my mind on what's the
>> real issue.
>
> Thanks.  Let me try helping you understand what I think is the issue
> here.
>
> Suppose you are maintaining a Lisp package which includes a Makefile
> used to byte-compile all the Lisp files as part of the installation
> procedure of the package.  This Makefile has targets and dependencies
> that build *.elc files from *.el files.  How do you modify the
> Makefile to produce *.eln files during the build, while maintaining
> the dependencies between the Lisp files? the dependencies would mean
> that when file1.el is modified, you need to recompile not only that
> file1.el, but also file2.el and file3.el.  How do you get new
> file2.eln and file3.eln via Makefile rules in this case?
>
> I hope I succeeded to explain the issue.

I see thanks.

I think the way to do it is to proceed as we do in the current Emacs
build that is; express to make only the .elc targets and produce them
using `batch-byte-native-compile-for-bootstrap'.

This is like `batch-byte-native-compile' but produces also the .elc
files.

At this point if packages wants to use this function we should probably
rename it as is not anymore in use only for our build process.

Does this make sense?

Thanks

  Andrea

bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: raman@google.com, 48743@debbugs.gnu.org
> Date: Mon, 31 May 2021 16:56:51 +0000
> 
> I think the way to do it is to proceed as we do in the current Emacs
> build that is; express to make only the .elc targets and produce them
> using `batch-byte-native-compile-for-bootstrap'.
> 
> This is like `batch-byte-native-compile' but produces also the .elc
> files.
> 
> At this point if packages wants to use this function we should probably
> rename it as is not anymore in use only for our build process.

Yes, we should rename it, and we should also provide an easy way of
controlling where the *.eln files are deposited, since currently that
function is tightly coupled with the Emacs build process and the
directory structure of the Emacs source tree.

Thanks.

bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: raman@google.com, 48743@debbugs.gnu.org
>> Date: Mon, 31 May 2021 16:56:51 +0000
>> 
>> I think the way to do it is to proceed as we do in the current Emacs
>> build that is; express to make only the .elc targets and produce them
>> using `batch-byte-native-compile-for-bootstrap'.
>> 
>> This is like `batch-byte-native-compile' but produces also the .elc
>> files.
>> 
>> At this point if packages wants to use this function we should probably
>> rename it as is not anymore in use only for our build process.
>
> Yes, we should rename it,

Agree, what should be a good name for that?

> and we should also provide an easy way of
> controlling where the *.eln files are deposited, since currently that
> function is tightly coupled with the Emacs build process and the
> directory structure of the Emacs source tree.

ATM one could push the new target directory in
`native-comp-eln-load-path' maybe using -eval just before invoking the
batch compilation, if this is not sufficient what should be a better
interface to offer?

TIA

  Andrea

bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: raman@google.com, 48743@debbugs.gnu.org
> Date: Mon, 31 May 2021 18:27:10 +0000
> 
> > Yes, we should rename it,
> 
> Agree, what should be a good name for that?

How about batch-byte+native-compile ?

> > and we should also provide an easy way of
> > controlling where the *.eln files are deposited, since currently that
> > function is tightly coupled with the Emacs build process and the
> > directory structure of the Emacs source tree.
> 
> ATM one could push the new target directory in
> `native-comp-eln-load-path' maybe using -eval just before invoking the
> batch compilation

Yes, I thought about this, but it could cause unintended consequences,
and in any case I wouldn't call that "easy".  Can we have a simple
variable that could be bound via --eval?

bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: raman@google.com, 48743@debbugs.gnu.org
>> Date: Mon, 31 May 2021 18:27:10 +0000
>> 
>> > Yes, we should rename it,
>> 
>> Agree, what should be a good name for that?
>
> How about batch-byte+native-compile ?

Sounds good.

>> > and we should also provide an easy way of
>> > controlling where the *.eln files are deposited, since currently that
>> > function is tightly coupled with the Emacs build process and the
>> > directory structure of the Emacs source tree.
>> 
>> ATM one could push the new target directory in
>> `native-comp-eln-load-path' maybe using -eval just before invoking the
>> batch compilation
>
> Yes, I thought about this, but it could cause unintended consequences,
> and in any case I wouldn't call that "easy".  Can we have a simple
> variable that could be bound via --eval?

How would you suggest this variable to behave?

Thanks

  Andrea

bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: raman@google.com, 48743@debbugs.gnu.org
> Date: Mon, 31 May 2021 18:59:41 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Can we have a simple variable that could be bound via --eval?
> 
> How would you suggest this variable to behave?

If it is bound, use the value instead of using the last member of
native-comp-eln-load-path.

bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: raman@google.com, 48743@debbugs.gnu.org
>> Date: Mon, 31 May 2021 18:59:41 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Can we have a simple variable that could be bound via --eval?
>> 
>> How would you suggest this variable to behave?
>
> If it is bound, use the value instead of using the last member of
> native-comp-eln-load-path.

Okay c4b02dad9b a32e65b357 are implementing the renaming and the
addition of the `native-compile-target-directory' variable.  Please have
a look.

TIA

  Andrea

bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

[Andrea Corallo]

Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of
text editors" <bug-gnu-emacs@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Andrea Corallo <akrl@sdf.org>
>>> Cc: raman@google.com, 48743@debbugs.gnu.org
>>> Date: Mon, 31 May 2021 18:59:41 +0000
>>> 
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>> 
>>> > Can we have a simple variable that could be bound via --eval?
>>> 
>>> How would you suggest this variable to behave?
>>
>> If it is bound, use the value instead of using the last member of
>> native-comp-eln-load-path.
>
> Okay c4b02dad9b a32e65b357 are implementing the renaming and the
> addition of the `native-compile-target-directory' variable.  Please have
> a look.

Closing this as I think a solution was provided.

Happy to reopen if necessary.

Thanks

  Andrea

bug#48743: 28.0.50; batch-native-compile should produce .elc files as well

[T.V Raman via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Not quite; the underlying issue -- eln files should be generated from
.elc and not .el files remains.

This bites external packages -- code bundled with Emacs does not have
an issue.

I'll stop asking if the above is deemed "below the line" by  the
emacs-devel team; but with the caveat that I for one will avoid
native-compile entirely  since when this breaks, it breaks
non-deterministically.

Andrea Corallo writes:
 > Andrea Corallo via "Bug reports for GNU Emacs, the Swiss army knife of
 > text editors" <bug-gnu-emacs@gnu.org> writes:
 > 
 > > Eli Zaretskii <eliz@gnu.org> writes:
 > >
 > >>> From: Andrea Corallo <akrl@sdf.org>
 > >>> Cc: raman@google.com, 48743@debbugs.gnu.org
 > >>> Date: Mon, 31 May 2021 18:59:41 +0000
 > >>> 
 > >>> Eli Zaretskii <eliz@gnu.org> writes:
 > >>> 
 > >>> > Can we have a simple variable that could be bound via --eval?
 > >>> 
 > >>> How would you suggest this variable to behave?
 > >>
 > >> If it is bound, use the value instead of using the last member of
 > >> native-comp-eln-load-path.
 > >
 > > Okay c4b02dad9b a32e65b357 are implementing the renaming and the
 > > addition of the `native-compile-target-directory' variable.  Please have
 > > a look.
 > 
 > Closing this as I think a solution was provided.
 > 
 > Happy to reopen if necessary.
 > 
 > Thanks
 > 
 >   Andrea

-- 

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮

--

Thanks,

--Raman(I Search, I Find, I Misplace, I Research)
♉ Id: kg:/m/0285kf1  🦮