bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

[Phil Sainty]

Now that the native-compilation feature is merged, it would be very
useful to be able to build Emacs --with-native-compilation but be
able to choose to inhibit that functionality at start time via a
command-line option such as 'emacs --no-native-compilation', which
would cause Emacs to load/execute only .el and .elc files.

This will enable users to easily compare functionality with and
without native-compilation, so that native-compilation bugs can be
more easily identified and reproduced without requiring people to
maintain more than one build of Emacs in order to test how the
traditional interpreters behave.

I'm not sure if/how this ties in with the portable dumper.  Perhaps
there are .eln files included in the dump?  If so, perhaps the dump
would need to include both the .elc and the .eln code, and choose
which to use based on the new option.


-Phil

bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

[Eli Zaretskii]

> From: Phil Sainty <psainty@orcon.net.nz>
> Date: Mon, 26 Apr 2021 17:52:52 +1200
> Cc: Andrea Corallo <akrl@sdf.org>
> 
> Now that the native-compilation feature is merged, it would be very
> useful to be able to build Emacs --with-native-compilation but be
> able to choose to inhibit that functionality at start time via a
> command-line option such as 'emacs --no-native-compilation', which
> would cause Emacs to load/execute only .el and .elc files.
> 
> This will enable users to easily compare functionality with and
> without native-compilation, so that native-compilation bugs can be
> more easily identified and reproduced without requiring people to
> maintain more than one build of Emacs in order to test how the
> traditional interpreters behave.
> 
> I'm not sure if/how this ties in with the portable dumper.  Perhaps
> there are .eln files included in the dump?  If so, perhaps the dump
> would need to include both the .elc and the .eln code, and choose
> which to use based on the new option.

Andrea will correct me, but I think this is not trivial to implement,
not even close.  Indeed, the contents of the pdumper file is different
in the two cases, and I see no easy way of having both byte-compiled
and native-compiled stuff live together in the same dump (they define
the same functions, remember?).

We could perhaps provide a special value of --temacs= switch to
temacs, so that the same temacs executable could be dumped into 2
different *.pdmp files, one with natively-compiled preloaded stuff,
the other with byte-compiled stuff; then users could use the existing
option --dump-file= to start Emacs with the non-standard pdumper file
(they will also need to set comp-deferred-compilation to nil to
prevent any run-time native-compilations once Emacs starts).

But frankly, I would hesitate to complicate Emacs even for the latter
possibility.  What you ask for doesn't seem to be a user-level
feature, it is mainly important for Emacs developers, and those can
always build 2 separate binaries (e.g., I already did).  Building a
differently-configured Emacs, even from the same Git repository, is so
easy that I don't really see a justification for a feature like you
describe.

(As for reproducing problems easily: it isn't hard to run the
interpreted or byte-compiled Lisp, if you can identify the relevant
Lisp files involved in the problem: just load them manually.  Andrea,
am I missing something?)

bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phil Sainty <psainty@orcon.net.nz>
>> Date: Mon, 26 Apr 2021 17:52:52 +1200
>> Cc: Andrea Corallo <akrl@sdf.org>
>> 
>> Now that the native-compilation feature is merged, it would be very
>> useful to be able to build Emacs --with-native-compilation but be
>> able to choose to inhibit that functionality at start time via a
>> command-line option such as 'emacs --no-native-compilation', which
>> would cause Emacs to load/execute only .el and .elc files.
>> 
>> This will enable users to easily compare functionality with and
>> without native-compilation, so that native-compilation bugs can be
>> more easily identified and reproduced without requiring people to
>> maintain more than one build of Emacs in order to test how the
>> traditional interpreters behave.
>> 
>> I'm not sure if/how this ties in with the portable dumper.  Perhaps
>> there are .eln files included in the dump?  If so, perhaps the dump
>> would need to include both the .elc and the .eln code, and choose
>> which to use based on the new option.
>
> Andrea will correct me, but I think this is not trivial to implement,
> not even close.  Indeed, the contents of the pdumper file is different
> in the two cases, and I see no easy way of having both byte-compiled
> and native-compiled stuff live together in the same dump (they define
> the same functions, remember?).
>
> We could perhaps provide a special value of --temacs= switch to
> temacs, so that the same temacs executable could be dumped into 2
> different *.pdmp files, one with natively-compiled preloaded stuff,
> the other with byte-compiled stuff; then users could use the existing
> option --dump-file= to start Emacs with the non-standard pdumper file
> (they will also need to set comp-deferred-compilation to nil to
> prevent any run-time native-compilations once Emacs starts).
>
> But frankly, I would hesitate to complicate Emacs even for the latter
> possibility.  What you ask for doesn't seem to be a user-level
> feature, it is mainly important for Emacs developers, and those can
> always build 2 separate binaries (e.g., I already did).  Building a
> differently-configured Emacs, even from the same Git repository, is so
> easy that I don't really see a justification for a feature like you
> describe.
>
> (As for reproducing problems easily: it isn't hard to run the
> interpreted or byte-compiled Lisp, if you can identify the relevant
> Lisp files involved in the problem: just load them manually.  Andrea,
> am I missing something?)

No you are not, once we bootstrap and dump a native compiled Emacs
there's no way we can undone it and get the equivalent one with only
bytecode.

Other than I can mention some knobs we already have that might partially
help here:

- inibith the automatic native compilation of new code with
 `comp-deferred-compilation'.

- prevent .eln from being loaded in place of bytecode with
  `load-no-native'.

Thanks

  Andrea

bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

[Phil Sainty]

> Eli Zaretskii <eliz@gnu.org> writes:
>> We could perhaps provide a special value of --temacs= switch to
>> temacs, so that the same temacs executable could be dumped into 2
>> different *.pdmp files, one with natively-compiled preloaded stuff,
>> the other with byte-compiled stuff; then users could use the existing
>> option --dump-file= to start Emacs with the non-standard pdumper file
>> (they will also need to set comp-deferred-compilation to nil to
>> prevent any run-time native-compilations once Emacs starts).

That sounds like a good solution (and maybe one which could be wrapped
up under a new option, if the --with-native-compilation build process
automatically generated both *.pdmp files, and Emacs knows what they
both are).


>> But frankly, I would hesitate to complicate Emacs even for the latter
>> possibility.  What you ask for doesn't seem to be a user-level
>> feature, it is mainly important for Emacs developers, and those can
>> always build 2 separate binaries (e.g., I already did).  Building a
>> differently-configured Emacs, even from the same Git repository, is so
>> easy that I don't really see a justification for a feature like you
>> describe.

I guess time will tell.  My feeling was that if end users encounter
native-comp bugs that are not trivial for the maintainers to reproduce
(e.g. some collection of third-party packages is involved), then it
might be super helpful to be able to ask them to test with native-comp
disabled, to confirm whether or not that is a factor.  As many users
will, in future, be running a native-comp Emacs which has been pre-
packaged for their OS, they will not easily be able to perform such a
test without such a feature.

It would definitely be a "nice to have".  However as it's evidentially
non-trivial to support this feature, I don't know whether the effort
would actually prove worthwhile.


>> (As for reproducing problems easily: it isn't hard to run the
>> interpreted or byte-compiled Lisp, if you can identify the relevant
>> Lisp files involved in the problem: just load them manually.

I did think of that, but my feeling was that it's just not the same
thing as inhibiting the native-compilation entirely.  But as an existing
alternative which would probably do the job in most cases, it's hard to
argue with.


On 27/04/21 2:10 am, Andrea Corallo wrote:
> Other than I can mention some knobs we already have that might partially
> help here:
> 
> - prevent .eln from being loaded in place of bytecode with
>   `load-no-native'.

Yes, that's good to know about.  I see now that "apropos-variable native"
is very useful (my bad for not thinking of that earlier).

That var should definitely be noted in the manual once we have some in-
built docs for this; but in the meantime it might be very helpful to
update https://akrl.sdf.org/gccemacs.html with a collection of the ways
that users can tweak/test this feature?


> - inhibit the automatic native compilation of new code with
>  `comp-deferred-compilation'.

This, OTOH, doesn't use the "native" keyword at all.

Could we rename any such variables so that everything to do with
native compilation includes the word "native"?  That's a dramatically
more specific term than "compilation", so it would seem good if it
was an easy way to find/identify the native-comp options.


-Phil

bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

[Eli Zaretskii]

> Cc: 48025@debbugs.gnu.org
> From: Phil Sainty <psainty@orcon.net.nz>
> Date: Tue, 27 Apr 2021 16:28:24 +1200
> 
> >> But frankly, I would hesitate to complicate Emacs even for the latter
> >> possibility.  What you ask for doesn't seem to be a user-level
> >> feature, it is mainly important for Emacs developers, and those can
> >> always build 2 separate binaries (e.g., I already did).  Building a
> >> differently-configured Emacs, even from the same Git repository, is so
> >> easy that I don't really see a justification for a feature like you
> >> describe.
> 
> I guess time will tell.  My feeling was that if end users encounter
> native-comp bugs that are not trivial for the maintainers to reproduce
> (e.g. some collection of third-party packages is involved), then it
> might be super helpful to be able to ask them to test with native-comp
> disabled, to confirm whether or not that is a factor.  As many users
> will, in future, be running a native-comp Emacs which has been pre-
> packaged for their OS, they will not easily be able to perform such a
> test without such a feature.

I agree that we should probably revisit the issue after we have more
experience with native-compilation.

> > - inhibit the automatic native compilation of new code with
> >  `comp-deferred-compilation'.
> 
> This, OTOH, doesn't use the "native" keyword at all.
> 
> Could we rename any such variables so that everything to do with
> native compilation includes the word "native"?

Yes, I think it's a good idea.  Perhaps also the commands in comp.el
and even some non-interactive functions?

bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: 48025@debbugs.gnu.org
>> From: Phil Sainty <psainty@orcon.net.nz>
>> Date: Tue, 27 Apr 2021 16:28:24 +1200
>> 
>> >> But frankly, I would hesitate to complicate Emacs even for the latter
>> >> possibility.  What you ask for doesn't seem to be a user-level
>> >> feature, it is mainly important for Emacs developers, and those can
>> >> always build 2 separate binaries (e.g., I already did).  Building a
>> >> differently-configured Emacs, even from the same Git repository, is so
>> >> easy that I don't really see a justification for a feature like you
>> >> describe.
>> 
>> I guess time will tell.  My feeling was that if end users encounter
>> native-comp bugs that are not trivial for the maintainers to reproduce
>> (e.g. some collection of third-party packages is involved), then it
>> might be super helpful to be able to ask them to test with native-comp
>> disabled, to confirm whether or not that is a factor.  As many users
>> will, in future, be running a native-comp Emacs which has been pre-
>> packaged for their OS, they will not easily be able to perform such a
>> test without such a feature.
>
> I agree that we should probably revisit the issue after we have more
> experience with native-compilation.
>
>> > - inhibit the automatic native compilation of new code with
>> >  `comp-deferred-compilation'.
>> 
>> This, OTOH, doesn't use the "native" keyword at all.
>> 
>> Could we rename any such variables so that everything to do with
>> native compilation includes the word "native"?
>
> Yes, I think it's a good idea.  Perhaps also the commands in comp.el
> and even some non-interactive functions?

I'll be happy to rename these functions if we come-up with a list.

  Andrea

bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Phil Sainty <psainty@orcon.net.nz>, 48025@debbugs.gnu.org
> Date: Wed, 28 Apr 2021 19:39:38 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> Could we rename any such variables so that everything to do with
> >> native compilation includes the word "native"?
> >
> > Yes, I think it's a good idea.  Perhaps also the commands in comp.el
> > and even some non-interactive functions?
> 
> I'll be happy to rename these functions if we come-up with a list.

Here's a list I came up with:

 comp-limple-mode
 comp-speed
 comp-debug
 comp-verbose
 comp-always-compile
 comp-bootstrap-deny-list
 comp-never-optimize-functions
 comp-async-jobs-number
 comp-async-cu-done-functions
 comp-async-all-done-hook
 comp-async-env-modifier-form
 comp-async-report-warnings-errors
 comp-async-query-on-exit
 comp-native-driver-options
 comp-warning-on-missing-source

bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Phil Sainty <psainty@orcon.net.nz>, 48025@debbugs.gnu.org
>> Date: Wed, 28 Apr 2021 19:39:38 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> Could we rename any such variables so that everything to do with
>> >> native compilation includes the word "native"?
>> >
>> > Yes, I think it's a good idea.  Perhaps also the commands in comp.el
>> > and even some non-interactive functions?
>> 
>> I'll be happy to rename these functions if we come-up with a list.
>
> Here's a list I came up with:
>
>  comp-limple-mode
>  comp-speed
>  comp-debug
>  comp-verbose
>  comp-always-compile
>  comp-bootstrap-deny-list
>  comp-never-optimize-functions
>  comp-async-jobs-number
>  comp-async-cu-done-functions
>  comp-async-all-done-hook
>  comp-async-env-modifier-form
>  comp-async-report-warnings-errors
>  comp-async-query-on-exit
>  comp-native-driver-options
>  comp-warning-on-missing-source

Thanks, should the renaming be comp-* to native-* ?

  Andrea

bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: psainty@orcon.net.nz, 48025@debbugs.gnu.org
> Date: Thu, 29 Apr 2021 14:33:17 +0000
> 
> >  comp-limple-mode
> >  comp-speed
> >  comp-debug
> >  comp-verbose
> >  comp-always-compile
> >  comp-bootstrap-deny-list
> >  comp-never-optimize-functions
> >  comp-async-jobs-number
> >  comp-async-cu-done-functions
> >  comp-async-all-done-hook
> >  comp-async-env-modifier-form
> >  comp-async-report-warnings-errors
> >  comp-async-query-on-exit
> >  comp-native-driver-options
> >  comp-warning-on-missing-source
> 
> Thanks, should the renaming be comp-* to native-* ?

I think they all should begin with native-comp- and
comp-native-driver-options should become native-comp-driver-options.
Also, I'd prefer renaming comp-async-jobs-number to
native-comp-number-of-async-jobs.

Thanks.

bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

[Eli Zaretskii]

> Date: Thu, 29 Apr 2021 18:05:01 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: psainty@orcon.net.nz, 48025@debbugs.gnu.org
> 
> > >  comp-limple-mode
> > >  comp-speed
> > >  comp-debug
> > >  comp-verbose
> > >  comp-always-compile
> > >  comp-bootstrap-deny-list
> > >  comp-never-optimize-functions
> > >  comp-async-jobs-number
> > >  comp-async-cu-done-functions
> > >  comp-async-all-done-hook
> > >  comp-async-env-modifier-form
> > >  comp-async-report-warnings-errors
> > >  comp-async-query-on-exit
> > >  comp-native-driver-options
> > >  comp-warning-on-missing-source
> > 
> > Thanks, should the renaming be comp-* to native-* ?
> 
> I think they all should begin with native-comp- and
> comp-native-driver-options should become native-comp-driver-options.
> Also, I'd prefer renaming comp-async-jobs-number to
> native-comp-number-of-async-jobs.

And one more variable to rename:

  comp-eln-load-path

bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Thu, 29 Apr 2021 18:05:01 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: psainty@orcon.net.nz, 48025@debbugs.gnu.org
>> 
>> > >  comp-limple-mode
>> > >  comp-speed
>> > >  comp-debug
>> > >  comp-verbose
>> > >  comp-always-compile
>> > >  comp-bootstrap-deny-list
>> > >  comp-never-optimize-functions
>> > >  comp-async-jobs-number
>> > >  comp-async-cu-done-functions
>> > >  comp-async-all-done-hook
>> > >  comp-async-env-modifier-form
>> > >  comp-async-report-warnings-errors
>> > >  comp-async-query-on-exit
>> > >  comp-native-driver-options
>> > >  comp-warning-on-missing-source
>> > 
>> > Thanks, should the renaming be comp-* to native-* ?
>> 
>> I think they all should begin with native-comp- and
>> comp-native-driver-options should become native-comp-driver-options.
>> Also, I'd prefer renaming comp-async-jobs-number to
>> native-comp-number-of-async-jobs.
>
> And one more variable to rename:
>
>   comp-eln-load-path

Yeah was going to suggest it :)

Okay I'd just wait a bit for some other opinion/suggestion then I'll
take care of this.

Thanks

  Andrea

bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: psainty@orcon.net.nz, 48025@debbugs.gnu.org
> Date: Thu, 29 Apr 2021 15:22:28 +0000
> 
> Okay I'd just wait a bit for some other opinion/suggestion then I'll
> take care of this.

Sure, there's no rush.

bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

[Andrea Corallo via Bug reports for GNU Emacs, the Swiss army knife of text editors]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: psainty@orcon.net.nz, 48025@debbugs.gnu.org
>> Date: Thu, 29 Apr 2021 15:22:28 +0000
>> 
>> Okay I'd just wait a bit for some other opinion/suggestion then I'll
>> take care of this.
>
> Sure, there's no rush.

Should be done as of fbbcbed10e, hopefully I've done it with no errors
(works for me here).

  Andrea

bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: psainty@orcon.net.nz, 48025@debbugs.gnu.org
> Date: Thu, 06 May 2021 15:19:47 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> Should be done as of fbbcbed10e, hopefully I've done it with no errors
> (works for me here).

Thanks!

bug#48025: 28.0.50; Add an invocation option to inhibit native-compilation functionality

[Lars Ingebrigtsen]

Phil Sainty <psainty@orcon.net.nz> writes:

> Now that the native-compilation feature is merged, it would be very
> useful to be able to build Emacs --with-native-compilation but be
> able to choose to inhibit that functionality at start time via a
> command-line option such as 'emacs --no-native-compilation', which
> would cause Emacs to load/execute only .el and .elc files.

(I'm going through old bug reports that unfortunately weren't resolved
at the time.)

Skimming this bug report, it seems like the conclusion here was that
this would be very difficult to implement, and that the effect (allowing
users to compare) wasn't compelling enough, so I'm closing this bug
report.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no