Re: Native compilation on as default?

Re: Native compilation on as default?

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
>> Date: Wed, 25 Oct 2023 16:50:17 -0400
>> 
>> So I think we could have a new mode, still controlled by
>> native-comp-async-report-warnings-errors, that filters out all the
>> uninteresting warnings (that the programmer already got during byte
>> compilation) but still report this important one.
>> 
>> So even if the package developer doesn't use native compilation it can
>> get the bug report for the issue.
>> 
>> I suspect this might be a good compromise/solution.
>> 
>> WDYT?
>
> Sounds like a plan to me, thanks.

Back on this,

Okay I've implemented the functionality with 8e5baaddec2.

I was in doubt of adding a new value to
'native-comp-async-report-warnings-errors' or add a new knob, but what
we report seemed to me orthogonal to how we do it, so I went for the
second option introducing
'native-comp-async-report-warnings-errors-kind'.

I we decide it's not optimal I don't feel strong about it and I'm happy
to change the approach.

Anyway with this change as per default we report only important warnings
and (all) errors, and we do it presenting the *Warnings* buffer to the
user (which I believe this is the behavior we wanted).

Indeed if something can be improved in the nomenclature or the doc feel
free to do it or come with suggestions.

Thanks!

  Andrea

Re: Native compilation on as default?

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: Andrea Corallo <acorallo@gnu.org>,  stefankangas@gmail.com,
>   emacs-devel@gnu.org
> Date: Thu, 29 Feb 2024 05:56:31 -0500
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > Sounds like a plan to me, thanks.
> 
> Back on this,
> 
> Okay I've implemented the functionality with 8e5baaddec2.

Thanks, I renamed the variable, to make it a bit less of mouthful, and
also added a NEWS entry.

> Anyway with this change as per default we report only important warnings
> and (all) errors, and we do it presenting the *Warnings* buffer to the
> user (which I believe this is the behavior we wanted).

Thanks.  Let's see if people are happy with these two levels, or we
need more.

> Indeed if something can be improved in the nomenclature or the doc feel
> free to do it or come with suggestions.

Done.

Re: Native compilation on as default?

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <acorallo@gnu.org>
>> Cc: Andrea Corallo <acorallo@gnu.org>,  stefankangas@gmail.com,
>>   emacs-devel@gnu.org
>> Date: Thu, 29 Feb 2024 05:56:31 -0500
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> > Sounds like a plan to me, thanks.
>> 
>> Back on this,
>> 
>> Okay I've implemented the functionality with 8e5baaddec2.
>
> Thanks, I renamed the variable, to make it a bit less of mouthful, and
> also added a NEWS entry.
>
>> Anyway with this change as per default we report only important warnings
>> and (all) errors, and we do it presenting the *Warnings* buffer to the
>> user (which I believe this is the behavior we wanted).
>
> Thanks.  Let's see if people are happy with these two levels, or we
> need more.
>
>> Indeed if something can be improved in the nomenclature or the doc feel
>> free to do it or come with suggestions.
>
> Done.

Wonderful thanks.

I'm wondering if we could use the NEWS entry (or something else) to
suggest all the users who set in the past
'native-comp-async-report-warnings-errors' to nil or silent to give it a
try with the new default configuration.  What we report now should be
most likely cured or reported.

Regards

  Andrea

Re: Native compilation on as default?

[Eli Zaretskii]

> From: Andrea Corallo <acorallo@gnu.org>
> Cc: stefankangas@gmail.com,  emacs-devel@gnu.org
> Date: Thu, 29 Feb 2024 10:07:00 -0500
> 
> I'm wondering if we could use the NEWS entry (or something else) to
> suggest all the users who set in the past
> 'native-comp-async-report-warnings-errors' to nil or silent to give it a
> try with the new default configuration.

Done, thanks.

Re: Should native compilation be enabled by default?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Should native compilation the default -- ever?  Is that the right
direction to go?  That is a bigger question than how we might schedule
the work.  It calls for thought.

Enabling native compilation has disadvantages as well as advantages.

Is it wise for the choice be controlled by whether libgccjit is
installed?  I don't think so.  If your machine has libgccjit installed
because you installed some unrelated program which uses libgccjit,
is it right for that to control this decision about building Emacs?
I don't think so.

The possible answers for the question ae not just "yes" and "no".
Intermediate anwers are possible too.

For instance, one choice could be "sometimes".  That would build the
capacity to do native compilation, but would actually do this by
default only on certain files.  We would do that for files whose users
tend to want them to be faster.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Should native compilation be enabled by default?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, acorallo@gnu.org, stefankangas@gmail.com,
> 	emacs-devel@gnu.org
> Date: Fri, 01 Mar 2024 22:08:58 -0500
> 
> Should native compilation the default -- ever?  Is that the right
> direction to go?  That is a bigger question than how we might schedule
> the work.  It calls for thought.
> 
> Enabling native compilation has disadvantages as well as advantages.

What are the disadvantages?  I don't see any significant ones, which
is why we decided to turn on native compilation by default in Emacs 30.

Individual users might have their reasons for turning it off, which is
why we have a configure-time option for that.  For example, my main
development tree for the master branch is built without native
compilation, because I favor speed of build in that case, which is
important for my development work.  So in that tree I disabled native
compilation (I have another tree, which i build more rarely, where I
didn't disable it.)  This kind of considerations are possible for
whatever reasons, so we have an easy way of disabling the feature.
But in general, I see no reason to think there are disadvantages that
affect most of the users, or even a large portion of them.

Re: Should native compilation be enabled by default?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > Enabling native compilation has disadvantages as well as advantages.

  > What are the disadvantages?

If Emacs compilation by default causes native compilation, it will be
slower.

It will also be more fragile.  Byte-compilation is a self-contained
Emacs feature, and aside from occasional bugs that affect specific
code, it never breaks.  On general principles we can see that native
compilation is likely to go wrong because something has disappeared,
or because of bugs in other programs that you wouldnt otherwise ever
use.

Native compilation is useful mainly for power users who want
to run Lisp programs that normally are too slow.  There is no sense
directing most users into doing things gthe complex way instead
of the simple way.

The people who participate in emacs-devel tend to be power users.
What they think is not a good guide to what is useful for most users.
To find that out, we should poll the users.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Should native compilation be enabled by default?

[T.V Raman]

native comp also fails on older and more complex lisp packages one
example is the vm mail reader -- I've been using VM since 1990  and it
remains the biggest reason I've disabled native-comp.
-- 

Re: Should native compilation be enabled by default?

[Andrea Corallo]

"T.V Raman" <raman@google.com> writes:

> native comp also fails on older and more complex lisp packages one
> example is the vm mail reader -- I've been using VM since 1990  and it
> remains the biggest reason I've disabled native-comp.

Please file a bug.

  Andrea

Re: Should native compilation be enabled by default?

[Andrea Corallo]

Richard Stallman <rms@gnu.org> writes:

> [[[ To any NSA and FBI agents reading my email: please consider    ]]]
> [[[ whether defending the US Constitution against all enemies,     ]]]
> [[[ foreign or domestic, requires you to follow Snowden's example. ]]]
>
>   > > Enabling native compilation has disadvantages as well as advantages.
>
>   > What are the disadvantages?
>
> If Emacs compilation by default causes native compilation, it will be
> slower.
>
> It will also be more fragile.  Byte-compilation is a self-contained
> Emacs feature, and aside from occasional bugs that affect specific
> code, it never breaks.  On general principles we can see that native
> compilation is likely to go wrong because something has disappeared,
> or because of bugs in other programs that you wouldnt otherwise ever
> use.
>
> Native compilation is useful mainly for power users who want
> to run Lisp programs that normally are too slow.  There is no sense
> directing most users into doing things gthe complex way instead
> of the simple way.

I don't see why native compilation should be more complex for users,
AFAIK is transparent.

Anyway as a data point: AFAIK major distros decided since a while to
ship with native compilation as (I assume) they deemed it beneficial for
their users.  I believe they even did it *before* native compilation
went on by default (on system with libgccjit installed) in our
configure.

  Andrea

Re: Should native compilation be enabled by default?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Mon, 04 Mar 2024 22:42:32 -0500
> 
>   > > Enabling native compilation has disadvantages as well as advantages.
> 
>   > What are the disadvantages?
> 
> If Emacs compilation by default causes native compilation, it will be
> slower.

This is true, but the slowdown is IME insignificant on a reasonably
modern system.  And if Emacs is installed from a distro (which AFAIU
is what most users do nowadays), the slowdown doesn't exist at all,
because the *.eln files come prebuilt with the binaries.

> It will also be more fragile.  Byte-compilation is a self-contained
> Emacs feature, and aside from occasional bugs that affect specific
> code, it never breaks.  On general principles we can see that native
> compilation is likely to go wrong because something has disappeared,
> or because of bugs in other programs that you wouldnt otherwise ever
> use.

That was a valid consideration when native compilation was first added
to Emacs 28.  Which is why we kept it off by default for 2 major
releases.  Now we have enough experience to tell that the problems,
such as they were, are all but solved, and native compilation is
nowadays no more fragile than byte compilation, what with all the
improvements and optimizations introduced lately into the byte
compiler.

> Native compilation is useful mainly for power users who want
> to run Lisp programs that normally are too slow.

I disagree, based on my experience.  I see a significant speedup when
using Rmail for just reading email, something that I cannot
characterize as "power use" of Emacs.  Even restarting a large Emacs
session by restoring from a desktop file is noticeably faster.  I
don't think having a lot of buffers in a long-running session
qualifies me as a "power user" just because of this factor.

And there are other significant speed advantages.

> The people who participate in emacs-devel tend to be power users.
> What they think is not a good guide to what is useful for most users.
> To find that out, we should poll the users.

Which is why we didn't decide on the default before collecting a lot
of experience and processing the bug reports we had about the related
issues.  Having solved them, I think we can be quite sure users aren't
unhappy with native compilation, quite the contrary.

Re: Should native compilation be enabled by default?

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
> Date: Mon, 04 Mar 2024 20:51:23 -0800
> 
> native comp also fails on older and more complex lisp packages one
> example is the vm mail reader -- I've been using VM since 1990  and it
> remains the biggest reason I've disabled native-comp.

We have very easy ways of disabling native compilation of specific
Lisp files, no need to disable it globally.

Re: Should native compilation be enabled by default?

[T.V Raman]

this was discussed more than 2 years ago and closed -- the issue is
the use of macros in older packages and native-comp compiling files in
an order different from byte-comp.


Andrea Corallo writes:
 > "T.V Raman" <raman@google.com> writes:
 > 
 > > native comp also fails on older and more complex lisp packages one
 > > example is the vm mail reader -- I've been using VM since 1990  and it
 > > remains the biggest reason I've disabled native-comp.
 > 
 > Please file a bug.
 > 
 >   Andrea

-- 

Re: Should native compilation be enabled by default?

[T.V Raman]

but users then have to edit files from packages that they didn't
write, works for a power user, but not for a typical user; unless we
turn typical user into a power user, or restrict ourselves to just
power users. I'm not going to personally stand in the way of native
comp becoming the default because when it does I still know how to
disable it as I build my own Emacs.


Eli Zaretskii writes:
 > > From: "T.V Raman" <raman@google.com>
 > > Cc: Eli Zaretskii <eliz@gnu.org>,  emacs-devel@gnu.org
 > > Date: Mon, 04 Mar 2024 20:51:23 -0800
 > > 
 > > native comp also fails on older and more complex lisp packages one
 > > example is the vm mail reader -- I've been using VM since 1990  and it
 > > remains the biggest reason I've disabled native-comp.
 > 
 > We have very easy ways of disabling native compilation of specific
 > Lisp files, no need to disable it globally.

-- 

Re: Should native compilation be enabled by default?

[Andrea Corallo]

"T.V Raman" <raman@google.com> writes:

> this was discussed more than 2 years ago and closed -- the issue is
> the use of macros in older packages and native-comp compiling files in
> an order different from byte-comp.

AFAIK we compile correctly correct code.  If your code is missing some
require and is relying on some specific compilation order using the byte
compiler such that some library it's by chance loaded when some
compilation unit is compiled, well it's just buggy code.

If this is unmaintanied code and no-one including you want to fix it,
well just add it to `native-comp-jit-compilation-deny-list'.

Thanks

  Andrea

Re: Should native compilation be enabled by default?

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Date: Tue, 5 Mar 2024 07:04:07 -0800
> Cc: raman@google.com,
>     rms@gnu.org,
>     emacs-devel@gnu.org
> 
> but users then have to edit files from packages that they didn't
> write

That's one method of disabling native compilation.  If that is not
appropriate in some cases, we also have
native-comp-jit-compilation-deny-list to cover those.

Re: Should native compilation be enabled by default?

[Suhail Singh]

"T.V Raman" <raman@google.com> writes:

> this was discussed more than 2 years ago and closed

The bug in question, I believe, is [48743].  Since it took me some
(small, but not zero) effort to look it up, I'm sharing here for common
awareness.

[48743]: <https://issues.guix.gnu.org/48743>

-- 
Suhail

Re: Should native compilation be enabled by default?

[T.V Raman]

it's not my code, which is the issue. VM is a third party package and
no longer maintained, but I happen to use it as my primary mail reder

It uses  macros where it should likely have used defsubst.

From what I remember and this might ell have changed:

1. It's build el->elc relied on specific ordering of byte compilation.
2. Native comp from .elc->.eln -- rather than .el->.eln
3. 1 2 and the different order of compilation resulted in broken eln
   files that threw errors at random when using VM

   
   

Andrea Corallo writes:
 > "T.V Raman" <raman@google.com> writes:
 > 
 > > this was discussed more than 2 years ago and closed -- the issue is
 > > the use of macros in older packages and native-comp compiling files in
 > > an order different from byte-comp.
 > 
 > AFAIK we compile correctly correct code.  If your code is missing some
 > require and is relying on some specific compilation order using the byte
 > compiler such that some library it's by chance loaded when some
 > compilation unit is compiled, well it's just buggy code.
 > 
 > If this is unmaintanied code and no-one including you want to fix it,
 > well just add it to `native-comp-jit-compilation-deny-list'.
 > 
 > Thanks
 > 
 >   Andrea

-- 

Re: Should native compilation be enabled by default?

[T.V Raman]

First I am hearing of that ar, glad it is there.

So for the average user, who adds things to that list? Some examples
in the docs -- especially illustrating  how to use it to "fix broken
packages" might improve things.


Eli Zaretskii writes:
 > > From: "T.V Raman" <raman@google.com>
 > > Date: Tue, 5 Mar 2024 07:04:07 -0800
 > > Cc: raman@google.com,
 > >     rms@gnu.org,
 > >     emacs-devel@gnu.org
 > > 
 > > but users then have to edit files from packages that they didn't
 > > write
 > 
 > That's one method of disabling native compilation.  If that is not
 > appropriate in some cases, we also have
 > native-comp-jit-compilation-deny-list to cover those.

-- 

Re: Should native compilation be enabled by default?

[Andrea Corallo]

"T.V Raman" <raman@google.com> writes:

> but users then have to edit files from packages that they didn't
> write, works for a power user, but not for a typical user;

IME typical users don't use unmaintained buggy packages, quite the
opposite.

Re: Should native compilation be enabled by default?

[Eli Zaretskii]

> From: "T.V Raman" <raman@google.com>
> Date: Tue, 5 Mar 2024 08:03:31 -0800
> Cc: raman@google.com,
>     rms@gnu.org,
>     emacs-devel@gnu.org
> 
> 
> First I am hearing of that ar, glad it is there.

So now you could try using that instead of disabling native
compilation globally.

> So for the average user, who adds things to that list? Some examples
> in the docs -- especially illustrating  how to use it to "fix broken
> packages" might improve things.

What do you mean?  That variable is a user option, documented as
follows:

    "List of regexps to exclude matching files from deferred native compilation.
  Files whose names match any regexp are excluded from native compilation."

So it is a list of regular expressions intended for _users_ to
customize; the default is nil.  The regular expressions should match
the file names which you don't want Emacs to natively compile.  I
think this is simple and clear enough to not require any elaborate
examples.  The reasons why the user would like to avoid compiling some
files are beside the point -- it could be because the files are broken
and unmaintained, but it could also be for any number of other
reasons.

Re: Should native compilation be enabled by default?

[T.V Raman]

Thank you for tracking that down, I hope it helps moving the ball
forward.

-- 

Re: Should native compilation be enabled by default?

[T.V Raman]

Andrea Corallo <acorallo@gnu.org> writes:

Glad you think so:-)> "T.V Raman" <raman@google.com> writes:
>
>> but users then have to edit files from packages that they didn't
>> write, works for a power user, but not for a typical user;
>
> IME typical users don't use unmaintained buggy packages, quite the
> opposite.

-- 

Re: Should native compilation be enabled by default?

[Andrea Corallo]

"T.V Raman" <raman@google.com> writes:

> it's not my code, which is the issue. VM is a third party package and
> no longer maintained, but I happen to use it as my primary mail reder
>
> It uses  macros where it should likely have used defsubst.
>
>>From what I remember and this might ell have changed:
>
> 1. It's build el->elc relied on specific ordering of byte compilation.
> 2. Native comp from .elc->.eln -- rather than .el->.eln
> 3. 1 2 and the different order of compilation resulted in broken eln
>    files that threw errors at random when using VM

Yes, it's a missing require that is unidentified by the byte-compiler.
We are discussing this since like three years, I believe should be
sufficient.

As suggested if nobody wants to fix it just adding the package to
'native-comp-jit-compilation-deny-list' will work around the bug.

  Andrea

Re: Should native compilation be enabled by default?

[Björn Bidar]

Richard Stallman <rms@gnu.org> writes:

>   > > Enabling native compilation has disadvantages as well as advantages.
>
>   > What are the disadvantages?
>
> If Emacs compilation by default causes native compilation, it will be
> slower.
>
At first a little but once the compilation for a file is done Emacs will
be faster.
> It will also be more fragile.  Byte-compilation is a self-contained
> Emacs feature, and aside from occasional bugs that affect specific
> code, it never breaks.  On general principles we can see that native
> compilation is likely to go wrong because something has disappeared,
> or because of bugs in other programs that you wouldnt otherwise ever
> use.
>

Native compilation adds libgccjit as a dependency but from what I saw
the not self contained or self contained nature of native-compilation
was never an issue.

> Native compilation is useful mainly for power users who want
> to run Lisp programs that normally are too slow.  There is no sense
> directing most users into doing things gthe complex way instead
> of the simple way.

I don't "power user" is the right word here. Most users that actively
choose native compilation had some kind of use case that was to slow
e.g. programming with lsp that made use Emacs with native compilation
but I would say most of them were experienced.

Some distributions ship native compilation by default e.g. openSUSE.

Re: Should native compilation be enabled by default?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > > If Emacs compilation by default causes native compilation, it will be
  > > slower.
  > >
  > At first a little but once the compilation for a file is done Emacs will
  > be faster.

That is true in a sense, but also misleading due to simplification.
Some operations are annoyingly slow and their slowness is due to
running loops in Lisp code.  Those could be noticiably faster with
native compilation _of the code containing those loops_.
Native-compiling some files could be worth while.

But most operations are either (1) fast enough for a user-driven
interactive program or (2) get their slowness from either running C
code or (on large files) swapping.  For them, native compilation
is not worth while.

The best trade-off could be to mark some individual source files as
worth-while for native compilation, and by default do native compilation
only for those.

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Should native compilation be enabled by default?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  >   On general principles we can see that native
  > > compilation is likely to go wrong because something has disappeared,
  > > or because of bugs in other programs that you wouldnt otherwise ever
  > > use.
  > >
  > > Native compilation is useful mainly for power users who want
  > > to run Lisp programs that normally are too slow.  There is no sense
  > > directing most users into doing things gthe complex way instead
  > > of the simple way.

  > I don't see why native compilation should be more complex for users,
  > AFAIK is transparent.

I meant complex in one specific sense.  You're talking about a
different sense, so we are talking past each other,

  > Anyway as a data point: AFAIK major distros decided since a while to
  > ship with native compilation as (I assume) they deemed it beneficial for
  > their users.

I ssppose so.  But the fact that they think so is not a convincing
argument for us.

If we poll the users, we would have a basis to conclude that those
distros' choice is wise, or that it was hasty and unwise.  It is
better to make sure of this than to follow the herd.

As I stated a few days ago, there may be a better alternative
somewhere between "yes, always" and "no, never".

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Should native compilation be enabled by default?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > This is true, but the slowdown is IME insignificant on a reasonably
  > modern system.

Could you explain what "reasonably modern" refer to OS releases, to CPUs, or what?

The only CPUs we can morally recommend to people are the ones before
the hardware backdoors became impossible to disable completely.  They
are around 15 years old.  Does installing a current GNU/Linux distro
release on one of those computers add up to a "reasonably modern system"?

  > I disagree, based on my experience.  I see a significant speedup when
  > using Rmail for just reading email, something that I cannot
  > characterize as "power use" of Emacs.

Could you tell me more about this experience?  Which operations become
faster?  How big is your Rmail file?  How many messages are in it?

  > And there are other significant speed advantages.

Would you please tell me more?


-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Should native compilation be enabled by default?

[Richard Stallman]

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

  > I disagree, based on my experience.  I see a significant speedup when
  > using Rmail for just reading email, something that I cannot
  > characterize as "power use" of Emacs.

Could you tell me more about this experience?  Which operations become
faster?  How big is your Rmail file?  How many messages are in it?

  > And there are other significant speed advantages.

Would you please tell me more operatkions that get significantly faster?

-- 
Dr Richard Stallman (https://stallman.org)
Chief GNUisance of the GNU Project (https://gnu.org)
Founder, Free Software Foundation (https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Re: Should native compilation be enabled by default?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> Date: Thu, 07 Mar 2024 21:29:29 -0500
> 
> The best trade-off could be to mark some individual source files as
> worth-while for native compilation, and by default do native compilation
> only for those.

While ideally true, in practice this would be a significant
maintenance burden.  The initial research of whether a given file can
benefit from native compilation enough to justify it is already a
large job, even just for files that are part of Emacs, because the
actual gains depend on the functions used and the use patterns, which
can and do vary widely.  Then the results of this initial research
should be somehow kept up-to-date as the Lisp code changes and also as
libgccjit and GCC in general becomes a better compiler that can
optimize better.

So instead, we take an opportunistic way forward that is much easier
maintenance-wise: we require native compilation to never produce much
worse results than byte-compiled code would.  We also require native
compilation to produce code that is on average faster than
byte-compiled code.  This way, we don't have to worry about file-local
issues unless the native code is significantly slower, in which case
we ask users to report a bug, which we can investigate and solve.

I believe this is similar to how optimizations are introduced into a
compiler.  In particular, no one suggests that each individual source
file be tagged with the optimizations that produce significant gains
for that file.  However, as exception, a given source file can be
tagged with optimizations that should or should not be done for it.
We support similar features for native compilation: the
no-native-compile cookie and the various native-comp-* variables that
can be set in file-local variables to control the aspects of native
compilation and optimizations for those exceptional files.

So I think we have a reasonably good arrangement, both on average and
where specific exceptional cases need special handling, and we do that
in a way that doesn't present any significant maintenance burden.

Re: Should native compilation be enabled by default?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: eliz@gnu.org, emacs-devel@gnu.org
> Date: Thu, 07 Mar 2024 21:29:51 -0500
> 
>   > Anyway as a data point: AFAIK major distros decided since a while to
>   > ship with native compilation as (I assume) they deemed it beneficial for
>   > their users.
> 
> I ssppose so.  But the fact that they think so is not a convincing
> argument for us.
> 
> If we poll the users, we would have a basis to conclude that those
> distros' choice is wise, or that it was hasty and unwise.  It is
> better to make sure of this than to follow the herd.

We in effect polled the users already: we have the experience of 2
major Emacs releases under our belt that provided native compilation
as an opt-in feature.  Based on that experience, we know that most
people do use native compilation, and many even compile all the Lisp
files ahead of time (instead of relying on JIT compilation when a Lisp
file is first loaded into Emacs).  And the fact that distros decided
to ship Emacs with native compilation is another indication that users
like it, and are not bothered with the few downsides, especially if
they don't have to build the *.eln files themselves.

We could decide that this evidence is not enough for us if we had some
basis for thinking there are problems lurking under the surface that
we should consider.  But we have no such evidence, so I don't see why
we should try digging deeper for signs of problems which are at this
point purely imaginary, at least from where I stand (and that includes
my own personal experience).

Re: Should native compilation be enabled by default?

[Eli Zaretskii]

> From: Richard Stallman <rms@gnu.org>
> Cc: emacs-devel@gnu.org
> Date: Thu, 07 Mar 2024 21:29:54 -0500
> 
>   > This is true, but the slowdown is IME insignificant on a reasonably
>   > modern system.
> 
> Could you explain what "reasonably modern" refer to OS releases, to CPUs, or what?

CPU and memory.

> The only CPUs we can morally recommend to people are the ones before
> the hardware backdoors became impossible to disable completely.

This is not about recommending CPU to people, this is about the level
of CPU power and memory size that most Emacs users have nowadays.
Given the popularity of native compilation, I conclude that very few
of them stick to old CPUs and small memory sizes.

> They are around 15 years old.  Does installing a current GNU/Linux distro
> release on one of those computers add up to a "reasonably modern system"?

I don't know enough about those systems to answer that question.  If
it turns out that those systems are too slow for building Emacs with
native compilation, there's the --without-native-compilation
configure-time switch, which is very easy to use, and needs only to be
specified once for every source tree.

IOW, the default configuration of the Emacs build is supposed to cater
to the average system which Emacs users have, not to the lowest
possible ones.  For example, we build by default with image libraries
if those are installed, even if the user always invokes "emacs -nw"
because the system cannot endure running an X server.  For systems
that are far from the average, we offer configure-time options to
disable features that get in the way for some reason.

>   > I disagree, based on my experience.  I see a significant speedup when
>   > using Rmail for just reading email, something that I cannot
>   > characterize as "power use" of Emacs.
> 
> Could you tell me more about this experience?  Which operations become
> faster?

Reading new messages and displaying messages, especially if the
message is in HTML (as opposed to plain text).

> How big is your Rmail file?  How many messages are in it?

Currently about 75 MB and more than 2K messages.  I purge it weekly,
and after a purge it becomes about 1/3rd that size.

>   > And there are other significant speed advantages.
> 
> Would you please tell me more?

This was all discussed during development of Emacs 28, two years ago.
I don't remember all the details, and cannot afford looking through
the archives to find the discussions with benchmarks.  Maybe someone
else (Andrea?) could look them up.  I do remember the conclusion:
natively-compiled code was on average faster by a factor of 1.5 to 2.
And that is good enough for me, and I think should be enough for
anyone, to support the decision of making this the default, as soon as
the initial issues with it (and there were issues, believe me) are
resolved.

Re: Should native compilation be enabled by default?

[Emanuel Berg]

Björn Bidar wrote:

>> Native compilation is useful mainly for power users who
>> want to run Lisp programs that normally are too slow.
>> There is no sense directing most users into doing things
>> gthe complex way instead of the simple way.
>
> I don't "power user" is the right word here. Most users that
> actively choose native compilation had some kind of use case
> that was to slow e.g. programming with lsp that made use
> Emacs with native compilation but I would say most of them
> were experienced.

It isn't true that native compilation is only for power users.
It is for everyone that cares for a snappy user experience in
their everyday Emacs use. This is the biggest benefit from
native compilation - it makes Emacs faster all the time, every
time you push a button, almost, so it is not just when the
supposed power users execute some really exotic and advanced
Elisp (not sure who does that at all or anyway, but it sounds
cool so let's include them as well).

Native compilation is very good, even for fast, modern
computers it makes the interactive feel of Emacs much faster,
so the whole Emacs experience feels much more snappy, more
responsive and more plain fun to use.

So to me, the issue is not "should we add it or not", we
absolutely should, it is rather a matter of _how_ to add it so
not to expose the user to a seemingly more complicated install
process or annoying messages.

If we get it to work transparently (invisible) as intended
from the get-go for everyone, it is a no brainer if we should
have it or not.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Should native compilation be enabled by default?

[Colin Baxter]

>>>>> Emanuel Berg <incal@dataswamp.org> writes:

    > Bjö️rn Bidar wrote:
    >>> Native compilation is useful mainly for power users who want to
    >>> run Lisp programs that normally are too slow.  There is no sense
    >>> directing most users into doing things gthe complex way instead
    >>> of the simple way.
    >> 
    >> I don't "power user" is the right word here. Most users that
    >> actively choose native compilation had some kind of use case that
    >> was to slow e.g. programming with lsp that made use Emacs with
    >> native compilation but I would say most of them were experienced.

    > It isn't true that native compilation is only for power users.  It
    > is for everyone that cares for a snappy user experience in their
    > everyday Emacs use. This is the biggest benefit from native
    > compilation - it makes Emacs faster all the time, every time you
    > push a button, almost, so it is not just when the supposed power
    > users execute some really exotic and advanced Elisp (not sure who
    > does that at all or anyway, but it sounds cool so let's include
    > them as well).

    > Native compilation is very good, even for fast, modern computers
    > it makes the interactive feel of Emacs much faster, so the whole
    > Emacs experience feels much more snappy, more responsive and more
    > plain fun to use.

    > So to me, the issue is not "should we add it or not", we
    > absolutely should, it is rather a matter of _how_ to add it so not
    > to expose the user to a seemingly more complicated install process
    > or annoying messages.

    > If we get it to work transparently (invisible) as intended from
    > the get-go for everyone, it is a no brainer if we should have it
    > or not.

    > -- underground experts united https://dataswamp.org/~incal

Compiling emacs with native compilation is not quite as straightforward
as one might think. Some people have had issues installing the
appropriate libgccjit. See for example
https://www.reddit.com/r/emacs/comments/rojo7y/emacs_native_compilation_cannot_find_libgccjit/?rdt=60542 

I know that's not strictly an emacs problem, but nevertheless it's a
problem that a (linux) user must solve or least know about if they want
native compilation.

Best wishes,

Re: Should native compilation be enabled by default?

[Colin Baxter]

>>>>> Ihor Radchenko <yantar92@posteo.net> writes:

    > Colin Baxter <m43cap@yandex.com> writes:
    >> Compiling emacs with native compilation is not quite as
    >> straightforward as one might think. Some people have had issues
    >> installing the appropriate libgccjit. See for example
    >> https://www.reddit.com/r/emacs/comments/rojo7y/emacs_native_compilation_cannot_find_libgccjit/?rdt=60542
    >> 
    >> I know that's not strictly an emacs problem, but nevertheless
    >> it's a problem that a (linux) user must solve or least know about
    >> if they want native compilation.

    > That post is two years old, from the times when GCC 10 (that has
    > libgccjit) was just released. Now, after two years, when the
    > latest GCC is GCC 13, we can reasonably expect that most Linux
    > installations have libgccjit.

Well, mine doesn't.

Re: Should native compilation be enabled by default?

[Colin Baxter]

My argument is against the notion that default native compilation is a
"no brainer". I suggest it's a "some brainer". There are issues about
what libgccjit flag to pass when compiling, as was pointed out in the
link, whether two years old or not.

Re: Should native compilation be enabled by default?

[Corwin Brust]

On Fri, Mar 15, 2024 at 8:29 AM Colin Baxter <m43cap@yandex.com> wrote:
>
>
> My argument is against the notion that default native compilation is a
> "no brainer". I suggest it's a "some brainer". There are issues about
> what libgccjit flag to pass when compiling, as was pointed out in the
> link, whether two years old or not.
>

It sounds you have a sytem without libgccjit available but also like
you do not want native comp enabled.  Is that right?

IIUC, the plan would be that, if we don't have libgccjit installed
Emacs, we will always get (according to defaults) a build build
without native comp.  Does that address your concern?

Re: Should native compilation be enabled by default?

[Colin Baxter]

>>>>> Corwin Brust <corwin@bru.st> writes:

    > On Fri, Mar 15, 2024 at 8:29 AM Colin Baxter <m43cap@yandex.com> wrote:
    >> 
    >> 
    >> My argument is against the notion that default native compilation
    >> is a "no brainer". I suggest it's a "some brainer". There are
    >> issues about what libgccjit flag to pass when compiling, as was
    >> pointed out in the link, whether two years old or not.
    >> 

    > It sounds you have a sytem without libgccjit available but also
    > like you do not want native comp enabled.  Is that right?

    > IIUC, the plan would be that, if we don't have libgccjit installed
    > Emacs, we will always get (according to defaults) a build build
    > without native comp.  Does that address your concern?

Yes.

I think I didn't explain my concerns properly. To me, "default"
means that all a user need do is ./configure, make, make install. This
would I think (or thought) have a good chance of failing if native
compilation was made default. This is partially solved if, as you say,
the build will still succeed if the user does not have libgccjit.

There remains the possibilities, however, that the user has libgccjit
but doesn't know it, or that it's version is not compatible with the
installed gcc. Perhaps some words of explanation in the INSTALL file is
what's needed?

Best wishes,

Re: Should native compilation be enabled by default?

[Eli Zaretskii]

> From: Colin Baxter <m43cap@yandex.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 15 Mar 2024 13:14:54 +0000
> 
> >>>>> Ihor Radchenko <yantar92@posteo.net> writes:
> 
>     > That post is two years old, from the times when GCC 10 (that has
>     > libgccjit) was just released. Now, after two years, when the
>     > latest GCC is GCC 13, we can reasonably expect that most Linux
>     > installations have libgccjit.
> 
> Well, mine doesn't.

If your system doesn't have libgccjit installed, the configure script
will automatically produce a build without native compilation.

Re: Should native compilation be enabled by default?

[Emanuel Berg]

Colin Baxter wrote:

> There remains the possibilities, however, that the user has
> libgccjit but doesn't know it, or that it's version is not
> compatible with the installed gcc. Perhaps some words of
> explanation in the INSTALL file is what's needed?

Perhaps, but it should also be part of the installation
process to find out if all the requirements are present.

And tell the user if this isn't so.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Should native compilation be enabled by default?

[Eli Zaretskii]

> From: Colin Baxter <m43cap@yandex.com>
> Cc: emacs-devel@gnu.org
> Date: Fri, 15 Mar 2024 13:29:20 +0000
> 
> 
> My argument is against the notion that default native compilation is a
> "no brainer". I suggest it's a "some brainer". There are issues about
> what libgccjit flag to pass when compiling, as was pointed out in the
> link, whether two years old or not.

What do you mean by "what libgccjit flag to pass when compiling"?  I
didn't find anything in that old post which matches that description.

Re: Should native compilation be enabled by default?

[Emanuel Berg]

Colin Baxter wrote:

> My argument is against the notion that default native
> compilation is a "no brainer". I suggest it's a "some
> brainer". There are issues about what libgccjit flag to pass
> when compiling, as was pointed out in the link, whether two
> years old or not.

The installation and compilation process must always be made
as simple and unobtrusive as possible for the user, avoiding
the overdose of messages.

That is the one concern I can think of with native
compilation.

I configure like this,

  --with-native-compilation=aot

then installation takes a little longer but you don't get all
the messages later on. Since everything is are to be compiled
anyway, I don't see why that has to be done little by little.
Better to just do it once and be done with it IMO.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Should native compilation be enabled by default?

[Ihor Radchenko]

Colin Baxter <m43cap@yandex.com> writes:

> Compiling emacs with native compilation is not quite as straightforward
> as one might think. Some people have had issues installing the
> appropriate libgccjit. See for example
> https://www.reddit.com/r/emacs/comments/rojo7y/emacs_native_compilation_cannot_find_libgccjit/?rdt=60542 
>
> I know that's not strictly an emacs problem, but nevertheless it's a
> problem that a (linux) user must solve or least know about if they want
> native compilation.

That post is two years old, from the times when GCC 10 (that has
libgccjit) was just released. Now, after two years, when the latest GCC
is GCC 13, we can reasonably expect that most Linux installations have
libgccjit.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>

Re: Should native compilation be enabled by default?

[Andrea Corallo]

Emanuel Berg <incal@dataswamp.org> writes:

> Colin Baxter wrote:
>
>> My argument is against the notion that default native
>> compilation is a "no brainer". I suggest it's a "some
>> brainer". There are issues about what libgccjit flag to pass
>> when compiling, as was pointed out in the link, whether two
>> years old or not.
>
> The installation and compilation process must always be made
> as simple and unobtrusive as possible for the user, avoiding
> the overdose of messages.
>
> That is the one concern I can think of with native
> compilation.
>
> I configure like this,
>
>   --with-native-compilation=aot
>
> then installation takes a little longer but you don't get all
> the messages later on.

One gets messages only if a file being compiled has issues, BTW since
8e5baaddec2 we are by default way more selctive on what we report during
native compilation so I don't think this problem still exists.

> Since everything is are to be compiled anyway, I don't see why that
> has to be done little by little.  Better to just do it once and be
> done with it IMO.

Not everything needs to be compiled unless one needs to load everything.

  Andrea

Re: Should native compilation be enabled by default?

[Colin Baxter]

>>>>> Eli Zaretskii <eliz@gnu.org> writes:

    >> From: Colin Baxter <m43cap@yandex.com> Cc: emacs-devel@gnu.org
    >> Date: Fri, 15 Mar 2024 13:29:20 +0000
    >> 
    >> 
    >> My argument is against the notion that default native compilation
    >> is a "no brainer". I suggest it's a "some brainer". There are
    >> issues about what libgccjit flag to pass when compiling, as was
    >> pointed out in the link, whether two years old or not.

    > What do you mean by "what libgccjit flag to pass when compiling"?
    > I didn't find anything in that old post which matches that
    > description.


My understanding, reading the reddit link, is that the version XX of
libgccjit should match that of gcc if the former is to be found. In the
event that it isn't CC="gcc-XX" should be exported before ./configure. I
stand corrected if that isn't the case.

Re: Should native compilation be enabled by default?

[Emanuel Berg]

Andrea Corallo wrote:

> One gets messages only if a file being compiled has issues,
> BTW since 8e5baaddec2 we are by default way more selctive on
> what we report during native compilation so I don't think
> this problem still exists.

If people intend to fix those issues it is okay to report
them, and ideally everyone that writes Elisp should make sure
their code don't produce any warnings anywhere.

Me, personally I enjoy making improvements to my source and
I never care or stop to think if the improvements are big or
small. But seeing the same warnings over and over in other
people's source makes me think maybe they don't respect them
and for this reason don't fix them.

So we should examine what warnings people actually considers
important enough to fix.

Real issues obviously should always be reported.

>> Since everything is are to be compiled anyway, I don't see
>> why that has to be done little by little. Better to just do
>> it once and be done with it IMO.
>
> Not everything needs to be compiled unless one needs to
> load everything.

Yes, but everything loaded needs to be compiled and compiling
everything implies whatever is called for is ready. I don't
think searching the cache will ever be a factor so the only
drawback is installation time. But I don't install Emacs that
often and the process is automatic.

BTW, now this gets a bit confusing as this option,

  --with-native-compilation=aot

doesn't necessarily imply "everything" is compiled.
In practice, I interpreted it as a step in that direction -
except for ELPA packages and personal files, maybe? But I have
to look up the exact meaning.

But what I meant was it is compiled during the installation
process, and much less so during Emacs use. So you don't get
the warnings and popup buffers.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Should native compilation be enabled by default?

[Andrea Corallo]

Colin Baxter <m43cap@yandex.com> writes:

>>>>>> Eli Zaretskii <eliz@gnu.org> writes:
>
>     >> From: Colin Baxter <m43cap@yandex.com> Cc: emacs-devel@gnu.org
>     >> Date: Fri, 15 Mar 2024 13:29:20 +0000
>     >> 
>     >> 
>     >> My argument is against the notion that default native compilation
>     >> is a "no brainer". I suggest it's a "some brainer". There are
>     >> issues about what libgccjit flag to pass when compiling, as was
>     >> pointed out in the link, whether two years old or not.
>
>     > What do you mean by "what libgccjit flag to pass when compiling"?
>     > I didn't find anything in that old post which matches that
>     > description.
>
>
> My understanding, reading the reddit link, is that the version XX of
> libgccjit should match that of gcc if the former is to be found. In the
> event that it isn't CC="gcc-XX" should be exported before ./configure. I
> stand corrected if that isn't the case.

If a distro installs a non functional libgccjit that should be a bug
fixed in the distro, I don't think we should work around it changing the
system compiler.

  Andrea

Re: Should native compilation be enabled by default?

[Emanuel Berg]

Andrea Corallo wrote:

>> My understanding, reading the reddit link, is that the
>> version XX of libgccjit should match that of gcc if the
>> former is to be found. In the event that it isn't
>> CC="gcc-XX" should be exported before ./configure. I stand
>> corrected if that isn't the case.
>
> If a distro installs a non functional libgccjit that should
> be a bug fixed in the distro, I don't think we should work
> around it changing the system compiler.

It is a no brainer that all Emacs users should benefit from
native compilation, it is a some brainer how to make this
happen in practice without friction.

-- 
underground experts united
https://dataswamp.org/~incal

Re: Should native compilation be enabled by default?

[Eli Zaretskii]

> From: Emanuel Berg <incal@dataswamp.org>
> Date: Fri, 15 Mar 2024 21:35:59 +0100
> 
> Andrea Corallo wrote:
> 
> >> My understanding, reading the reddit link, is that the
> >> version XX of libgccjit should match that of gcc if the
> >> former is to be found. In the event that it isn't
> >> CC="gcc-XX" should be exported before ./configure. I stand
> >> corrected if that isn't the case.
> >
> > If a distro installs a non functional libgccjit that should
> > be a bug fixed in the distro, I don't think we should work
> > around it changing the system compiler.
> 
> It is a no brainer that all Emacs users should benefit from
> native compilation, it is a some brainer how to make this
> happen in practice without friction.

After 2 major releases (which did uncover some friction, now solved),
we are way past that.