Getting ready to land native-compilation on master

Getting ready to land native-compilation on master

[Eli Zaretskii]

AFAICT, we are quite ready to land this important feature.  The branch
was tested on several systems and is in good shape: many issues and
bugs were fixed, and currently no known issues remain that block the
merge.  (There's a lot yet to do wrt documenting the feature and its
various aspects, but that can be done on master after merging.)

If some of you have some unusual or rare or exotic system or Emacs
configuration, and did not yet try building the native-comp branch,
this is your chance to have a go before the merge: please build the
branch and report any issues or problems you bump into.

If no significant issues pop up within a week, I will ask Andrea to
merge the branch onto master the next weekend (i.e. around 17th of
April).

Last, but not least: I'd like to take this opportunity to thank Andrea
for his hard work and perseverance during this last year.  We would
not be where we are now with this feature without his devotion and
determination to see this through.  Thanks a lot!

Re: Getting ready to land native-compilation on master

[tomas]

[-- Attachment #1: Type: text/plain, Size: 302 bytes --]

On Fri, Apr 09, 2021 at 05:02:15PM +0300, Eli Zaretskii wrote:
> AFAICT, we are quite ready to land this important feature [...]

\o/

> Last, but not least: I'd like to take this opportunity to thank Andrea
> for his hard work [...]

Seconded, and thirded and...

Really impressive work.

Cheers
 - t

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: Getting ready to land native-compilation on master

[Stefan Kangas]

Eli Zaretskii <eliz@gnu.org> writes:

> If no significant issues pop up within a week, I will ask Andrea to
> merge the branch onto master the next weekend (i.e. around 17th of
> April).

Fantastic news!  Thank you so much Andrea for all the hard work to make
this happen, and congratulations for finally getting it merged to
master.

Thanks also to Eli for reviewing the branch and moving the merge
forward.

Re: Getting ready to land native-compilation on master

[T.V Raman]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=gb18030, Size: 1494 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:


1+ on all counts. About a couple of months ago (perhaps  longer) older
packages that used defmacro -- specifically the VM mail reader broke in
a couple of places, I remember sending Andrea email. Do we know if those
issues were fixed (Andrea would know)

And yes, this is really nice work and would be great to see it land.
> AFAICT, we are quite ready to land this important feature.  The branch
> was tested on several systems and is in good shape: many issues and
> bugs were fixed, and currently no known issues remain that block the
> merge.  (There's a lot yet to do wrt documenting the feature and its
> various aspects, but that can be done on master after merging.)
>
> If some of you have some unusual or rare or exotic system or Emacs
> configuration, and did not yet try building the native-comp branch,
> this is your chance to have a go before the merge: please build the
> branch and report any issues or problems you bump into.
>
> If no significant issues pop up within a week, I will ask Andrea to
> merge the branch onto master the next weekend (i.e. around 17th of
> April).
>
> Last, but not least: I'd like to take this opportunity to thank Andrea
> for his hard work and perseverance during this last year.  We would
> not be where we are now with this feature without his devotion and
> determination to see this through.  Thanks a lot!
>

-- 

Thanks,

--Raman
7©4 Id: kg:/m/0285kf1  •0Ü8

Re: Getting ready to land native-compilation on master

[Pip Cet]

On Fri, Apr 9, 2021 at 2:04 PM Eli Zaretskii <eliz@gnu.org> wrote:
> AFAICT, we are quite ready to land this important feature.  The branch
> was tested on several systems and is in good shape: many issues and
> bugs were fixed, and currently no known issues remain that block the
> merge.  (There's a lot yet to do wrt documenting the feature and its
> various aspects, but that can be done on master after merging.)

There are still (theoretical, at least) miscompilation bugs, last I
checked. I don't think these should block the merge and there's plenty
of time to fix them on the master branch.

I must say I'm unhappy with many of the changes that are introduced
outside of comp.c/comp.el (asynchronous compilation as a
default/forced-on-users feature, the .eln handling, the way
natively-compiled functions are subrs even though they have virtually
nothing in common with them...). But, again, that's also something
that can be discussed more reasonably once the branch has been merged,
and there's perhaps less of an attitude of owning the branch.

I also think that it would be reasonable to merge only the "essential"
native-compilation features at first, and leaving out things like the
half-written SSA optimizer that's currently on the branch (or was last
I checked). But my understanding is Andrea is unwilling to consider
this option, and it's much better to have a master branch that Andrea
continues to work on.

> If no significant issues pop up within a week, I will ask Andrea to
> merge the branch onto master the next weekend (i.e. around 17th of
> April).

Hooray!

> Last, but not least: I'd like to take this opportunity to thank Andrea
> for his hard work and perseverance during this last year.  We would
> not be where we are now with this feature without his devotion and
> determination to see this through.  Thanks a lot!

Thank you, Andrea!

Pip

Re: Getting ready to land native-compilation on master

[Thierry Volpiatto]

[-- Attachment #1: Type: text/plain, Size: 1214 bytes --]


Eli Zaretskii <eliz@gnu.org> writes:

> AFAICT, we are quite ready to land this important feature.  The branch
> was tested on several systems and is in good shape: many issues and
> bugs were fixed, and currently no known issues remain that block the
> merge.  (There's a lot yet to do wrt documenting the feature and its
> various aspects, but that can be done on master after merging.)
>
> If some of you have some unusual or rare or exotic system or Emacs
> configuration, and did not yet try building the native-comp branch,
> this is your chance to have a go before the merge: please build the
> branch and report any issues or problems you bump into.

I didn't have any answer about bug#46790, many things are broken for me
without fixing this...

> If no significant issues pop up within a week, I will ask Andrea to
> merge the branch onto master the next weekend (i.e. around 17th of
> April).
>
> Last, but not least: I'd like to take this opportunity to thank Andrea
> for his hard work and perseverance during this last year.  We would
> not be where we are now with this feature without his devotion and
> determination to see this through.  Thanks a lot!


-- 
Thierry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 686 bytes --]

Re: Getting ready to land native-compilation on master

[Jens C. Jensen]

> From: Eli Zaretskii <eliz@gnu.org>
> Date: Fri, 09 Apr 2021 17:02:15 +0300
> Cc: Andrea Corallo <akrl@sdf.org>

> AFAICT, we are quite ready to land this important feature.

This is great news!

I've been on the branch for several months now with no major issues, the only oddity I've found was that natively-compiled elisp are subrs (as Pip already mentioned), which messed up my syntax highlighting.

A big Thank-You to Andrea for the great work getting this far!

P.S. Is Andreas website on gccemacs[1] the proper place to get in-depth information about the implementation? It feels more like a change-log.


[1] https://akrl.sdf.org/gccemacs.html

Re: Getting ready to land native-compilation on master

[Sujith Manoharan]

Eli Zaretskii <eliz@gnu.org> writes:
> If some of you have some unusual or rare or exotic system or Emacs
> configuration, and did not yet try building the native-comp branch,
> this is your chance to have a go before the merge: please build the
> branch and report any issues or problems you bump into.

Should users be concerned about the current situation in GCC ?

https://gcc.gnu.org/pipermail/gcc/2021-April/235340.html

The above thread has statements from several core developers that
paint a rather dire picture for a simple user like me - it looks
like GCC is imploding after the recent developments in the FSF.

As someone who kinda lives inside Emacs as far as computing
is concerned, I would like to use gccemacs, but GCC seems to
be heading toward some major disruptive event...

Sujith

Re: Getting ready to land native-compilation on master

[Alex Bennée]

Eli Zaretskii <eliz@gnu.org> writes:

> AFAICT, we are quite ready to land this important feature.  The branch
> was tested on several systems and is in good shape: many issues and
> bugs were fixed, and currently no known issues remain that block the
> merge.  (There's a lot yet to do wrt documenting the feature and its
> various aspects, but that can be done on master after merging.)
>
> If some of you have some unusual or rare or exotic system or Emacs
> configuration, and did not yet try building the native-comp branch,
> this is your chance to have a go before the merge: please build the
> branch and report any issues or problems you bump into.
>
> If no significant issues pop up within a week, I will ask Andrea to
> merge the branch onto master the next weekend (i.e. around 17th of
> April).
>
> Last, but not least: I'd like to take this opportunity to thank Andrea
> for his hard work and perseverance during this last year.  We would
> not be where we are now with this feature without his devotion and
> determination to see this through.  Thanks a lot!

+1 - I've been running the native-comp branch as my daily work driver
for most of the last year. The few problems I've run into have been
quickly addressed by Andrea on the list so I'm perfectly happy it's
ready for a wider testing audience.

I'll switch my daily driver branch once merged ;-)

-- 
Alex Bennée

Re: Getting ready to land native-compilation on master

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

> AFAICT, we are quite ready to land this important feature.  The branch
> was tested on several systems and is in good shape: many issues and
> bugs were fixed, and currently no known issues remain that block the
> merge.  (There's a lot yet to do wrt documenting the feature and its
> various aspects, but that can be done on master after merging.)
>
> If some of you have some unusual or rare or exotic system or Emacs
> configuration, and did not yet try building the native-comp branch,
> this is your chance to have a go before the merge: please build the
> branch and report any issues or problems you bump into.
>
> If no significant issues pop up within a week, I will ask Andrea to
> merge the branch onto master the next weekend (i.e. around 17th of
> April).
>
> Last, but not least: I'd like to take this opportunity to thank Andrea
> for his hard work and perseverance during this last year.  We would
> not be where we are now with this feature without his devotion and
> determination to see this through.  Thanks a lot!

Hi Eli,

having worked through this process with you all was an honor, but even
more was a pleasure :) I'm honestly very proud of being part of this
team.

I'm looking forward to keep on working on this to improve quality,
performance and functionality.

Thanks!

  Andrea

Re: Getting ready to land native-compilation on master

[Eli Zaretskii]

> From: Sujith Manoharan <sujith.oct@gmail.com>
> Cc: Andrea Corallo <akrl@sdf.org>, emacs-devel@gnu.org
> Date: Fri, 09 Apr 2021 23:39:36 +0530
> 
> Should users be concerned about the current situation in GCC ?
> 
> https://gcc.gnu.org/pipermail/gcc/2021-April/235340.html

That's something each user will have to decide for themselves.
libgccjit is a relatively immature part of GCC anyway (I mean relative
to the other components, like the C and C++ compilers), and no one
promises us that it will be maintained actively enough to not become a
problem in the future.  What matters is that the current state of
libgccjit as of GCC 9 and up is "good enough" for our purposes.

Re: Getting ready to land native-compilation on master

[Stefan Monnier]

> I've been on the branch for several months now with no major issues, the
> only oddity I've found was that natively-compiled elisp are subrs (as Pip
> already mentioned), which messed up my syntax highlighting.

I'm curious: why would it affect syntax highlighting?


        Stefan

Re: Getting ready to land native-compilation on master

[Eli Zaretskii]

> From: Thierry Volpiatto <thievol@posteo.net>
> Cc: Andrea Corallo <akrl@sdf.org>, emacs-devel@gnu.org
> Date: Fri, 09 Apr 2021 18:35:48 +0200
> 
> I didn't have any answer about bug#46790, many things are broken for me
> without fixing this...

I replied there now, and I hope that we will be able to come up with
some solution, although I'm a bit skeptical.

Re: Getting ready to land native-compilation on master

[Jens C. Jensen]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 09 Apr 2021 14:53:14 -0400

> I'm curious: why would it affect syntax highlighting?

I'm using `highlight-defined', which adds faces to defined variables/functions, I don't think it messes with any built-in highlighting.

Re: Getting ready to land native-compilation on master

[Andrea Corallo via Emacs development discussions.]

"Jens C. Jensen" <jens@subst.net> writes:

>> From: Eli Zaretskii <eliz@gnu.org>
>> Date: Fri, 09 Apr 2021 17:02:15 +0300
>> Cc: Andrea Corallo <akrl@sdf.org>
>
>> AFAICT, we are quite ready to land this important feature.
>
> This is great news!
>
> I've been on the branch for several months now with no major issues,
> the only oddity I've found was that natively-compiled elisp are subrs
> (as Pip already mentioned), which messed up my syntax highlighting.

Hi Jens,

native compiled Lisp shares with primitives many aspects including for
instance calling conventions, OTOH this is just other native code we can
execute.  Because of that felt natural to me to have these functions as
another kind of subr (also some of the internal implementation was
consequentially shared).  IOW in the current implementation they are
really (slightly special) subrs.
       
I guess you have already figured that out but you can distinguish them
with `subr-native-elisp-p'.

> A big Thank-You to Andrea for the great work getting this far!

You are welcome :)

> P.S. Is Andreas website on gccemacs[1] the proper place to get in-depth information about the implementation? It feels more like a change-log.
>
>
> [1] https://akrl.sdf.org/gccemacs.html

Well, ATM as Eli mentioned, documentaiton is something a little lacking,
certanly in my dev blog some datastructure and mechanisms are
described/justified.  Other than that you might want to have a look to:

<https://zenodo.org/record/3736363>
<https://toobnix.org/videos/watch/1f997b3c-00dc-4f7d-b2ce-74538c194fa7>
<https://toobnix.org/videos/watch/b985c5ca-fdcf-46ff-92d5-e68922fe4821>

ATM to go further I think the approach would be dumping passes and/or
starting to look into the source code.  I tried my best to share with
the community what I was working on and how but is not the easiest when
the thing is being worked.

Thanks!

  Andrea

Re: Getting ready to land native-compilation on master

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> Date: Fri, 09 Apr 2021 19:08:17 +0000
> 
> > P.S. Is Andreas website on gccemacs[1] the proper place to get in-depth information about the implementation? It feels more like a change-log.
> >
> >
> > [1] https://akrl.sdf.org/gccemacs.html
> 
> Well, ATM as Eli mentioned, documentaiton is something a little lacking,
> certanly in my dev blog some datastructure and mechanisms are
> described/justified.  Other than that you might want to have a look to:
> 
> <https://zenodo.org/record/3736363>
> <https://toobnix.org/videos/watch/1f997b3c-00dc-4f7d-b2ce-74538c194fa7>
> <https://toobnix.org/videos/watch/b985c5ca-fdcf-46ff-92d5-e68922fe4821>
> 
> ATM to go further I think the approach would be dumping passes and/or
> starting to look into the source code.  I tried my best to share with
> the community what I was working on and how but is not the easiest when
> the thing is being worked.

I think it would be good to take the relevant parts of your blog and
rewrite them as introductory commentary to the code in comp.el, with
the goal of providing enough overview and background information to
let people discover the details by reading the code.

Re: Getting ready to land native-compilation on master

[Stefan Monnier]

>> I'm curious: why would it affect syntax highlighting?
> I'm using `highlight-defined', which adds faces to defined
> variables/functions, I don't think it messes with any
> built-in highlighting.

Still: why does `subrp` affect it?


        Stefan

Re: Getting ready to land native-compilation on master

[Andy Moreton]

On Fri 09 Apr 2021, Eli Zaretskii wrote:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
>> Date: Fri, 09 Apr 2021 19:08:17 +0000
>> 
>> > P.S. Is Andreas website on gccemacs[1] the proper place to get in-depth
>> > information about the implementation? It feels more like a change-log.
>> >
>> >
>> > [1] https://akrl.sdf.org/gccemacs.html
>> 
>> Well, ATM as Eli mentioned, documentaiton is something a little lacking,
>> certanly in my dev blog some datastructure and mechanisms are
>> described/justified.  Other than that you might want to have a look to:
>> 
>> <https://zenodo.org/record/3736363>
>> <https://toobnix.org/videos/watch/1f997b3c-00dc-4f7d-b2ce-74538c194fa7>
>> <https://toobnix.org/videos/watch/b985c5ca-fdcf-46ff-92d5-e68922fe4821>
>> 
>> ATM to go further I think the approach would be dumping passes and/or
>> starting to look into the source code.  I tried my best to share with
>> the community what I was working on and how but is not the easiest when
>> the thing is being worked.
>
> I think it would be good to take the relevant parts of your blog and
> rewrite them as introductory commentary to the code in comp.el, with
> the goal of providing enough overview and background information to
> let people discover the details by reading the code.

Perhaps some of this material could be added to the "GNU Emacs
Internals" section of the elisp manual.

    AndyM

Re: Getting ready to land native-compilation on master

[Thierry Volpiatto]

[-- Attachment #1: Type: text/plain, Size: 510 bytes --]


Eli Zaretskii <eliz@gnu.org> writes:

>> From: Thierry Volpiatto <thievol@posteo.net>
>> Cc: Andrea Corallo <akrl@sdf.org>, emacs-devel@gnu.org
>> Date: Fri, 09 Apr 2021 18:35:48 +0200
>> 
>> I didn't have any answer about bug#46790, many things are broken for me
>> without fixing this...
>
> I replied there now,

Thanks.

> and I hope that we will be able to come up with some solution,
> although I'm a bit skeptical.

I am sure you will, thanks to all for this feature.

-- 
Thierry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 686 bytes --]

Re: Getting ready to land native-compilation on master

[Eli Zaretskii]

> From: Andy Moreton <andrewjmoreton@gmail.com>
> Date: Sat, 10 Apr 2021 01:07:40 +0100
> 
> >> <https://zenodo.org/record/3736363>
> >> <https://toobnix.org/videos/watch/1f997b3c-00dc-4f7d-b2ce-74538c194fa7>
> >> <https://toobnix.org/videos/watch/b985c5ca-fdcf-46ff-92d5-e68922fe4821>
> >> 
> >> ATM to go further I think the approach would be dumping passes and/or
> >> starting to look into the source code.  I tried my best to share with
> >> the community what I was working on and how but is not the easiest when
> >> the thing is being worked.
> >
> > I think it would be good to take the relevant parts of your blog and
> > rewrite them as introductory commentary to the code in comp.el, with
> > the goal of providing enough overview and background information to
> > let people discover the details by reading the code.
> 
> Perhaps some of this material could be added to the "GNU Emacs
> Internals" section of the elisp manual.

Some of it, probably.  But most of that stuff is inappropriate for the
manual.  We have in several files large commentaries that provide
overview of the design and implementation of various features.  For
example, the beginning of xdisp.c and coding.c, the overview of w32
subprocess support in w32proc.c, etc.  I was thinking about a similar
overview in comp.el.

Re: Getting ready to land native-compilation on master

[Jens C. Jensen]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Fri, 09 Apr 2021 18:43:59 -0400
> Cc: emacs-devel@gnu.org

>>> I'm curious: why would it affect syntax highlighting?
>> I'm using `highlight-defined', which adds faces to defined
>> variables/functions, I don't think it messes with any
>> built-in highlighting.
>
> Still: why does `subrp` affect it?

The problem seems to be the way `highlight-defined.el' determines if a function is built-in, using `(subrp (indirect-function #'some-function))'.

This evaluates to `nil' when an elisp function is defined or byte-compiled, but to `t' when it is native-compiled (or actually built-in).

The solution seems to be, like Andrea suggested, to test if a function is
`(and (subrp) (not (subr-native-elisp-p)))'.

Re: Getting ready to land native-compilation on master

[Andy Moreton]

On Sat 10 Apr 2021, Eli Zaretskii wrote:

>> From: Andy Moreton <andrewjmoreton@gmail.com>
>> Date: Sat, 10 Apr 2021 01:07:40 +0100
>> 
>> >> <https://zenodo.org/record/3736363>
>> >> <https://toobnix.org/videos/watch/1f997b3c-00dc-4f7d-b2ce-74538c194fa7>
>> >> <https://toobnix.org/videos/watch/b985c5ca-fdcf-46ff-92d5-e68922fe4821>
>> >> 
>> >> ATM to go further I think the approach would be dumping passes and/or
>> >> starting to look into the source code.  I tried my best to share with
>> >> the community what I was working on and how but is not the easiest when
>> >> the thing is being worked.
>> >
>> > I think it would be good to take the relevant parts of your blog and
>> > rewrite them as introductory commentary to the code in comp.el, with
>> > the goal of providing enough overview and background information to
>> > let people discover the details by reading the code.
>> 
>> Perhaps some of this material could be added to the "GNU Emacs
>> Internals" section of the elisp manual.
>
> Some of it, probably.  But most of that stuff is inappropriate for the
> manual.  We have in several files large commentaries that provide
> overview of the design and implementation of various features.  For
> example, the beginning of xdisp.c and coding.c, the overview of w32
> subprocess support in w32proc.c, etc.  I was thinking about a similar
> overview in comp.el.

Agreed: I expect that the lower level detail would go in comp.el and
a higher level overview in the manual.

    AndyM

Re: Getting ready to land native-compilation on master

[Stefan Monnier]

> The problem seems to be the way `highlight-defined.el' determines if
> a function is built-in, using `(subrp (indirect-function #'some-function))'.

Ah, so it purposefully decided to highlight "builtin" functions
differently.  Not knowing why this is done (whether a function is
implemented in C or in ELisp is an implementation detail, which can
change from one version of Emacs to another), it's hard to know what
should happen with native-compiled functions.


        Stefan

Re: Getting ready to land native-compilation on master

[Pankaj Jangid]

Eli Zaretskii <eliz@gnu.org> writes:

> AFAICT, we are quite ready to land this important feature.  The branch
> was tested on several systems and is in good shape: many issues and
> bugs were fixed, and currently no known issues remain that block the
> merge.  (There's a lot yet to do wrt documenting the feature and its
> various aspects, but that can be done on master after merging.)

I have started using Emacs from ‘native-comp’ branch last week. I am
doing this for the first time and following instructions from Andrea’s
gccemacs page.

I am on macos and after launching Emacs, I get this libgccjit
error. Is this something that I can ignore for now?

--8<---------------cut here---------------start------------->8---
Warning (comp): libgccjit.so: error: error invoking gcc driver Disable showing Disable logging
Warning (comp): /Users/pankaj/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/cl-lib.el.gz: Error: Internal native compiler error failed to compile Disable showing Disable logging
--8<---------------cut here---------------end--------------->8---

Further, there are lots of compiler warnings from different package, as I
use Emacs. That means the native compilation is definitely working.

Other than this I haven’t noticed anything that could be a blocker. My
several packages, Gnus, everything is working fine.

--
Regards
Pankaj Jangid

Re: Getting ready to land native-compilation on master

[Eli Zaretskii]

> From: Pankaj Jangid <pankaj@codeisgreat.org>
> Date: Wed, 14 Apr 2021 10:09:01 +0530
> 
> I am on macos and after launching Emacs, I get this libgccjit
> error. Is this something that I can ignore for now?
> 
> --8<---------------cut here---------------start------------->8---
> Warning (comp): libgccjit.so: error: error invoking gcc driver Disable showing Disable logging
> Warning (comp): /Users/pankaj/Applications/Emacs.app/Contents/Resources/lisp/emacs-lisp/cl-lib.el.gz: Error: Internal native compiler error failed to compile Disable showing Disable logging
> --8<---------------cut here---------------end--------------->8---
> 
> Further, there are lots of compiler warnings from different package, as I
> use Emacs. That means the native compilation is definitely working.

Thanks for the feedback.

Please report this problem using report-emacs-bug, and mention
feature/native-comp in the Subject.  When you do, please also tell
your version of GCC and libgccjit (you are using GCC, not clang that
disguises itself as GCC, right?).

Re: Getting ready to land native-compilation on master

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

> If no significant issues pop up within a week, I will ask Andrea to
> merge the branch onto master the next weekend (i.e. around 17th of
> April).

I have been trying out the branch for the last few days and I didn't
notice any functional issues.

What was is annoying is the "sudden" warning buffers pop up, triggered
by from what I understand is the asynchronous (?) compilation. Is there
a way to mitigate this, and if so, it should be documented.

-- 
	Philip K.

Re: Getting ready to land native-compilation on master

[Andrea Corallo via Emacs development discussions.]

Philip Kaludercic <philipk@posteo.net> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> If no significant issues pop up within a week, I will ask Andrea to
>> merge the branch onto master the next weekend (i.e. around 17th of
>> April).
>
> I have been trying out the branch for the last few days and I didn't
> notice any functional issues.
>
> What was is annoying is the "sudden" warning buffers pop up, triggered
> by from what I understand is the asynchronous (?) compilation. Is there
> a way to mitigate this, and if so, it should be documented.

I Philip,

one can act on the `comp-async-report-warnings-errors' customize setting
it to nil (default is t).

Thanks

  Andrea

Re: Getting ready to land native-compilation on master

[Philip Kaludercic]

Andrea Corallo <akrl@sdf.org> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>> If no significant issues pop up within a week, I will ask Andrea to
>>> merge the branch onto master the next weekend (i.e. around 17th of
>>> April).
>>
>> I have been trying out the branch for the last few days and I didn't
>> notice any functional issues.
>>
>> What was is annoying is the "sudden" warning buffers pop up, triggered
>> by from what I understand is the asynchronous (?) compilation. Is there
>> a way to mitigate this, and if so, it should be documented.
>
> I Philip,
>
> one can act on the `comp-async-report-warnings-errors' customize setting
> it to nil (default is t).

That should do it, thank you.

Does it make sense to have this enabled by default? At the very least,
ELPA packages should be addressed that trigger needless issues such as
overlong lines.

> Thanks
>
>   Andrea
>

-- 
	Philip K.

Re: Getting ready to land native-compilation on master

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Date: Wed, 14 Apr 2021 12:35:01 +0200
> Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> 
> > one can act on the `comp-async-report-warnings-errors' customize setting
> > it to nil (default is t).
> 
> That should do it, thank you.
> 
> Does it make sense to have this enabled by default?

We decided not to do that, because these warnings point out to real
problems in the code, which just go unnoticed by the byte-compiler.
They should be fixed ASAP.

> At the very least, ELPA packages should be addressed that trigger
> needless issues such as overlong lines.

I don't understand how overlong lines are significant in this context,
please elaborate.

Thanks.

Re: Getting ready to land native-compilation on master

[Andrea Corallo via Emacs development discussions.]

Philip Kaludercic <philipk@posteo.net> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Philip Kaludercic <philipk@posteo.net> writes:
>>
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>
>>>> If no significant issues pop up within a week, I will ask Andrea to
>>>> merge the branch onto master the next weekend (i.e. around 17th of
>>>> April).
>>>
>>> I have been trying out the branch for the last few days and I didn't
>>> notice any functional issues.
>>>
>>> What was is annoying is the "sudden" warning buffers pop up, triggered
>>> by from what I understand is the asynchronous (?) compilation. Is there
>>> a way to mitigate this, and if so, it should be documented.
>>
>> I Philip,
>>
>> one can act on the `comp-async-report-warnings-errors' customize setting
>> it to nil (default is t).
>
> That should do it, thank you.
>
> Does it make sense to have this enabled by default? At the very least,
> ELPA packages should be addressed that trigger needless issues such as
> overlong lines.

I think this was discussed a while ago here around and the consensus was
"for now" to keep it t as default so developers can come to know and
address issues, for the future I guess likely is going changed.

  Andrea

Re: Getting ready to land native-compilation on master

[Eli Zaretskii]

> Date: Wed, 14 Apr 2021 13:45:29 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org, akrl@sdf.org
> 
> > From: Philip Kaludercic <philipk@posteo.net>
> > Date: Wed, 14 Apr 2021 12:35:01 +0200
> > Cc: Eli Zaretskii <eliz@gnu.org>, emacs-devel@gnu.org
> > 
> > > one can act on the `comp-async-report-warnings-errors' customize setting
> > > it to nil (default is t).
> > 
> > That should do it, thank you.
> > 
> > Does it make sense to have this enabled by default?
> 
> We decided not to do that

Sorry, maybe you meant why it is enabled by default?  In that case, we
decided that would be a good default for the reasons I explained in my
previous message.

Re: Getting ready to land native-compilation on master

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> At the very least, ELPA packages should be addressed that trigger
>> needless issues such as overlong lines.
>
> I don't understand how overlong lines are significant in this context,
> please elaborate.

If I'm not mistaken, packages such as yasnippet generated warnings
(again, several minutes after Emacs itself was started) including more
or more or less trivial issues such as overlong lines.

-- 
	Philip K.

Re: Getting ready to land native-compilation on master

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: akrl@sdf.org,  emacs-devel@gnu.org
> Date: Wed, 14 Apr 2021 14:57:44 +0200
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> At the very least, ELPA packages should be addressed that trigger
> >> needless issues such as overlong lines.
> >
> > I don't understand how overlong lines are significant in this context,
> > please elaborate.
> 
> If I'm not mistaken, packages such as yasnippet generated warnings
> (again, several minutes after Emacs itself was started) including more
> or more or less trivial issues such as overlong lines.

Please show such a message, because I still don't think I understand,
perhaps because I don't know enough about yasnippet.

Re: Getting ready to land native-compilation on master

[Philip Kaludercic]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Philip Kaludercic <philipk@posteo.net>
>> Cc: akrl@sdf.org,  emacs-devel@gnu.org
>> Date: Wed, 14 Apr 2021 14:57:44 +0200
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> At the very least, ELPA packages should be addressed that trigger
>> >> needless issues such as overlong lines.
>> >
>> > I don't understand how overlong lines are significant in this
>> > context,
>> > please elaborate.
>> 
>> If I'm not mistaken, packages such as yasnippet generated warnings
>> (again, several minutes after Emacs itself was started) including
>> more
>> or more or less trivial issues such as overlong lines.
>
> Please show such a message, because I still don't think I understand,
> perhaps because I don't know enough about yasnippet.

From what I understand it is the regular compiler warning, just like you
would get without native-compile. So it is not related to yasnippet or
any other package itself. I tried it out, and in my case it looked like
this:

        Compiling file /home/philip/.config/emacs/elpa/yasnippet-0.14.0/yasnippet.el at Wed Apr 14 15:57:58 2021
        yasnippet.el:475:1: Warning: defvar `yas-after-exit-snippet-hook' docstring
            wider than 80 characters
        yasnippet.el:557:1: Warning: custom-declare-variable `yas-keymap-disable-hook'
            docstring wider than 80 characters

The difference being that this is not generated when the package is
installed, but suddenly appears in a buffer.

-- 
	Philip K.

Re: Getting ready to land native-compilation on master

[Stefan Kangas]

Philip Kaludercic <philipk@posteo.net> writes:

> From what I understand it is the regular compiler warning, just like you
> would get without native-compile. So it is not related to yasnippet or
> any other package itself. I tried it out, and in my case it looked like
> this:
>
>         Compiling file /home/philip/.config/emacs/elpa/yasnippet-0.14.0/yasnippet.el at Wed Apr 14 15:57:58 2021
>         yasnippet.el:475:1: Warning: defvar `yas-after-exit-snippet-hook' docstring
>             wider than 80 characters
>         yasnippet.el:557:1: Warning: custom-declare-variable `yas-keymap-disable-hook'
>             docstring wider than 80 characters

These warnings should be fixed in yasnippet itself.

> The difference being that this is not generated when the package is
> installed, but suddenly appears in a buffer.

This shouldn't be a problem once we flip
`comp-async-report-warnings-errors' to nil by default.  AFAIK, the plan
is to do that in time for Emacs 28.  As others have explained, we keep
it set to t for now as it indicates real problems.

Re: Getting ready to land native-compilation on master

[Eli Zaretskii]

> From: Philip Kaludercic <philipk@posteo.net>
> Cc: akrl@sdf.org,  emacs-devel@gnu.org
> Date: Wed, 14 Apr 2021 16:00:46 +0200
> 
> >> If I'm not mistaken, packages such as yasnippet generated warnings
> >> (again, several minutes after Emacs itself was started) including
> >> more
> >> or more or less trivial issues such as overlong lines.
> >
> > Please show such a message, because I still don't think I understand,
> > perhaps because I don't know enough about yasnippet.
> 
> >From what I understand it is the regular compiler warning, just like you
> would get without native-compile. So it is not related to yasnippet or
> any other package itself. I tried it out, and in my case it looked like
> this:
> 
>         Compiling file /home/philip/.config/emacs/elpa/yasnippet-0.14.0/yasnippet.el at Wed Apr 14 15:57:58 2021
>         yasnippet.el:475:1: Warning: defvar `yas-after-exit-snippet-hook' docstring
>             wider than 80 characters
>         yasnippet.el:557:1: Warning: custom-declare-variable `yas-keymap-disable-hook'
>             docstring wider than 80 characters

Ah, overlong lines in _docstrings_...  Yes, that's known, but why not
fix this issue in the first place?  I get the same warning when
byte-compiling with Emacs 28, so this is not new with the
native-compile branch.

Re: Getting ready to land native-compilation on master

[Philip Kaludercic]

Stefan Kangas <stefankangas@gmail.com> writes:

> Philip Kaludercic <philipk@posteo.net> writes:
>
>> From what I understand it is the regular compiler warning, just like you
>> would get without native-compile. So it is not related to yasnippet or
>> any other package itself. I tried it out, and in my case it looked like
>> this:
>>
>>         Compiling file /home/philip/.config/emacs/elpa/yasnippet-0.14.0/yasnippet.el at Wed Apr 14 15:57:58 2021
>>         yasnippet.el:475:1: Warning: defvar `yas-after-exit-snippet-hook' docstring
>>             wider than 80 characters
>>         yasnippet.el:557:1: Warning: custom-declare-variable `yas-keymap-disable-hook'
>>             docstring wider than 80 characters
>
> These warnings should be fixed in yasnippet itself.

Of course.

>> The difference being that this is not generated when the package is
>> installed, but suddenly appears in a buffer.
>
> This shouldn't be a problem once we flip
> `comp-async-report-warnings-errors' to nil by default.  AFAIK, the plan
> is to do that in time for Emacs 28.  As others have explained, we keep
> it set to t for now as it indicates real problems.

Good to hear, that certainly sounds sensible.

-- 
	Philip K.

Re: Getting ready to land native-compilation on master

[Dmitry Gutov]

On 14.04.2021 17:37, Eli Zaretskii wrote:
> Ah, overlong lines in_docstrings_...  Yes, that's known, but why not
> fix this issue in the first place?  I get the same warning when
> byte-compiling with Emacs 28, so this is not new with the
> native-compile branch.

I guess the issue is that native compilation happens after the package 
is installed, so the warnings come basically out of the blue.

If installed packages were native-compiled during the installation 
process, it would be less surprising. Though of course there are 
downsides to that approach as well.

Re: Getting ready to land native-compilation on master

[Stefan Monnier]

>> The difference being that this is not generated when the package is
>> installed, but suddenly appears in a buffer.
>
> This shouldn't be a problem once we flip
> `comp-async-report-warnings-errors' to nil by default.  AFAIK, the plan
> is to do that in time for Emacs 28.  As others have explained, we keep
> it set to t for now as it indicates real problems.

The warnings mentioned in this thread are those we normally see when we
byte-compile the file, which should be a separate step from the async
compilation to native code, so AFAICT showing those warnings during
native compilation is not very useful (what would be useful would be
warnings/info specific to async/native compilation).


        Stefan

Re: Getting ready to land native-compilation on master

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Philip Kaludercic <philipk@posteo.net>,  Eli Zaretskii <eliz@gnu.org>,
>   akrl@sdf.org,  emacs-devel@gnu.org
> Date: Wed, 14 Apr 2021 10:50:45 -0400
> 
> >> The difference being that this is not generated when the package is
> >> installed, but suddenly appears in a buffer.
> >
> > This shouldn't be a problem once we flip
> > `comp-async-report-warnings-errors' to nil by default.  AFAIK, the plan
> > is to do that in time for Emacs 28.  As others have explained, we keep
> > it set to t for now as it indicates real problems.
> 
> The warnings mentioned in this thread are those we normally see when we
> byte-compile the file, which should be a separate step from the async
> compilation to native code, so AFAICT showing those warnings during
> native compilation is not very useful

They are useful because native compilation frequently flags issues
that byte compilation doesn't (because the former happens in a more
pristine environment than the latter).

Re: Getting ready to land native-compilation on master

[wilde]

Eli Zaretskii <eliz@gnu.org> wrote:
> If some of you have some unusual or rare or exotic system or Emacs
> configuration, and did not yet try building the native-comp branch,
> this is your chance to have a go before the merge: please build the
> branch and report any issues or problems you bump into.

Today I build and started using Emacs from the on a small i368 system
with NetBSD 9.1:

System spec:
  CPU: Intel(R) Atom(TM) CPU N270   @ 1.60GHz
  RAM: 2 GiB

Build:
- I build libgccjit from gcc 10.2.0 manually as libgccjit seems not to
  be available from pkgsrc (not checked very thoroughly though) at least
  I found no `pkgin' installable binaries...

- I had to manually remove `-ldl' from LIBGCCJIT in src/Makefile as this
  is not needed on *BSD (and triggers an error during build, when
  present)

  IMO, this should be fixed in configure.ac (The same problem might
  exist for --with-modules, in case it is supported on *BSD, didn't test
  though...)

- I hat to disable memory protection on the system to make the native
  compiler work:
    sysctl -w security.pax.mprotect.global=0
    sysctl -w security.pax.mprotect.enabled=0
  This is not a Emacs specific problem but a problem with libgccjit
  itself.  (The basic gcc jit "Hello World" example also fails with
  memory protection in place).

The native compiling Emacs itself seems to run fine (I'm right now
writing this message in gnus on the said system) and the performance
improvement is very noticeable on this small system.  Thanks a LOT to
Andrea for this major improvement of GNU Emacs!

cheers
sascha

Re: Getting ready to land native-compilation on master

[Eli Zaretskii]

> From: wilde@sha-bang.de
> Cc: emacs-devel@gnu.org,  Andrea Corallo <akrl@sdf.org>
> Date: Wed, 14 Apr 2021 17:53:33 +0200
> 
> Today I build and started using Emacs from the on a small i368 system
> with NetBSD 9.1:
> 
> System spec:
>   CPU: Intel(R) Atom(TM) CPU N270   @ 1.60GHz
>   RAM: 2 GiB
> 
> Build:
> - I build libgccjit from gcc 10.2.0 manually as libgccjit seems not to
>   be available from pkgsrc (not checked very thoroughly though) at least
>   I found no `pkgin' installable binaries...
> 
> - I had to manually remove `-ldl' from LIBGCCJIT in src/Makefile as this
>   is not needed on *BSD (and triggers an error during build, when
>   present)

Andrea, looks like NetBSD is in the same boat as OpenBSD, where we
already refrain from using -ldl.

> - I hat to disable memory protection on the system to make the native
>   compiler work:
>     sysctl -w security.pax.mprotect.global=0
>     sysctl -w security.pax.mprotect.enabled=0
>   This is not a Emacs specific problem but a problem with libgccjit
>   itself.  (The basic gcc jit "Hello World" example also fails with
>   memory protection in place).

I'm not sure I understand why this happens, but I think this issue
should be reported to the GCC Bugzilla.

> The native compiling Emacs itself seems to run fine (I'm right now
> writing this message in gnus on the said system) and the performance
> improvement is very noticeable on this small system.  Thanks a LOT to
> Andrea for this major improvement of GNU Emacs!

Thanks for testing the branch.

Re: Getting ready to land native-compilation on master

[Eli Zaretskii]

> From: Dmitry Gutov <dgutov@yandex.ru>
> Date: Wed, 14 Apr 2021 17:48:11 +0300
> Cc: akrl@sdf.org, emacs-devel@gnu.org
> 
> On 14.04.2021 17:37, Eli Zaretskii wrote:
> > Ah, overlong lines in_docstrings_...  Yes, that's known, but why not
> > fix this issue in the first place?  I get the same warning when
> > byte-compiling with Emacs 28, so this is not new with the
> > native-compile branch.
> 
> I guess the issue is that native compilation happens after the package 
> is installed, so the warnings come basically out of the blue.

I was originally annoyed by this, but eventually concluded that it's
all part of the automatic background compilation, something we didn't
have before.  I think with time people will get used to it, and the
surprise will go away.  And if someone is really annoyed, they can
always disable automatic background compilation altogether.

I tend to think that turning warnings off is not a good idea, for the
same reasons we don't by default turn off byte-compilation warnings.
But we will cross this bridge when we come to it.

Re: Getting ready to land native-compilation on master

[Alex Bennée]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: wilde@sha-bang.de
>> Cc: emacs-devel@gnu.org,  Andrea Corallo <akrl@sdf.org>
>> Date: Wed, 14 Apr 2021 17:53:33 +0200
>> 
>> Today I build and started using Emacs from the on a small i368 system
>> with NetBSD 9.1:
>> 
>> System spec:
>>   CPU: Intel(R) Atom(TM) CPU N270   @ 1.60GHz
>>   RAM: 2 GiB
>> 
>> Build:
>> - I build libgccjit from gcc 10.2.0 manually as libgccjit seems not to
>>   be available from pkgsrc (not checked very thoroughly though) at least
>>   I found no `pkgin' installable binaries...
>> 
>> - I had to manually remove `-ldl' from LIBGCCJIT in src/Makefile as this
>>   is not needed on *BSD (and triggers an error during build, when
>>   present)
>
> Andrea, looks like NetBSD is in the same boat as OpenBSD, where we
> already refrain from using -ldl.
>
>> - I hat to disable memory protection on the system to make the native
>>   compiler work:
>>     sysctl -w security.pax.mprotect.global=0
>>     sysctl -w security.pax.mprotect.enabled=0
>>   This is not a Emacs specific problem but a problem with libgccjit
>>   itself.  (The basic gcc jit "Hello World" example also fails with
>>   memory protection in place).
>
> I'm not sure I understand why this happens, but I think this issue
> should be reported to the GCC Bugzilla.

I suspect this is a change similar to the recent MacOS one where
eXecutable pages cannot be mapped as Writable. The QEMU project recently
had to implement split mappings for it's JIT to workaround this although
I dare say there is probably a JIT interface that should be used to give
suitable access to pages but that is something for the BSD experts to
chime in on.

-- 
Alex Bennée

Re: Getting ready to land native-compilation on master

[Eli Zaretskii]

> From: Alex Bennée <alex.bennee@linaro.org>
> Cc: wilde@sha-bang.de, akrl@sdf.org, emacs-devel@gnu.org
> Date: Wed, 14 Apr 2021 18:28:49 +0100
> 
> >>     sysctl -w security.pax.mprotect.global=0
> >>     sysctl -w security.pax.mprotect.enabled=0
> >>   This is not a Emacs specific problem but a problem with libgccjit
> >>   itself.  (The basic gcc jit "Hello World" example also fails with
> >>   memory protection in place).
> >
> > I'm not sure I understand why this happens, but I think this issue
> > should be reported to the GCC Bugzilla.
> 
> I suspect this is a change similar to the recent MacOS one where
> eXecutable pages cannot be mapped as Writable. The QEMU project recently
> had to implement split mappings for it's JIT to workaround this although
> I dare say there is probably a JIT interface that should be used to give
> suitable access to pages but that is something for the BSD experts to
> chime in on.

Are you sure this is relevant?  libgccjit, contrary to its name, is
not JIT code production, at least in Emacs.  It produces a file, a
shared library which is later loaded.  Why does this have to need
executable memory pages?

Re: Getting ready to land native-compilation on master

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: wilde@sha-bang.de
>> Cc: emacs-devel@gnu.org,  Andrea Corallo <akrl@sdf.org>
>> Date: Wed, 14 Apr 2021 17:53:33 +0200
>> 
>> Today I build and started using Emacs from the on a small i368 system
>> with NetBSD 9.1:
>> 
>> System spec:
>>   CPU: Intel(R) Atom(TM) CPU N270   @ 1.60GHz
>>   RAM: 2 GiB
>> 
>> Build:
>> - I build libgccjit from gcc 10.2.0 manually as libgccjit seems not to
>>   be available from pkgsrc (not checked very thoroughly though) at least
>>   I found no `pkgin' installable binaries...
>> 
>> - I had to manually remove `-ldl' from LIBGCCJIT in src/Makefile as this
>>   is not needed on *BSD (and triggers an error during build, when
>>   present)
>
> Andrea, looks like NetBSD is in the same boat as OpenBSD, where we
> already refrain from using -ldl.

Right thanks for the hint, bfaa6df492 should do the job; Sascha please
have a try.

  Andrea

PS thanks also for fixing 0c1fc9d581

Re: Getting ready to land native-compilation on master

[Stefan Kangas]

Andrea Corallo via "Emacs development discussions."
<emacs-devel@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: wilde@sha-bang.de
>>> Cc: emacs-devel@gnu.org,  Andrea Corallo <akrl@sdf.org>
>>> Date: Wed, 14 Apr 2021 17:53:33 +0200
>>>
>>> Today I build and started using Emacs from the on a small i368 system
>>> with NetBSD 9.1:
>>>
>>> System spec:
>>>   CPU: Intel(R) Atom(TM) CPU N270   @ 1.60GHz
>>>   RAM: 2 GiB
>>>
>>> Build:
>>> - I build libgccjit from gcc 10.2.0 manually as libgccjit seems not to
>>>   be available from pkgsrc (not checked very thoroughly though) at least
>>>   I found no `pkgin' installable binaries...
>>>
>>> - I had to manually remove `-ldl' from LIBGCCJIT in src/Makefile as this
>>>   is not needed on *BSD (and triggers an error during build, when
>>>   present)
>>
>> Andrea, looks like NetBSD is in the same boat as OpenBSD, where we
>> already refrain from using -ldl.
>
> Right thanks for the hint, bfaa6df492 should do the job; Sascha please
> have a try.

The above seems to be talking about NetBSD, but bfaa6df492 adds
"freebsd" to configure.ac.  Is that expected?

Re: Getting ready to land native-compilation on master

[Andrea Corallo via Emacs development discussions.]

Stefan Kangas <stefankangas@gmail.com> writes:

> Andrea Corallo via "Emacs development discussions."
> <emacs-devel@gnu.org> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>>> From: wilde@sha-bang.de
>>>> Cc: emacs-devel@gnu.org,  Andrea Corallo <akrl@sdf.org>
>>>> Date: Wed, 14 Apr 2021 17:53:33 +0200
>>>>
>>>> Today I build and started using Emacs from the on a small i368 system
>>>> with NetBSD 9.1:
>>>>
>>>> System spec:
>>>>   CPU: Intel(R) Atom(TM) CPU N270   @ 1.60GHz
>>>>   RAM: 2 GiB
>>>>
>>>> Build:
>>>> - I build libgccjit from gcc 10.2.0 manually as libgccjit seems not to
>>>>   be available from pkgsrc (not checked very thoroughly though) at least
>>>>   I found no `pkgin' installable binaries...
>>>>
>>>> - I had to manually remove `-ldl' from LIBGCCJIT in src/Makefile as this
>>>>   is not needed on *BSD (and triggers an error during build, when
>>>>   present)
>>>
>>> Andrea, looks like NetBSD is in the same boat as OpenBSD, where we
>>> already refrain from using -ldl.
>>
>> Right thanks for the hint, bfaa6df492 should do the job; Sascha please
>> have a try.
>
> The above seems to be talking about NetBSD, but bfaa6df492 adds
> "freebsd" to configure.ac.  Is that expected?

No it's not, 686259e65a should do better, thanks for catching!

  Andrea

Re: Getting ready to land native-compilation on master

[wilde]

Andrea Corallo via "Emacs development discussions." <emacs-devel@gnu.org> wrote:
> Stefan Kangas <stefankangas@gmail.com> writes:
>> Andrea Corallo via "Emacs development discussions."
>> <emacs-devel@gnu.org> writes:
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>>> From: wilde@sha-bang.de
>>>>> Date: Wed, 14 Apr 2021 17:53:33 +0200
>>>>>
>>>>> Today I build and started using Emacs from the on a small i368 system
>>>>> with NetBSD 9.1:

>>>>> - I had to manually remove `-ldl' from LIBGCCJIT in src/Makefile as this
>>>>>   is not needed on *BSD (and triggers an error during build, when
>>>>>   present)
>>>>
>>>> Andrea, looks like NetBSD is in the same boat as OpenBSD, where we
>>>> already refrain from using -ldl.
>>>
>>> Right thanks for the hint, bfaa6df492 should do the job; Sascha please
>>> have a try.
>>
>> The above seems to be talking about NetBSD, but bfaa6df492 adds
>> "freebsd" to configure.ac.  Is that expected?
>
> No it's not, 686259e65a should do better, thanks for catching!

I can confirm, that the branch now builds without manual changes to the
Makefile on NetBSD.  Thanks!

As side note: it seems you reverted the change for freebsd, my
understanding is that the libc of all *BSD is rather similar in this
respect, therefor I'd expect the change to be needed for freebsd as
well.  :)

cheers
sascha

Re: Getting ready to land native-compilation on master

[Alex Bennée]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Alex Bennée <alex.bennee@linaro.org>
>> Cc: wilde@sha-bang.de, akrl@sdf.org, emacs-devel@gnu.org
>> Date: Wed, 14 Apr 2021 18:28:49 +0100
>> 
>> >>     sysctl -w security.pax.mprotect.global=0
>> >>     sysctl -w security.pax.mprotect.enabled=0
>> >>   This is not a Emacs specific problem but a problem with libgccjit
>> >>   itself.  (The basic gcc jit "Hello World" example also fails with
>> >>   memory protection in place).
>> >
>> > I'm not sure I understand why this happens, but I think this issue
>> > should be reported to the GCC Bugzilla.
>> 
>> I suspect this is a change similar to the recent MacOS one where
>> eXecutable pages cannot be mapped as Writable. The QEMU project recently
>> had to implement split mappings for it's JIT to workaround this although
>> I dare say there is probably a JIT interface that should be used to give
>> suitable access to pages but that is something for the BSD experts to
>> chime in on.
>
> Are you sure this is relevant?

Not entirely - it was an educated guess but the write up in the BSD
pages seems to indicate this behaviour:

   PaX MPROTECT
     PaX MPROTECT implements memory protection restrictions, meant to comple-
     ment non-executable mappings.  The purpose is to prevent situations where
     malicious code attempts to mark writable memory regions as executable,
     often by trashing arguments to an mprotect(2) call.

     While it can be enabled globally, NetBSD provides a tool, paxctl(8), to
     enable PaX MPROTECT on a per-program basis.

     Example usage:

           # paxctl +M /usr/sbin/sshd

     Enabling PaX MPROTECT globally:

           # sysctl -w security.pax.mprotect.global=1

     PaX MPROTECT affects the following three uses:

           ·   Processes that utilize code generation (such as the JVM) might
               need to have MPROTECT disabled.

           ·   Miscompiled programs that have text relocations, will now core
               dump instead of having their relocations corrected.  You will
               need to fix those programs (recompile them properly).

           ·   Debugger breakpoints: gdb(1) needs to be able to write to the
               text segment in order to insert and delete breakpoints.  This
               will not work unless MPROTECT is disabled on the executable.

> libgccjit, contrary to its name, is not JIT code production, at least
> in Emacs. It produces a file, a shared library which is later loaded.
> Why does this have to need executable memory pages?

Well any page with code needs to executable. The question is how does
libgccjit handle it? Does it write a stream of bytes to a file that is
then mapped in r-x to be executed or does it map a file -w- to write the
instruction stream and then just remap it afterwards r-x when it's
"loaded". If it's ultimately the same page changes permissions I can
expect the above would complain.

I'm not familiar enough with how libgccjit generates and loads the code.

-- 
Alex Bennée

Re: Getting ready to land native-compilation on master

[wilde]

Eli Zaretskii <eliz@gnu.org> wrote:
>> - I hat to disable memory protection on the system to make the native
>>   compiler work:
>>     sysctl -w security.pax.mprotect.global=0
>>     sysctl -w security.pax.mprotect.enabled=0
>>   This is not a Emacs specific problem but a problem with libgccjit
>>   itself.  (The basic gcc jit "Hello World" example also fails with
>>   memory protection in place).
>
> I'm not sure I understand why this happens, but I think this issue
> should be reported to the GCC Bugzilla.

It turns out, that disabling security.pax.mprotect.global is enough,
however this still poses a potential security risk and means that
libgccjit can not be used on default setup of NetBSD.

I created a upstream report:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100096

cheers
sascha

Re: Getting ready to land native-compilation on master

[Andrea Corallo via Emacs development discussions.]

wilde@sha-bang.de writes:

> Eli Zaretskii <eliz@gnu.org> wrote:
>>> - I hat to disable memory protection on the system to make the native
>>>   compiler work:
>>>     sysctl -w security.pax.mprotect.global=0
>>>     sysctl -w security.pax.mprotect.enabled=0
>>>   This is not a Emacs specific problem but a problem with libgccjit
>>>   itself.  (The basic gcc jit "Hello World" example also fails with
>>>   memory protection in place).
>>
>> I'm not sure I understand why this happens, but I think this issue
>> should be reported to the GCC Bugzilla.
>
> It turns out, that disabling security.pax.mprotect.global is enough,
> however this still poses a potential security risk and means that
> libgccjit can not be used on default setup of NetBSD.
>
> I created a upstream report:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100096

Thanks

  Andrea

Re: Getting ready to land native-compilation on master

[wilde]

Hi Eli,

Eli Zaretskii <eliz@gnu.org> wrote:
> AFAICT, we are quite ready to land this important feature.
[…]
> If no significant issues pop up within a week, I will ask Andrea to
> merge the branch onto master the next weekend (i.e. around 17th of
> April).

As this did not happen yet I assume that significant bugs, blocking the
merge, did pop up (one of which might have been the PATH related issues
I was also experiencing and testing).

Please forgive my ignorance, but is there a place where the progress on
the merge and the list of blocking issues are tracked?

I don't want to be a nuisance, but I'm really looking forward to this
great feature being merged into master...  :)

cheers
sascha

Re: Getting ready to land native-compilation on master

[Eli Zaretskii]

> From: wilde@sha-bang.de
> Cc: emacs-devel@gnu.org,  Andrea Corallo <akrl@sdf.org>
> Date: Thu, 22 Apr 2021 12:09:38 +0200
> 
> Hi Eli,
> 
> Eli Zaretskii <eliz@gnu.org> wrote:
> > AFAICT, we are quite ready to land this important feature.
> […]
> > If no significant issues pop up within a week, I will ask Andrea to
> > merge the branch onto master the next weekend (i.e. around 17th of
> > April).
> 
> As this did not happen yet I assume that significant bugs, blocking the
> merge, did pop up (one of which might have been the PATH related issues
> I was also experiencing and testing).

No blocking bugs are known at this point, we just delayed the merge
due to the issue with symlinks.  If no new problems are reported, I
will ask Andrea to merge this coming weekend.

Re: Getting ready to land native-compilation on master

[wilde]

wilde@sha-bang.de wrote:
>>> - I hat to disable memory protection on the system to make the native
>>>   compiler work:
[...]
> I created a upstream report:
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100096

Just in case someone else stumbles across this thread: the upstream bug
has been resolved and should be fixed in subsequent releases of gcc.
(Its even cherry picked for the gcc 10 release branch)

Re: Getting ready to land native-compilation on master

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

\o/

On Tue, Apr 27, 2021 at 11:26 AM <wilde@sha-bang.de> wrote:

> wilde@sha-bang.de wrote:
> >>> - I hat to disable memory protection on the system to make the native
> >>>   compiler work:
> [...]
> > I created a upstream report:
> > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100096
>
> Just in case someone else stumbles across this thread: the upstream bug
> has been resolved and should be fixed in subsequent releases of gcc.
> (Its even cherry picked for the gcc 10 release branch)
>
>
Congratulations and thank you!

In a similar vein, for the benefit of those who may find it useful later:
The below linked thread celebrates the completed merge of "gccmacs" (the
ability for Emacs to transform elisp into "native" compilation units) into
master for release (probably) with Emacs 28 (dates TBD or IDK ;)

https://lists.gnu.org/archive/html/emacs-devel/2021-04/msg01175.html

-- 
*Corwin*
612-217-1742
612-695-4276 (signal)
*corwin@bru.st <corwin@bru.st>*

[-- Attachment #2: Type: text/html, Size: 2411 bytes --]

Re: Getting ready to land native-compilation on master

[Jose E. Marchesi]

> wilde@sha-bang.de wrote:
>>>> - I hat to disable memory protection on the system to make the native
>>>>   compiler work:
> [...]
>> I created a upstream report:
>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100096
>
> Just in case someone else stumbles across this thread: the upstream bug
> has been resolved and should be fixed in subsequent releases of gcc.
> (Its even cherry picked for the gcc 10 release branch)

This is very exciting news.
Congratulations for the great work! :)

Re: Getting ready to land native-compilation on master

[Arthur Miller]

"Jose E. Marchesi" <jemarch@gnu.org> writes:

>> wilde@sha-bang.de wrote:
>>>>> - I hat to disable memory protection on the system to make the native
>>>>>   compiler work:
>> [...]
>>> I created a upstream report:
>>> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100096
>>
>> Just in case someone else stumbles across this thread: the upstream bug
>> has been resolved and should be fixed in subsequent releases of gcc.
>> (Its even cherry picked for the gcc 10 release branch)
>
> This is very exciting news.
> Congratulations for the great work! :)

Indeed, congrats from me too. Thanks to Andrea, Eli and all of you who
worked hard to make this possible! <3