Porting my conf to 28.0.60 (MS Windows + native-comp)

Porting my conf to 28.0.60 (MS Windows + native-comp)

[Corwin Brust]

Greetings Emacs developers!

This morning I was able to get my configuration fairly portable between 26/2728.
I'm running Windows 10 and using msys32_64 to compile.  I'd already
spent some time getting msys updated to include libgccjit and I'm not
going into that process here.  Rather this is a quick note to share my
configuration with you, along with a couple of things I ran into and
haven't solved.

My configuration is on GitLab.  I use a number of packages from MELPA,
and I have a personal configuration "management system", where I
create simple packages for groups of configuration based on
use/propose of each group.   This let's me easily adjust which things
load when Emacs starts, and quickly bring in "hunks" of customized
functionality when I'm doing stuff outside my regular day-to-day Emacs
use.  For more information on how I do this start at elisp/init.el

My conf repo:
https://gitlab.com/mplscorwin/dotfiles/

These two commits show the changes I made:
1. an early init to adjust `package-user-dir'
https://gitlab.com/mplscorwin/dotfiles/-/commit/666f4b130a9925be8c989e1e6695f2e03d169b96

2. the rest of my various changes:
https://gitlab.com/mplscorwin/dotfiles/-/commit/74f36d0e540188eb231dbce6b0ae003078067983

In the second link the principle changes are to init.el and
init-package-management.el - I'm afraid there are some unrelated
changes (theme changes, etc) mixed in.  For the most part if I've
commented stuff out (in any of the elisp files) it is because I've not
yet gotten that working with 28/native-comp.

Finally, here's the command I'm using to complete Emacs:

#+BEGIN_SRC sh

git pull && make clean && ./autogen.sh && ./configure
--with-native-compilation && make -j 20

#+END_SRC

I play to play around with adding NATIVE_FULL_AOT=1 to the end of that
string but I'd like to solve the "too many pipes" issue first, given
that's solvable.

Aside the few (melpa) packages I'm having trouble with and commented
out (yaml-mode and forge), I also had to expressly add a line for
`fullframe', but I'm not sure why.  (Maybe I should have had this all
along but I don't remember an error when installing everything on/from
an Emacs 26 on a GNU/linux system.)

I'm also having some trouble with magit.  In fact, magit is completely
unusable but I've not started digging deeply as to /why/ that is.

Otherwise, my only concern has to do with "too many pipes" errors when
I remove my elpa folder and force Emacs to re-fetch all of my
"non-core" packages when it next starts-up. I would be grateful indeed
for any additional* workarounds the team could suggest for that issue.

* I have some workarounds already in place already but they haven't
quite solved the issue.
I found these starting from use-package issue #96 and have added links
in my conf code just above where I've attempted to use them.
  https://github.com/jwiegley/emacs-async/issues/96

TIA for your thoughts and suggestions and (as always) for keeping the
best text editor in history moving forward.

Warm regards,
Corwin

Re: Porting my conf to 28.0.60 (MS Windows + native-comp)

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Sun, 17 Oct 2021 13:50:08 -0500
> 
> Otherwise, my only concern has to do with "too many pipes" errors when
> I remove my elpa folder and force Emacs to re-fetch all of my
> "non-core" packages when it next starts-up. I would be grateful indeed
> for any additional* workarounds the team could suggest for that issue.

Does it happen only with Emacs 28, or with previous versions as well?

I cannot access your Gitlab repository (Gitlab doesn't like my
browser), so you'll have to tell what customizations you did that
could potentially cause Emacs to spawn many subprocesses and/o0r
network connections.  On MS-Windows, there's a hard limit of 31, I
think.

If this doesn't help, please submit a bug report with all the details.

Re: Porting my conf to 28.0.60 (MS Windows + native-comp)

[Corwin Brust]

Thank you, Eli.  A quick one to share newer, better repo links.
More to come.

On Sun, Oct 17, 2021 at 2:20 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> I cannot access your Gitlab repository (Gitlab doesn't like my
> browser), [...]
>

This was the nudge I needed to migrate away from GitLab completely.

My dot-files are now maintained here:
  https://git.sr.ht/~mplscorwin/dotfiles

This link is the commit with the changes made to use native-comp (and
a few others, ug):
  https://git.sr.ht/~mplscorwin/dotfiles/commit/74f36d0e540188eb231dbce6b0ae003078067983

This link is the commit where I setup early-init to swap around
`package-user-dir':
  https://git.sr.ht/~mplscorwin/dotfiles/commit/666f4b130a9925be8c989e1e6695f2e03d169b96

I'll next work on seeing which Emacs versions I can produce the 'too
many pipes' error under and finding a minimum reproducer.  Perhaps
also a bug if I convince myself it is one.

Re: Porting my conf to 28.0.60 (MS Windows + native-comp)

[Corwin Brust]

Hi again Eli, all.

On Sun, Oct 17, 2021 at 2:20 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Corwin Brust <corwin@bru.st>
> > Date: Sun, 17 Oct 2021 13:50:08 -0500
> >
> > Otherwise, my only concern has to do with "too many pipes" errors when
> > I remove my elpa folder and force Emacs to re-fetch all of my
> > "non-core" packages when it next starts-up. I would be grateful indeed
> > for any additional* workarounds the team could suggest for that issue.
>
> Does it happen only with Emacs 28, or with previous versions as well?

I wasn't able to reproduce under Emacs 27.1

> If this doesn't help, please submit a bug report with all the details.

Feel free to let me know otherwise, but I'm planning to wait for
Andras' reply to your note several hours back before I open a bug
report.  It seems entirely possible I've not used the correct settings
in attempting to throttle this.  Moreover, I forgot to commit the
changes to my .emacs, where I'd stuffed this bit:

#+BEGIN_SRC emacs-lisp

;; tweaks for native-comp
;; https://ddavis.io/posts/emacs-native-centos7/#deferred-and-asynchronous-compilation
;; via https://github.com/jwiegley/emacs-async/issues/96
;; helper boolean I use here and later in my init.el

(defconst my-using-native-comp
  (and (not (version< emacs-version "28.0"))
       (fboundp 'native-comp-available-p)
       (native-comp-available-p))
  "Indicates we are using native compilation (via libgccjit).")

(when my-using-native-comp
  (message "Native compilation is avialable; throttling")
  (setq native-comp-deferred-compilation t)
  (setq native-comp-async-query-on-exit t)
  (setq native-comp-async-jobs-number 4)
  (setq native-comp-async-report-warnings-errors nil))

#+END_SRC

My .emacs is now updated, here:
https://git.sr.ht/~mplscorwin/dotfiles/commit/0c88634b903393ec4f64fb250a4bee62c47003f9

Re: Porting my conf to 28.0.60 (MS Windows + native-comp)

[Andrea Corallo]

Corwin Brust <corwin@bru.st> writes:

> Hi again Eli, all.
>
> On Sun, Oct 17, 2021 at 2:20 PM Eli Zaretskii <eliz@gnu.org> wrote:
>>
>> > From: Corwin Brust <corwin@bru.st>
>> > Date: Sun, 17 Oct 2021 13:50:08 -0500
>> >
>> > Otherwise, my only concern has to do with "too many pipes" errors when
>> > I remove my elpa folder and force Emacs to re-fetch all of my
>> > "non-core" packages when it next starts-up. I would be grateful indeed
>> > for any additional* workarounds the team could suggest for that issue.
>>
>> Does it happen only with Emacs 28, or with previous versions as well?
>
> I wasn't able to reproduce under Emacs 27.1

Just a naive question. Have you tried Emacs 28 non native compiled?

>> If this doesn't help, please submit a bug report with all the details.
>
> Feel free to let me know otherwise, but I'm planning to wait for
> Andras' reply to your note several hours back before I open a bug
> report.  It seems entirely possible I've not used the correct settings
> in attempting to throttle this.  Moreover, I forgot to commit the
> changes to my .emacs, where I'd stuffed this bit:
>
> #+BEGIN_SRC emacs-lisp
>
> ;; tweaks for native-comp
> ;; https://ddavis.io/posts/emacs-native-centos7/#deferred-and-asynchronous-compilation
> ;; via https://github.com/jwiegley/emacs-async/issues/96
> ;; helper boolean I use here and later in my init.el
>
> (defconst my-using-native-comp
>   (and (not (version< emacs-version "28.0"))
>        (fboundp 'native-comp-available-p)
>        (native-comp-available-p))
>   "Indicates we are using native compilation (via libgccjit).")
>
> (when my-using-native-comp
>   (message "Native compilation is avialable; throttling")
>   (setq native-comp-deferred-compilation t)
>   (setq native-comp-async-query-on-exit t)
>   (setq native-comp-async-jobs-number 4)
>   (setq native-comp-async-report-warnings-errors nil))
>
> #+END_SRC

I see nothing wrong with the settings above (but haven't looked into
all your config).

BR

  Andrea

Re: Porting my conf to 28.0.60 (MS Windows + native-comp)

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 536 bytes --]

Hi Andrea

On Mon, Oct 18, 2021, 03:38 Andrea Corallo <akrl@sdf.org> wrote:

> Corwin Brust <corwin@bru.st> writes:
>
> > Hi again Eli, all.
> >
> > On Sun, Oct 17, 2021 at 2:20 PM Eli Zaretskii <eliz@gnu.org> wrote:
> >> [snip]
> >>
> >> Does it happen only with Emacs 28, or with previous versions as well?
> >
> > I wasn't able to reproduce under Emacs 27.1
>
> Just a naive question. Have you tried Emacs 28 non native compiled?



Without native comp or with pre-compilation during build I don't get these
'too many pipes' errors.

[-- Attachment #2: Type: text/html, Size: 1125 bytes --]

Re: Porting my conf to 28.0.60 (MS Windows + native-comp)

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Sun, 17 Oct 2021 20:51:02 -0500
> Cc: Emacs developers <emacs-devel@gnu.org>
> 
> ;; tweaks for native-comp
> ;; https://ddavis.io/posts/emacs-native-centos7/#deferred-and-asynchronous-compilation
> ;; via https://github.com/jwiegley/emacs-async/issues/96
> ;; helper boolean I use here and later in my init.el
> 
> (defconst my-using-native-comp
>   (and (not (version< emacs-version "28.0"))
>        (fboundp 'native-comp-available-p)
>        (native-comp-available-p))
>   "Indicates we are using native compilation (via libgccjit).")
> 
> (when my-using-native-comp
>   (message "Native compilation is avialable; throttling")
>   (setq native-comp-deferred-compilation t)
>   (setq native-comp-async-query-on-exit t)
>   (setq native-comp-async-jobs-number 4)
>   (setq native-comp-async-report-warnings-errors nil))

I see nothing here that could cause Emacs to spawn too many child
processes for native-compilation: you limit the number of simultaneous
sub-processes to 4, which shouldn't be a problem.

So I still think you should try preparing a reproducer for that
strange error message.  When it happens, could you perhaps see how
many network connections does Emacs have (with, for example, netstat)?

Re: Porting my conf to 28.0.60 (MS Windows + native-comp)

[H. Dieter Wilhelm]

Hi Corwin

Hope I'll soon find the time to look at your configuration.

Corwin Brust <corwin@bru.st> writes:

> Finally, here's the command I'm using to complete Emacs:
>
> #+BEGIN_SRC sh
>
> git pull && make clean && ./autogen.sh && ./configure
> --with-native-compilation && make -j 20
>
> #+END_SRC

Thanks for the hint on gcc-emacs. :-)

> I'm also having some trouble with magit.  In fact, magit is completely
> unusable but I've not started digging deeply as to /why/ that is.

I didn't test magit extensively but it seems to be fully functional
under emacs-28.0.60 (Windows 10).

  Dieter

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Porting my conf to 28.0.60 (MS Windows + native-comp)

[H. Dieter Wilhelm]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> Hi Corwin
>
> Hope I'll soon find the time to look at your configuration.
>
> Corwin Brust <corwin@bru.st> writes:
>
>> Finally, here's the command I'm using to complete Emacs:
>>
>> #+BEGIN_SRC sh
>>
>> git pull && make clean && ./autogen.sh && ./configure
>> --with-native-compilation && make -j 20
>>
>> #+END_SRC
>
> Thanks for the hint on gcc-emacs. :-)
>
>> I'm also having some trouble with magit.  In fact, magit is completely
>> unusable but I've not started digging deeply as to /why/ that is.
>
> I didn't test magit extensively but it seems to be fully functional
> under emacs-28.0.60 (Windows 10).

I forgot to mention that I've installed git for windows before MSYS2,
I'm not sure which executable magit is actually using...