Possibly defaulting native-compile-target-directory to XDG_CACHE_HOME/...

Possibly defaulting native-compile-target-directory to XDG_CACHE_HOME/...

[Rob Browning]

We're still trying to resolve some native compilation related crashes in
Debian in the hope that we'll be able to have 28 in the next stable
release --with-native-compilation.  (Stage one of the freeze begins on
Jan 12.)

One remaining issue is that emacs crashes when HOME=/nope in some of our
test suites and perhapssome other packaging or install-related
activities.  For example: https://bugs.debian.org/1021842

(To refresh, Debian policies require packages not to write outside of a
 few specified locations in certain situations, e.g. package builds, and
 HOME is not one of the specified locations.)

I can't recall from previous discussions whether that crash would be
considered a bug, or whether emacs just requires a writable cache dir
now (which is fine if so).

But I have discovered that if we allow the redirection of the
native-compile-target-directory to XDG_CACHE_HOME via something like
this in comp.el:

  (defvar native-compile-target-directory (getenv "XDG_CACHE_HOME")
    "When non-nil force the target directory for the eln files being compiled.")

then we can resolve the issue:

  export HOME=/nope
  export XDG_CACHE_HOME="$(mktemp -d ...)"
  run-tests...

Of course, I assume the default above isn't quite right, i.e. if we were
to pursue this, I imagine we'd actually want to default
native-compile-target-directory to something more like
"$XDG_CACHE_HOME/emacs/eln-cache" when XDG_CACHE_DIR is set.

I also noticed that emacs already respects XDG_CONFIG_HOME, so I
imagined relying on XDG_CACHE_HOME might not be entirely suspect.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Possibly defaulting native-compile-target-directory to XDG_CACHE_HOME/...

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Date: Fri, 16 Dec 2022 00:08:37 -0600
> 
> 
> We're still trying to resolve some native compilation related crashes in
> Debian in the hope that we'll be able to have 28 in the next stable
> release --with-native-compilation.  (Stage one of the freeze begins on
> Jan 12.)
> 
> One remaining issue is that emacs crashes when HOME=/nope in some of our
> test suites and perhapssome other packaging or install-related
> activities.  For example: https://bugs.debian.org/1021842
> 
> (To refresh, Debian policies require packages not to write outside of a
>  few specified locations in certain situations, e.g. package builds, and
>  HOME is not one of the specified locations.)
> 
> I can't recall from previous discussions whether that crash would be
> considered a bug, or whether emacs just requires a writable cache dir
> now (which is fine if so).

The latter, AFAIU.

Re: Possibly defaulting native-compile-target-directory to XDG_CACHE_HOME/...

[Stefan Monnier]

> I can't recall from previous discussions whether that crash would be
> considered a bug, or whether emacs just requires a writable cache dir
> now (which is fine if so).

I think it depends what you mean by "crash".  To me "crash" means
something like a segfault or coredump, and I expect we treat it as a bug
in Emacs.
OTOH if you get a meaningful error message before the "crash", then the
meaningful error message should hopefully clarify whether it's a bug in
Emacs or a bug in your use of it.


        Stefan

Re: Possibly defaulting native-compile-target-directory to XDG_CACHE_HOME/...

[Rob Browning]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

> I think it depends what you mean by "crash".  To me "crash" means
> something like a segfault or coredump, and I expect we treat it as a
> bug in Emacs.  OTOH if you get a meaningful error message before the
> "crash", then the meaningful error message should hopefully clarify
> whether it's a bug in Emacs or a bug in your use of it.

It's the error described in the bug report, fwiw:
https://bugs.debian.org/1021842 i.e.

  ...
  apply((closure ((elc-time . 2) (el-time . 3) t) nil (spy-on 'file-attribut...
  (closure ((elc-time . 2) (el-time . 3) t) nil (spy-on 'file-attributes :an...
  spy-on(file-attributes :and-call-fake (closure ((elc-time . 2) (el-time . ...
  buttercup--spy-on-and-call-replacement(file-attributes (lambda (&rest args...
  comp-subr-trampoline-install(file-attributes)
  comp-trampoline-compile(file-attributes)
  error("Cannot find suitable directory for output in `native-comp-eln-load-...
  error: (error "Cannot find suitable directory for output in ‘native-comp-eln-load-path’")

But since it doesn't happen with the upstream tree, we haven't filed a bug.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Possibly defaulting native-compile-target-directory to XDG_CACHE_HOME/...

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Rob Browning <rlb@defaultvalue.org>

>> I can't recall from previous discussions whether that crash would be
>> considered a bug, or whether emacs just requires a writable cache dir
>> now (which is fine if so).

> The latter, AFAIU.

OK, so then until/unless Emacs supports some way to redirect that from
HOME (like XDG_CACHE_HOME or something), then I suppose that means that
an Emacs with native compilation no longer intends to support a
read-only (or missing) HOME.

-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Possibly defaulting native-compile-target-directory to XDG_CACHE_HOME/...

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 17 Dec 2022 11:42:50 -0600
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: Rob Browning <rlb@defaultvalue.org>
> 
> >> I can't recall from previous discussions whether that crash would be
> >> considered a bug, or whether emacs just requires a writable cache dir
> >> now (which is fine if so).
> 
> > The latter, AFAIU.
> 
> OK, so then until/unless Emacs supports some way to redirect that from
> HOME (like XDG_CACHE_HOME or something), then I suppose that means that
> an Emacs with native compilation no longer intends to support a
> read-only (or missing) HOME.

I don't see how this follows from what I said.  There should be a
writable directory, but it doesn't have to be under HOME.  You can
change the value of native-comp-eln-load-path to have there any
directories you like.  We've been through this during the previous
discussion, I believe.

Or what am I missing?

Re: Possibly defaulting native-compile-target-directory to XDG_CACHE_HOME/...

[Rob Browning]

Eli Zaretskii <eliz@gnu.org> writes:

> I don't see how this follows from what I said.  There should be a
> writable directory, but it doesn't have to be under HOME.  You can
> change the value of native-comp-eln-load-path to have there any
> directories you like.  We've been through this during the previous
> discussion, I believe.

I may not have understood where all of that ended up.

Given that there's no way to establish the location via an envt var.
Are you saying that anyone who needs to run emacs with an unwritable
home would need to add one or more command line arguments to every
invocation (i.e. we'd need to do that across all of the relevant debian
testing scripts, etc., some of which may be embedded in upstream
frameworks -- I guess we might also just shadow emacs in the path with a
wrapper script...)?

...and if any of the relevant command line settings don't happen early
enough, then that'd be considered a bug that we'd want to report and see
about fixing?

Oh, and I can finally reproduce that buttercup crash with an upstream
tree.  This should do it:

  # Build and install emacs-28.2 as root or "user1"

  # As "user2" (not root or user1)
  git clone https://github.com/jorgenschaefer/emacs-buttercup.git
  cd emacs-buttercup
  export HOME=/nope
  rm -f *.elc
  emacs -Q -batch -L . -f batch-byte-compile buttercup.el buttercup-compat.el
  emacs -Q -batch -L . -l buttercup -f buttercup-run-discover

The key appears to be that the install must be as some other user.

Thanks
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Possibly defaulting native-compile-target-directory to XDG_CACHE_HOME/...

[Rob Browning]

Rob Browning <rlb@defaultvalue.org> writes:

> ...I guess we might also just shadow emacs in the path with a
> wrapper script...)?

...and if this does seem like the approach y'all would prefer upstream,
then I suppose we'll get started implementing/testing some variation of
that in all the relevant places (see if a set of --evals is sufficient
everywhere).

Thanks again
-- 
Rob Browning
rlb @defaultvalue.org and @debian.org
GPG as of 2011-07-10 E6A9 DA3C C9FD 1FF8 C676 D2C4 C0F0 39E9 ED1B 597A
GPG as of 2002-11-03 14DD 432F AE39 534D B592 F9A0 25C8 D377 8C7E 73A4

Re: Possibly defaulting native-compile-target-directory to XDG_CACHE_HOME/...

[Eli Zaretskii]

> From: Rob Browning <rlb@defaultvalue.org>
> Cc: emacs-devel@gnu.org
> Date: Sat, 17 Dec 2022 14:37:18 -0600
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I don't see how this follows from what I said.  There should be a
> > writable directory, but it doesn't have to be under HOME.  You can
> > change the value of native-comp-eln-load-path to have there any
> > directories you like.  We've been through this during the previous
> > discussion, I believe.
> 
> I may not have understood where all of that ended up.
> 
> Given that there's no way to establish the location via an envt var.
> Are you saying that anyone who needs to run emacs with an unwritable
> home would need to add one or more command line arguments to every
> invocation (i.e. we'd need to do that across all of the relevant debian
> testing scripts, etc., some of which may be embedded in upstream
> frameworks -- I guess we might also just shadow emacs in the path with a
> wrapper script...)?

No.  I'm saying that these users should add a one-liner to their init
file.

> ...and if any of the relevant command line settings don't happen early
> enough, then that'd be considered a bug that we'd want to report and see
> about fixing?

I don't understand this part.  What do you mean by "happen early
enough"?

> Oh, and I can finally reproduce that buttercup crash with an upstream
> tree.  This should do it:
> 
>   # Build and install emacs-28.2 as root or "user1"
> 
>   # As "user2" (not root or user1)
>   git clone https://github.com/jorgenschaefer/emacs-buttercup.git
>   cd emacs-buttercup
>   export HOME=/nope
>   rm -f *.elc
>   emacs -Q -batch -L . -f batch-byte-compile buttercup.el buttercup-compat.el
>   emacs -Q -batch -L . -l buttercup -f buttercup-run-discover
> 
> The key appears to be that the install must be as some other user.

Is there something in emacs-buttercup that makes it necessary for
reproducing this?  Or would any byte-compiled file do?

Anyway, what happens if you try this with Emacs 29?  Emacs 28 is in
the past for us: there will be no more releases from the emacs-28
branch.  And on Emacs 29 we made some changes which I think should
prevent these problems.

Re: Possibly defaulting native-compile-target-directory to XDG_CACHE_HOME/...

[Sean Whitton]

Hello,

On Sun 18 Dec 2022 at 10:48AM +02, Eli Zaretskii wrote:

> Anyway, what happens if you try this with Emacs 29?  Emacs 28 is in
> the past for us: there will be no more releases from the emacs-28
> branch.  And on Emacs 29 we made some changes which I think should
> prevent these problems.

Could you remind me which changes you mean?

As an update, I backported EMACS_INHIBIT_AUTOMATIC_NATIVE_COMPILATION to
Emacs 28 earlier this week, and uploaded that to Debian, and we're now
working to export that variable in all the relevant places.
So, I believe native comp in Debian is finally all sorted :)

-- 
Sean Whitton

Re: Possibly defaulting native-compile-target-directory to XDG_CACHE_HOME/...

[Eli Zaretskii]

> From: Sean Whitton <spwhitton@spwhitton.name>
> Cc: Rob Browning <rlb@defaultvalue.org>,  emacs-devel@gnu.org
> Date: Sat, 24 Dec 2022 16:03:36 -0700
> 
> On Sun 18 Dec 2022 at 10:48AM +02, Eli Zaretskii wrote:
> 
> > Anyway, what happens if you try this with Emacs 29?  Emacs 28 is in
> > the past for us: there will be no more releases from the emacs-28
> > branch.  And on Emacs 29 we made some changes which I think should
> > prevent these problems.
> 
> Could you remind me which changes you mean?

I don't have the pointers at the moment, so I can only point to the
"Native Compilation" section of NEWS and to Git logs.  Sorry.

Re: Possibly defaulting native-compile-target-directory to XDG_CACHE_HOME/...

[Eli Zaretskii]

> Date: Sun, 25 Dec 2022 08:44:54 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: rlb@defaultvalue.org, emacs-devel@gnu.org
> 
> > From: Sean Whitton <spwhitton@spwhitton.name>
> > Cc: Rob Browning <rlb@defaultvalue.org>,  emacs-devel@gnu.org
> > Date: Sat, 24 Dec 2022 16:03:36 -0700
> > 
> > On Sun 18 Dec 2022 at 10:48AM +02, Eli Zaretskii wrote:
> > 
> > > Anyway, what happens if you try this with Emacs 29?  Emacs 28 is in
> > > the past for us: there will be no more releases from the emacs-28
> > > branch.  And on Emacs 29 we made some changes which I think should
> > > prevent these problems.
> > 
> > Could you remind me which changes you mean?
> 
> I don't have the pointers at the moment, so I can only point to the
> "Native Compilation" section of NEWS and to Git logs.  Sorry.

And also the changes that prevent compiling trampolines in async
subprocesses, see commits 1a8015b837 and 5ad5b797f7.