automake's .el support vs. recent loss of byte-compile-dest-file

automake's .el support vs. recent loss of byte-compile-dest-file

[Jim Meyering]

Hello,

In May, this happened:

https://git.savannah.gnu.org/cgit/emacs.git/commit/?id=233cfb0ea93ecdd2b63298be4243059e2e7a91fd

  commit 233cfb0ea93ecdd2b63298be4243059e2e7a91fd
  Author: Glenn Morris <rgm@gnu.org>
  Date:   Sat May 6 18:01:34 2017 -0700

      Remove obsolete method of changing byte-compile-dest-file

      * lisp/emacs-lisp/bytecomp.el (byte-compile-dest-file):
      Define unconditionally.

That broke automake's .el.elc rule, as used in any automake-using
project that distributes an .el file. I noticed this because I want to
make a new release of GNU idutils (which distributes idutils.el). With
emacs built from that commit or newer, "make distcheck" now fails,
because the source is in a read-only directory, and the rule's attempt
to tell the byte-compiler to put the output in the current (writable)
directory is ignored. Hence the write fails, and "make distcheck" fails.

As for the automake-emitted rule, here's the code:

  https://git.savannah.gnu.org/cgit/automake.git/tree/lib/am/lisp.am#n44

I can make it work with this automake change:

-           --eval "(defun byte-compile-dest-file (f) \"$@\")" \
+           --eval "(setq byte-compile-dest-file-function (lambda (_) \"$@\"))" \

But that will fail for those lacking byte-compile-dest-file-function,
which was introduced in 23.1 (2009).

Do you think it's worth supporting pre-23 emacs? If so, what's the best
way to make it work both before and after that May change?

Thanks,
Jim

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Paul Eggert]

Jim Meyering wrote:
> Do you think it's worth supporting pre-23 emacs?

Ha! We just had a thread discussing this, where I wrote:

> currently I would say that Emacs 23.1 is the oldest Emacs we currently need to worry about

For details as to why, please see:

https://lists.gnu.org/r/emacs-devel/2017-11/msg00500.html

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Jim Meyering]

On Wed, Nov 22, 2017 at 8:56 PM, Paul Eggert <eggert@cs.ucla.edu> wrote:
> Jim Meyering wrote:
>>
>> Do you think it's worth supporting pre-23 emacs?
>
>
> Ha! We just had a thread discussing this, where I wrote:
>
>> currently I would say that Emacs 23.1 is the oldest Emacs we currently
>> need to worry about
>
> For details as to why, please see:
>
> https://lists.gnu.org/r/emacs-devel/2017-11/msg00500.html

Thanks.
Good to know the automake fix will be small.

However, bear in mind that once that May change makes it into an emacs
release, automake-generated Makefile.in files will be broken in that
respect.

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Eli Zaretskii]

On November 23, 2017 7:02:07 AM GMT+02:00, Jim Meyering <jim@meyering.net> wrote:
> On Wed, Nov 22, 2017 at 8:56 PM, Paul Eggert <eggert@cs.ucla.edu>
> wrote:
> > Jim Meyering wrote:
> >>
> >> Do you think it's worth supporting pre-23 emacs?
> >
> >
> > Ha! We just had a thread discussing this, where I wrote:
> >
> >> currently I would say that Emacs 23.1 is the oldest Emacs we
> currently
> >> need to worry about
> >
> > For details as to why, please see:
> >
> > https://lists.gnu.org/r/emacs-devel/2017-11/msg00500.html
> 
> Thanks.
> Good to know the automake fix will be small.
> 
> However, bear in mind that once that May change makes it into an emacs
> release, automake-generated Makefile.in files will be broken in that
> respect.

This incompatible change is not in NEWS.  We announced the function
as obsolete in Emacs 23.2, but evidently Automake didn't take notice.
So I'd rather we restored that functionality, perhaps with some
annoying warning to encourage Automake to get their act together
sooner rather than later.  Breaking all the Makefile.in files out there
doesn't sound TRT to me.

Comments?

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Jim Meyering]

On Wed, Nov 22, 2017 at 10:34 PM, Eli Zaretskii <eliz@gnu.org> wrote:
> On November 23, 2017 7:02:07 AM GMT+02:00, Jim Meyering <jim@meyering.net> wrote:
...
>> However, bear in mind that once that May change makes it into an emacs
>> release, automake-generated Makefile.in files will be broken in that
>> respect.
>
> This incompatible change is not in NEWS.  We announced the function
> as obsolete in Emacs 23.2, but evidently Automake didn't take notice.
> So I'd rather we restored that functionality, perhaps with some
> annoying warning to encourage Automake to get their act together
> sooner rather than later.  Breaking all the Makefile.in files out there
> doesn't sound TRT to me.

I've just posted an automake fix:
https://lists.gnu.org/archive/html/automake/2017-11/msg00038.html

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Eli Zaretskii]

> From: Jim Meyering <jim@meyering.net>
> Date: Thu, 23 Nov 2017 08:13:58 -0800
> Cc: emacs-devel <emacs-devel@gnu.org>, Paul Eggert <eggert@cs.ucla.edu>
> 
> > This incompatible change is not in NEWS.  We announced the function
> > as obsolete in Emacs 23.2, but evidently Automake didn't take notice.
> > So I'd rather we restored that functionality, perhaps with some
> > annoying warning to encourage Automake to get their act together
> > sooner rather than later.  Breaking all the Makefile.in files out there
> > doesn't sound TRT to me.
> 
> I've just posted an automake fix:
> https://lists.gnu.org/archive/html/automake/2017-11/msg00038.html

Thanks.  The question that's important to us is, of course, when will
this fixed version be available widely enough for us to consider
removing the support for the old ones.

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Jim Meyering]

On Thu, Nov 23, 2017 at 8:37 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Jim Meyering <jim@meyering.net>
>> Date: Thu, 23 Nov 2017 08:13:58 -0800
>> Cc: emacs-devel <emacs-devel@gnu.org>, Paul Eggert <eggert@cs.ucla.edu>
>>
>> > This incompatible change is not in NEWS.  We announced the function
>> > as obsolete in Emacs 23.2, but evidently Automake didn't take notice.
>> > So I'd rather we restored that functionality, perhaps with some
>> > annoying warning to encourage Automake to get their act together
>> > sooner rather than later.  Breaking all the Makefile.in files out there
>> > doesn't sound TRT to me.
>>
>> I've just posted an automake fix:
>> https://lists.gnu.org/archive/html/automake/2017-11/msg00038.html
>
> Thanks.  The question that's important to us is, of course, when will
> this fixed version be available widely enough for us to consider
> removing the support for the old ones.

I agree. What Emacs does here will influence at least the wording of
the automake NEWS entry for that automake change I mentioned. Should I
call it a bug fix, because this behavior will make it into a release?
Or is it just a work-around for those who happen to be using
pre-release Emacs, and it won't affect anyone using an actual release?

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Eli Zaretskii]

> From: Jim Meyering <jim@meyering.net>
> Date: Sun, 26 Nov 2017 17:17:59 -0800
> Cc: emacs-devel <emacs-devel@gnu.org>, Paul Eggert <eggert@cs.ucla.edu>
> 
> >> I've just posted an automake fix:
> >> https://lists.gnu.org/archive/html/automake/2017-11/msg00038.html
> >
> > Thanks.  The question that's important to us is, of course, when will
> > this fixed version be available widely enough for us to consider
> > removing the support for the old ones.
> 
> I agree. What Emacs does here will influence at least the wording of
> the automake NEWS entry for that automake change I mentioned. Should I
> call it a bug fix, because this behavior will make it into a release?
> Or is it just a work-around for those who happen to be using
> pre-release Emacs, and it won't affect anyone using an actual release?

My intention, unless I hear objections, is to resurrect the removed
feature on the emacs-26 branch, but add an annoying warning there, so
that any packages and maintainers who didn't notice the deprecation
will do so sooner rather than later.  But the feature will remain
removed on the master branch, which will eventually become Emacs 27.
So I think your change is a bug fix, and its purpose is to make
Automake future-proof against imminent changes in Emacs.  It is also
for those who already use development snapshots of Emacs 27, because
the function will remain removed there.

Thanks.

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Jim Meyering]

On Mon, Nov 27, 2017 at 8:36 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Jim Meyering <jim@meyering.net>
>> Date: Sun, 26 Nov 2017 17:17:59 -0800
>> Cc: emacs-devel <emacs-devel@gnu.org>, Paul Eggert <eggert@cs.ucla.edu>
>>
>> >> I've just posted an automake fix:
>> >> https://lists.gnu.org/archive/html/automake/2017-11/msg00038.html
>> >
>> > Thanks.  The question that's important to us is, of course, when will
>> > this fixed version be available widely enough for us to consider
>> > removing the support for the old ones.
>>
>> I agree. What Emacs does here will influence at least the wording of
>> the automake NEWS entry for that automake change I mentioned. Should I
>> call it a bug fix, because this behavior will make it into a release?
>> Or is it just a work-around for those who happen to be using
>> pre-release Emacs, and it won't affect anyone using an actual release?
>
> My intention, unless I hear objections, is to resurrect the removed
> feature on the emacs-26 branch, but add an annoying warning there, so
> that any packages and maintainers who didn't notice the deprecation
> will do so sooner rather than later.  But the feature will remain
> removed on the master branch, which will eventually become Emacs 27.
> So I think your change is a bug fix, and its purpose is to make
> Automake future-proof against imminent changes in Emacs.  It is also
> for those who already use development snapshots of Emacs 27, because
> the function will remain removed there.

Thanks.
Another possible fix proposed by Glenn was to use -l bytecomp and to
continue to override byte-compile-dest-file.
However, I have hit a new seemingly unrelated snag: subdirs. This now
fails, where the argument to byte-compile-file is a dot-relative name
in a subdirectory:

$ (export TMPDIR=/tmp; mkdir sub && : > sub/x.el && emacs --batch -l
bytecomp --eval '(defun byte-compile-dest-file (f) "sub/x.elc")'
--eval "(unless (byte-compile-file \"sub/x.el\") (kill-emacs 1))" )
Creating file with prefix: No such file or directory, /tmp/sub/x.elc

While it worked fine with emacs built from a commit on master from
2017-07-29, some time between then and Aug 11, things changed so that
byte-compiling such a source file name requires the relative directory
name first be created in $TMPDIR.

Was that deliberate?

For reference, this was exposed because an automake test exercises
that code path: I ran it individually via this: make check
TESTS='t/lisp-subdir.sh'

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Paul Eggert]

Jim Meyering wrote:
> things changed so that
> byte-compiling such a source file name requires the relative directory
> name first be created in $TMPDIR.
> 
> Was that deliberate?

I can't imagine that it was, no.

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Eli Zaretskii]

> From: Jim Meyering <jim@meyering.net>
> Date: Tue, 28 Nov 2017 19:35:04 -0800
> Cc: emacs-devel <emacs-devel@gnu.org>, Paul Eggert <eggert@cs.ucla.edu>
> 
> $ (export TMPDIR=/tmp; mkdir sub && : > sub/x.el && emacs --batch -l
> bytecomp --eval '(defun byte-compile-dest-file (f) "sub/x.elc")'
> --eval "(unless (byte-compile-file \"sub/x.el\") (kill-emacs 1))" )
> Creating file with prefix: No such file or directory, /tmp/sub/x.elc

Does the patch below fix this?

diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
index e4d8a20..fa6b789 100644
--- a/lisp/emacs-lisp/bytecomp.el
+++ b/lisp/emacs-lisp/bytecomp.el
@@ -1924,7 +1924,8 @@ byte-compile-file
 		       ;; process is trying to load target-file (eg in a
 		       ;; parallel bootstrap), it does not risk getting a
 		       ;; half-finished file.  (Bug#4196)
-		       (tempfile (make-temp-file target-file))
+		       (tempfile
+                        (make-temp-file (file-name-nondirectory target-file)))
 		       (default-modes (default-file-modes))
 		       (temp-modes (logand default-modes #o600))
 		       (desired-modes (logand default-modes #o666))

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Jim Meyering]

On Wed, Nov 29, 2017 at 10:23 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> From: Jim Meyering <jim@meyering.net>
>> Date: Tue, 28 Nov 2017 19:35:04 -0800
>> Cc: emacs-devel <emacs-devel@gnu.org>, Paul Eggert <eggert@cs.ucla.edu>
>>
>> $ (export TMPDIR=/tmp; mkdir sub && : > sub/x.el && emacs --batch -l
>> bytecomp --eval '(defun byte-compile-dest-file (f) "sub/x.elc")'
>> --eval "(unless (byte-compile-file \"sub/x.el\") (kill-emacs 1))" )
>> Creating file with prefix: No such file or directory, /tmp/sub/x.elc
>
> Does the patch below fix this?
>
> diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
> index e4d8a20..fa6b789 100644
> --- a/lisp/emacs-lisp/bytecomp.el
> +++ b/lisp/emacs-lisp/bytecomp.el
> @@ -1924,7 +1924,8 @@ byte-compile-file
>                        ;; process is trying to load target-file (eg in a
>                        ;; parallel bootstrap), it does not risk getting a
>                        ;; half-finished file.  (Bug#4196)
> -                      (tempfile (make-temp-file target-file))
> +                      (tempfile
> +                        (make-temp-file (file-name-nondirectory target-file)))
>                        (default-modes (default-file-modes))
>                        (temp-modes (logand default-modes #o600))
>                        (desired-modes (logand default-modes #o666))

Hi Eli,
Yes, with emacs built from latest master plus your patch, running
automake's lisp-related tests now (with my in-preparation patch)
succeeds. I.e., these passed:

  make check TESTS="$(echo t/lisp*.sh)"

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Eli Zaretskii]

> From: Jim Meyering <jim@meyering.net>
> Date: Wed, 29 Nov 2017 18:28:02 -0800
> Cc: emacs-devel <emacs-devel@gnu.org>, Paul Eggert <eggert@cs.ucla.edu>
> 
> > diff --git a/lisp/emacs-lisp/bytecomp.el b/lisp/emacs-lisp/bytecomp.el
> > index e4d8a20..fa6b789 100644
> > --- a/lisp/emacs-lisp/bytecomp.el
> > +++ b/lisp/emacs-lisp/bytecomp.el
> > @@ -1924,7 +1924,8 @@ byte-compile-file
> >                        ;; process is trying to load target-file (eg in a
> >                        ;; parallel bootstrap), it does not risk getting a
> >                        ;; half-finished file.  (Bug#4196)
> > -                      (tempfile (make-temp-file target-file))
> > +                      (tempfile
> > +                        (make-temp-file (file-name-nondirectory target-file)))
> >                        (default-modes (default-file-modes))
> >                        (temp-modes (logand default-modes #o600))
> >                        (desired-modes (logand default-modes #o666))
> 
> Hi Eli,
> Yes, with emacs built from latest master plus your patch, running
> automake's lisp-related tests now (with my in-preparation patch)
> succeeds. I.e., these passed:
> 
>   make check TESTS="$(echo t/lisp*.sh)"

Thanks, pushed to the release branch.

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Eli Zaretskii]

> Date: Mon, 27 Nov 2017 18:36:19 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: eggert@cs.ucla.edu, emacs-devel@gnu.org
> 
> My intention, unless I hear objections, is to resurrect the removed
> feature on the emacs-26 branch, but add an annoying warning there, so
> that any packages and maintainers who didn't notice the deprecation
> will do so sooner rather than later.  But the feature will remain
> removed on the master branch, which will eventually become Emacs 27.

I see Glenn already did that (thanks!).

Glenn, did you intentionally leave this deprecated feature available
on master?

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Eli Zaretskii]

> Date: Sat, 02 Dec 2017 13:37:04 +0200
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: eggert@cs.ucla.edu, jim@meyering.net, emacs-devel@gnu.org
> 
> > My intention, unless I hear objections, is to resurrect the removed
> > feature on the emacs-26 branch, but add an annoying warning there, so
> > that any packages and maintainers who didn't notice the deprecation
> > will do so sooner rather than later.  But the feature will remain
> > removed on the master branch, which will eventually become Emacs 27.
> 
> I see Glenn already did that (thanks!).

Jim, can you double-check that the original Automake's "make
distcheck" works with the current emacs-26 branch?  TIA.

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Glenn Morris]

Eli Zaretskii wrote:

> did you intentionally leave this deprecated feature available on
> master?

It will be years before the default automake in some G/L distributions
gets the relevant update.

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Eli Zaretskii]

> From: Glenn Morris <rgm@gnu.org>
> Date: Sun, 03 Dec 2017 22:18:01 -0500
> Cc: eggert@cs.ucla.edu, jim@meyering.net, emacs-devel@gnu.org
> 
> Eli Zaretskii wrote:
> 
> > did you intentionally leave this deprecated feature available on
> > master?
> 
> It will be years before the default automake in some G/L distributions
> gets the relevant update.

I see, thanks.

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Jim Meyering]

On Sat, Dec 2, 2017 at 3:57 AM, Eli Zaretskii <eliz@gnu.org> wrote:
>> Date: Sat, 02 Dec 2017 13:37:04 +0200
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: eggert@cs.ucla.edu, jim@meyering.net, emacs-devel@gnu.org
>>
>> > My intention, unless I hear objections, is to resurrect the removed
>> > feature on the emacs-26 branch, but add an annoying warning there, so
>> > that any packages and maintainers who didn't notice the deprecation
>> > will do so sooner rather than later.  But the feature will remain
>> > removed on the master branch, which will eventually become Emacs 27.
>>
>> I see Glenn already did that (thanks!).
>
> Jim, can you double-check that the original Automake's "make
> distcheck" works with the current emacs-26 branch?  TIA.

Hi Eli,

I built emacs-26, made it first in my path, then confirmed that
Makefiles generated by stock fedora-27 automake
(automake-1.15.1-2.fc27) work with "make distcheck" on an
.el-distributing package (idutils).

Re: automake's .el support vs. recent loss of byte-compile-dest-file

[Eli Zaretskii]

> From: Jim Meyering <jim@meyering.net>
> Date: Sat, 16 Dec 2017 16:34:08 -0800
> Cc: Glenn Morris <rgm@gnu.org>, Paul Eggert <eggert@cs.ucla.edu>, emacs-devel <emacs-devel@gnu.org>
> 
> >> > My intention, unless I hear objections, is to resurrect the removed
> >> > feature on the emacs-26 branch, but add an annoying warning there, so
> >> > that any packages and maintainers who didn't notice the deprecation
> >> > will do so sooner rather than later.  But the feature will remain
> >> > removed on the master branch, which will eventually become Emacs 27.
> >>
> >> I see Glenn already did that (thanks!).
> >
> > Jim, can you double-check that the original Automake's "make
> > distcheck" works with the current emacs-26 branch?  TIA.
> 
> Hi Eli,
> 
> I built emacs-26, made it first in my path, then confirmed that
> Makefiles generated by stock fedora-27 automake
> (automake-1.15.1-2.fc27) work with "make distcheck" on an
> .el-distributing package (idutils).

Great, thanks for testing.  So I guess this issue can be closed now.