The emacs-28 release branch has been created

The emacs-28 release branch has been created

[Eli Zaretskii]

I've cut the emacs-28 branch, in preparation for the pretest of the
upcoming Emacs 28.1.

I encourage people who track the Emacs development to please switch to
this branch, build it and use it, so that any problems with it could
be reported sooner rather than later.

Barring any unexpected calamities, the first pretest of Emacs 28 will
be produced from this branch in a few weeks' time.

Windows Binaries Release: was The emacs-28 release branch

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

> I've cut the emacs-28 branch, in preparation for the pretest of the
> upcoming Emacs 28.1.
>
> I encourage people who track the Emacs development to please switch to
> this branch, build it and use it, so that any problems with it could
> be reported sooner rather than later.
>
> Barring any unexpected calamities, the first pretest of Emacs 28 will
> be produced from this branch in a few weeks' time.


As the release of Emacs-28 is soon, I would like to look for someone to
take over the role of producing windows releases.

I've done this job for quite a while now, since Emacs-25, when I used to
build the releases on windows partition that I had saved on an machine I
use for running Kodi in my living room. Since then, it has gone virtual,
the process has become mostly automatic, dependencies have been included
by default and there is an exe installer.

It is not a huge amount of work to just roll the releases but especially
after the last year, it's a more than I have time for.

In addition, there are a number of things that could be added: an msi or
msix installer, support for native comp. I cannot see having the time
for this into the foreseable future.

If anyone is interested, let me know. Happy to take anyone through the
process and provide as much support as I can.

Phil

Re: Windows Binaries Release: was The emacs-28 release branch

[Po Lu]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> As the release of Emacs-28 is soon, I would like to look for someone to
> take over the role of producing windows releases.
>
> I've done this job for quite a while now, since Emacs-25, when I used to
> build the releases on windows partition that I had saved on an machine I
> use for running Kodi in my living room. Since then, it has gone virtual,
> the process has become mostly automatic, dependencies have been included
> by default and there is an exe installer.
>
> It is not a huge amount of work to just roll the releases but especially
> after the last year, it's a more than I have time for.
>
> In addition, there are a number of things that could be added: an msi or
> msix installer, support for native comp. I cannot see having the time
> for this into the foreseable future.
>
> If anyone is interested, let me know. Happy to take anyone through the
> process and provide as much support as I can.
>
> Phil

Thanks for your effort.  One question though: can the builds be made
with Wine or ReactOS and an entirely free toolchain?

Re: Windows Binaries Release: was The emacs-28 release branch

[Phillip Lord]

Po Lu <luangruo@yahoo.com> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> As the release of Emacs-28 is soon, I would like to look for someone to
>> take over the role of producing windows releases.
>>
>> I've done this job for quite a while now, since Emacs-25, when I used to
>> build the releases on windows partition that I had saved on an machine I
>> use for running Kodi in my living room. Since then, it has gone virtual,
>> the process has become mostly automatic, dependencies have been included
>> by default and there is an exe installer.
>>
>> It is not a huge amount of work to just roll the releases but especially
>> after the last year, it's a more than I have time for.
>>
>> In addition, there are a number of things that could be added: an msi or
>> msix installer, support for native comp. I cannot see having the time
>> for this into the foreseable future.
>>
>> If anyone is interested, let me know. Happy to take anyone through the
>> process and provide as much support as I can.
>>
>> Phil
>
> Thanks for your effort.  One question though: can the builds be made
> with Wine or ReactOS and an entirely free toolchain?


No, not to my knowledge. Emacs does not cross-compile. It has to be
built on Windows, using msys2 and mingw64. The toolchain is free, the
operating system is not.

Phil

Re: Windows Binaries Release: was The emacs-28 release branch

[Konstantin Kharlamov]

On Mon, 2021-10-04 at 20:04 +0100, Phillip Lord wrote:
> Po Lu <luangruo@yahoo.com> writes:
> 
> > Phillip Lord <phillip.lord@russet.org.uk> writes:
> > 
> > > As the release of Emacs-28 is soon, I would like to look for someone to
> > > take over the role of producing windows releases.
> > > 
> > > I've done this job for quite a while now, since Emacs-25, when I used to
> > > build the releases on windows partition that I had saved on an machine I
> > > use for running Kodi in my living room. Since then, it has gone virtual,
> > > the process has become mostly automatic, dependencies have been included
> > > by default and there is an exe installer.
> > > 
> > > It is not a huge amount of work to just roll the releases but especially
> > > after the last year, it's a more than I have time for.
> > > 
> > > In addition, there are a number of things that could be added: an msi or
> > > msix installer, support for native comp. I cannot see having the time
> > > for this into the foreseable future.
> > > 
> > > If anyone is interested, let me know. Happy to take anyone through the
> > > process and provide as much support as I can.
> > > 
> > > Phil
> > 
> > Thanks for your effort.  One question though: can the builds be made
> > with Wine or ReactOS and an entirely free toolchain?
> 
> 
> No, not to my knowledge. Emacs does not cross-compile. It has to be
> built on Windows, using msys2 and mingw64. The toolchain is free, the
> operating system is not.

FWIW, neither of them counts as cross-compilation. A Windows app run under WINE sees the usual Windows environment, except it is actually simulated. Running msys2 and mingw64 under WINE would count as "native compilation".

Re: Windows Binaries Release: was The emacs-28 release branch

[Po Lu]

Konstantin Kharlamov <hi-angel@yandex.ru> writes:

>> No, not to my knowledge. Emacs does not cross-compile. It has to be
>> built on Windows, using msys2 and mingw64. The toolchain is free, the
>> operating system is not.
>
> FWIW, neither of them counts as cross-compilation. A Windows app run
> under WINE sees the usual Windows environment, except it is actually
> simulated. Running msys2 and mingw64 under WINE would count as "native
> compilation".

BTW, I wasn't able to build Emacs under msys2 and Wine.  temacs
experiences a segmentation fault during dumping (with the portable dumper).

Re: Windows Binaries Release: was The emacs-28 release branch

[Konstantin Kharlamov]

On Sun, 2021-10-17 at 08:43 +0800, Po Lu wrote:
> Konstantin Kharlamov <hi-angel@yandex.ru> writes:
> 
> > > No, not to my knowledge. Emacs does not cross-compile. It has to be
> > > built on Windows, using msys2 and mingw64. The toolchain is free, the
> > > operating system is not.
> > 
> > FWIW, neither of them counts as cross-compilation. A Windows app run
> > under WINE sees the usual Windows environment, except it is actually
> > simulated. Running msys2 and mingw64 under WINE would count as "native
> > compilation".
> 
> BTW, I wasn't able to build Emacs under msys2 and Wine.  temacs
> experiences a segmentation fault during dumping (with the portable dumper).

Yeah, WINE isn't perfect. But it is actively developed, so upon encountering such problems it might help to make sure you run latest released version (they release a new one every two weeks).

Re: Windows Binaries Release: was The emacs-28 release branch

[Phillip Lord]

Konstantin Kharlamov <hi-angel@yandex.ru> writes:

>> > 
>> > Thanks for your effort.  One question though: can the builds be made
>> > with Wine or ReactOS and an entirely free toolchain?
>> 
>> 
>> No, not to my knowledge. Emacs does not cross-compile. It has to be
>> built on Windows, using msys2 and mingw64. The toolchain is free, the
>> operating system is not.
>
> FWIW, neither of them counts as cross-compilation. A Windows app run
> under WINE sees the usual Windows environment, except it is actually
> simulated. Running msys2 and mingw64 under WINE would count as "native
> compilation".

Indeed, this would seem correct to me.

In which case, I would revise my answer from "I know that it wouldn't
work" to "I don't know that it does (or does not) work". Emacs mostly
does run under wine and I sometimes test it there.

Phil

Re: Windows Binaries Release: was The emacs-28 release branch

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 1877 bytes --]

Hi Pill!

On Sat, Oct 2, 2021, 12:10 Phillip Lord <phillip.lord@russet.org.uk> wrote:

>
> Eli Zaretskii <eliz@gnu.org> writes:
>
> > I've cut the emacs-28 branch, in preparation for the pretest of the
> > upcoming Emacs 28.1.
> >
> > I encourage people who track the Emacs development to please switch to
> > this branch, build it and use it, so that any problems with it could
> > be reported sooner rather than later.
> >
> > Barring any unexpected calamities, the first pretest of Emacs 28 will
> > be produced from this branch in a few weeks' time.
>
>
> As the release of Emacs-28 is soon, I would like to look for someone to
> take over the role of producing windows releases.
>
> I've done this job for quite a while now, since Emacs-25, when I used to
> build the releases on windows partition that I had saved on an machine I
> use for running Kodi in my living room. Since then, it has gone virtual,
> the process has become mostly automatic, dependencies have been included
> by default and there is an exe installer.
>
> It is not a huge amount of work to just roll the releases but especially
> after the last year, it's a more than I have time for.
>
> In addition, there are a number of things that could be added: an msi or
> msix installer, support for native comp. I cannot see having the time
> for this into the foreseable future.
>
> If anyone is interested, let me know.


I could be interested!

> Happy to take anyone through the
> process and provide as much support as I can.
>

At least to learn the requirements and expectations, I'm also eager to
understand whether you expect you might be available to a short video call,
and my recording that, either for postarity or more limited use per
preference/capture quality.

I think we would be fine using the EmacsConf BBB for this.  CC a couple
other conference volunteers, for awareness.


Phil
>
>

[-- Attachment #2: Type: text/html, Size: 3204 bytes --]

Re: Windows Binaries Release: was The emacs-28 release branch

[Phillip Lord]

Corwin Brust <corwin@bru.st> writes:

>> As the release of Emacs-28 is soon, I would like to look for someone to
>> take over the role of producing windows releases.
>>
>> I've done this job for quite a while now, since Emacs-25, when I used to
>> build the releases on windows partition that I had saved on an machine I
>> use for running Kodi in my living room. Since then, it has gone virtual,
>> the process has become mostly automatic, dependencies have been included
>> by default and there is an exe installer.
>>
>> It is not a huge amount of work to just roll the releases but especially
>> after the last year, it's a more than I have time for.
>>
>> In addition, there are a number of things that could be added: an msi or
>> msix installer, support for native comp. I cannot see having the time
>> for this into the foreseable future.
>>
>> If anyone is interested, let me know.
>
>
> I could be interested!

That's good!


>> Happy to take anyone through the
>> process and provide as much support as I can.
>>
>
> At least to learn the requirements and expectations, I'm also eager to
> understand whether you expect you might be available to a short video call,
> and my recording that, either for postarity or more limited use per
> preference/capture quality.
>
> I think we would be fine using the EmacsConf BBB for this.  CC a couple
> other conference volunteers, for awareness.



Sure. I'll contact you off-list.

Phil

Re: Windows Binaries Release: was The emacs-28 release branch

[H. Dieter Wilhelm]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Corwin Brust <corwin@bru.st> writes:
>
>>> As the release of Emacs-28 is soon, I would like to look for someone to
>>> take over the role of producing windows releases.
>>>
>>> If anyone is interested, let me know.
>>
>> I could be interested!

Me too

Would you mind to send me your recipes or shall I contact you
beforehand?

Thank you

      Dieter

Re: Windows Binaries Release: was The emacs-28 release branch

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 790 bytes --]

Hi Dieter!


On Sat, Oct 16, 2021, 05:04 H. Dieter Wilhelm <dieter@duenenhof-wilhelm.de>
wrote:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
> > Corwin Brust <corwin@bru.st> writes:
> >
> >>> As the release of Emacs-28 is soon, I would like to look for someone to
> >>> take over the role of producing windows releases.
> >>>
> >>> If anyone is interested, let me know.
> >>
> >> I could be interested!
>
> Me too
>

Yay!


> Would you mind to send me your recipes or shall I contact you
> beforehand?
>


Phil and I have been trying to connect to walk through the process, also.
I would welcome you, given we can eventually connect and  you are
interested.

I've suggested using the emacsconf BBB so we can easily make an informal
recording.


> Thank you
>
>       Dieter
>
>
>

[-- Attachment #2: Type: text/html, Size: 2050 bytes --]

Re: Windows Binaries Release: was The emacs-28 release branch

[H. Dieter Wilhelm]

Hi Corwin

Corwin Brust <corwin@bru.st> writes:

> Phil and I have been trying to connect to walk through the process,
> also.  I would welcome you, given we can eventually connect and you
> are interested.

Did you already get some files and do you know whether Phil has setup a
repository for the process?

> I've suggested using the emacsconf BBB so we can easily make an
> informal recording.

This is a very good idea for documentation purposes, please send me the
invitation or maybe some dates if you haven't set a deadline yet.
Preferably on Tuesdays or Thursdays, since then I'm usually in the home
office or maybe in the evening or at the weekend?  Oh wait, from which
time zone are you coming?

Thanks
        Dieter

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Windows Binaries Release: was The emacs-28 release branch

[Phillip Lord]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> Hi Corwin
>
> Corwin Brust <corwin@bru.st> writes:
>
>> Phil and I have been trying to connect to walk through the process,
>> also.  I would welcome you, given we can eventually connect and you
>> are interested.
>
> Did you already get some files and do you know whether Phil has setup a
> repository for the process?

All of the files needed to build the windows binaries are in the main
repository, specifically in admin/nt/dist-build.

I use a few other launch scripts which the python or shell script. This
is just so that I could build release or snapshot builds from the same
git checkout (or rather multiple worktrees of the same git repo) with
ease.

After that, you need a windows box, with a particular directory
structure. This is documented here:

admin/nt/dist-build/README-scripts

but it's a bit baroque and may not be completely accurate as I am
probably the only person to have run this stuff since Emacs-25.

Phil

Re: Windows Binaries Release: was The emacs-28 release branch

[H. Dieter Wilhelm]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> All of the files needed to build the windows binaries are in the main
> repository, specifically in admin/nt/dist-build.

> admin/nt/dist-build/README-scripts
>
> but it's a bit baroque and may not be completely accurate as I am
> probably the only person to have run this stuff since Emacs-25.

Emacs-25.1 5 years ago, don't worry.  :-) Hope (bloody beginner) with
your help and together with Corwin we can get it going.

Thanks a lot for the pointers.  

       Dieter

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Windows Binaries Release: was The emacs-28 release branch

[H. Dieter Wilhelm]

Hello Phil

Phillip Lord <phillip.lord@russet.org.uk> writes:

> All of the files needed to build the windows binaries are in the main
> repository, specifically in admin/nt/dist-build.
>
> I use a few other launch scripts which the python or shell script. This
> is just so that I could build release or snapshot builds from the same
> git checkout (or rather multiple worktrees of the same git repo) with
> ease.

Please tell me where are the "other launch scripts" located?  I don't
understand, among other things, where variables like $VERSION in
nt/dist-build/build-zips.sh are defined.

...
function git_up {
    echo [build] Making git worktree for Emacs $VERSION
    cd $REPO_DIR/emacs-$MAJOR_VERSION
    git pull
    git worktree add ../$BRANCH $BRANCH

    cd ../$BRANCH
    ./autogen.sh
}
...


  Dieter
  
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Windows Binaries Release: was The emacs-28 release branch

[Phillip Lord]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

>> All of the files needed to build the windows binaries are in the
>> main
>> repository, specifically in admin/nt/dist-build.
>>
>> I use a few other launch scripts which the python or shell
>> script. This
>> is just so that I could build release or snapshot builds from the
>> same
>> git checkout (or rather multiple worktrees of the same git repo)
>> with
>> ease.
>
> Please tell me where are the "other launch scripts" located?


They aren't anywhere accessible, because they are a bit specific to my
use.

But for instance, I use one called "build-emacs27.sh" which essentially
looks like:

rm -rf ~/emacs-build/build
rm -rf ~/emacs-build/install

cd emacs-27/admin/nt/dist-build


./build-zips.sh -g
./build-zips.sh


As you can see, it assumes a fairly rigid directory structure which is
defined in admin/nt/dist-build/README-scripts. I keep this script in the
~/emacs-build/git directory below which is ~/emacs-build/git/master
which is the main checkout.

The first part "./build-zips.sh -g" makes a new worktree with the
current release. The second part actually builds the zips.

In my version, I then use rsync to push them from the windows server on
which they are built to a linux box somewhere else. Then I upload them
to the GNU ftp server from there. You don't need to do that but it means
I don't have to set up GPG signing on the windows machine which was a
bit easier.


Then I have "build-27-deps.sh" which again is just a launch script.


set -o errexit
if test -f emacs-src; then
    rsync -r emacs-src emacs-src-cache
fi
rm -rf i686 x86_64 emacs-src
../git/emacs-27/admin/nt/dist-build/build-dep-zips.py -s 2>&1 | tee build-deps.log
cp emacs-27* ~/emacs-upload




> I don't understand, among other things, where variables like $VERSION
> in nt/dist-build/build-zips.sh are defined.


Yeah, this is a little bit hairy I am afraid, because the number of use
cases I wanted to support increased over time. Unless you set it
explicitly (which you don't normally need to do) VERSION is set by
hacking it out of configure.ac in this part of the script.


## ACTUAL_VERSION is the version declared by emacs
if [ -z $ACTUAL_VERSION ];
then
    ACTUAL_VERSION=`
  sed -n 's/^AC_INIT(GNU Emacs,[	 ]*\([^	 ,)]*\).*/\1/p' < ../../../configure.ac
`
fi


I can't remember when I used explicit versions. Probably if you want to
build a snapshot from both from the "master" branch and from the release
branch. You might want to do that, or you might not. I don't know how
many people use the snapshot builds, although they were a useful way of
getting testing.

The documentation on build-zips.sh and build-deps.py is poor (sorry!). I
will try and update it, and give some explanation for some of the fairly
obscure variable names (I can't for the life of me remember what the
"OF" in "OF_VERSION" stands for).

Phil

Re: Windows Binaries Release: was The emacs-28 release branch

[H. Dieter Wilhelm]

Hello Phil

Phillip Lord <phillip.lord@russet.org.uk> writes:

> They aren't anywhere accessible, because they are a bit specific to my
> use.
>
> But for instance, I use one called "build-emacs27.sh" which essentially
> looks like:
>
> rm -rf ~/emacs-build/build
> rm -rf ~/emacs-build/install
>
> cd emacs-27/admin/nt/dist-build
>
>
> ./build-zips.sh -g
> ./build-zips.sh
>

I created such a script but my problem is that chmod seems not to work
for me!

   chmod u+x build-zips.sh

doesn't change anything, so I can't run the scripts.  Do you know this
problem?


> Then I have "build-27-deps.sh" which again is just a launch script.
>
>
> set -o errexit
> if test -f emacs-src; then
>     rsync -r emacs-src emacs-src-cache
> fi
> rm -rf i686 x86_64 emacs-src
> ../git/emacs-27/admin/nt/dist-build/build-dep-zips.py -s 2>&1 | tee build-deps.log
> cp emacs-27* ~/emacs-upload

I don't get here the "emacs-src" and "emacs-src-cache" folders, they
aren't mention in README-scripts.  Do I have to create them by hand?

So I tried to run build-dep.zips.py -s by hand for (a snapshot) but I
haven't had luck:

Please have a look at the end and tell me which MSYS2 package contains
/mingw64/bin/ntldd?  I don't see it in the system and in pacman

Traceback (most recent call last):
  File "/c/Users/uidg1626/emacs-build/deps/../git/master/admin/nt/dist-build/build-dep-zips.py", line 267, in <module>
    gather_deps()
  File "/c/Users/uidg1626/emacs-build/deps/../git/master/admin/nt/dist-build/build-dep-zips.py", line 71, in gather_deps
    for dep in full_dll_dependency():
  File "/c/Users/uidg1626/emacs-build/deps/../git/master/admin/nt/dist-build/build-dep-zips.py", line 83, in full_dll_dependency
    deps = [dll_dependency(dep) for dep in DLL_REQ]
  File "/c/Users/uidg1626/emacs-build/deps/../git/master/admin/nt/dist-build/build-dep-zips.py", line 83, in <listcomp>
    deps = [dll_dependency(dep) for dep in DLL_REQ]
  File "/c/Users/uidg1626/emacs-build/deps/../git/master/admin/nt/dist-build/build-dep-zips.py", line 88, in dll_dependency
    output = check_output(["/mingw64/bin/ntldd", "--recursive",
  File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/mingw64/bin/ntldd' 


Thank you for your assistance

       Dieter

Re: Windows Binaries Release: was The emacs-28 release branch

[Óscar Fuentes]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> Hello Phil
>
> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> They aren't anywhere accessible, because they are a bit specific to my
>> use.
>>
>> But for instance, I use one called "build-emacs27.sh" which essentially
>> looks like:
>>
>> rm -rf ~/emacs-build/build
>> rm -rf ~/emacs-build/install
>>
>> cd emacs-27/admin/nt/dist-build
>>
>>
>> ./build-zips.sh -g
>> ./build-zips.sh
>>
>
> I created such a script but my problem is that chmod seems not to work
> for me!
>
>    chmod u+x build-zips.sh
>
> doesn't change anything, so I can't run the scripts.  Do you know this
> problem?

I don't know the context, but I'll chime anyway...

Are you executing the script in MS Windows? There is no "execute" bit
for script files. Try

bash build-zips.sh

Eshell requires execute permission on Win10, was Re: Windows Binaries Release: was The emacs-28 release branch

[H. Dieter Wilhelm]

Óscar Fuentes <ofv@wanadoo.es> writes:

> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

>> I created such a script but my problem is that chmod seems not to work
>> for me!
>>
>>    chmod u+x build-zips.sh
>>
>> doesn't change anything, so I can't run the scripts.  Do you know this
>> problem?
>
> I don't know the context, but I'll chime anyway...

Thanks for asking about the context, sorry!  Above I used *eshell* to
execute a script on Windows10.

I'm on MSYS2 and emacs-28

~/scripts $ ./build-28-deps.sh 
./build-28-deps.sh: not an executable file
~/scripts $ 

Is this a bug of eshell (under Windows)?

> Are you executing the script in MS Windows? There is no "execute" bit
> for script files. Try
>
> bash build-zips.sh

You're right, when I execute the same scripts under the MSYS2 shell it
works (this shell doesn't care about file permissions)!

Thank you

      Dieter

Re: Eshell requires execute permission on Win10, was Re: Windows Binaries Release: was The emacs-28 release branch

[Óscar Fuentes]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
>
>>> I created such a script but my problem is that chmod seems not to work
>>> for me!
>>>
>>>    chmod u+x build-zips.sh
>>>
>>> doesn't change anything, so I can't run the scripts.  Do you know this
>>> problem?
>>
>> I don't know the context, but I'll chime anyway...
>
> Thanks for asking about the context, sorry!  Above I used *eshell* to
> execute a script on Windows10.
>
> I'm on MSYS2 and emacs-28
>
> ~/scripts $ ./build-28-deps.sh 
> ./build-28-deps.sh: not an executable file
> ~/scripts $ 
>
> Is this a bug of eshell (under Windows)?

No, I don't think so. Eshell knows about executables, possibly with some
platform-specific nuances, and a text file that happens to contain a
bash script is not an executable on Windows, you need the POSIX
emulation layer and the added hacks provided by MSYS2 or Cygwin.

I wouldn't use Eshell for doing work that depends on the POSIX emulation
provided by MSYS2 (too many incompatible things involved!) Better use a
proper MSYS2 console. Make sure that the console is the correct one for
the architecture you are targeting. That is, depending on whether you
are building a 32 bits or 64 bits Emacs, start the MSYS2 console with
the shortcut for Mingw32 or Mingw64, respectively (I'm not sure how
those shortcuts are named nowadays, but it should be obvious.)

MinGW Sources, was: Windows Binaries Release

[H. Dieter Wilhelm]

Hello Óscar

in d:/Emacs/emacs-28/admin/nt/dist-build/build-dep-zips.py there is 

  SRC_REPO="https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources".

But it seems that the Sources folder vanished (for the moment?)!  Do
you, or anybody else, know where else sources of MinGW are?

Thank you

      Dieter

Re: MinGW Sources, was: Windows Binaries Release

[Óscar Fuentes]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> in d:/Emacs/emacs-28/admin/nt/dist-build/build-dep-zips.py there is 
>
>   SRC_REPO="https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources".
>
> But it seems that the Sources folder vanished (for the moment?)!  Do
> you, or anybody else, know where else sources of MinGW are?

Try

https://repo.msys2.org/mingw/sources/

That directory contains tarballs with the exact sources used to build
their respective binary packages.

Re: MinGW Sources, was: Windows Binaries Release

[H. Dieter Wilhelm]

Óscar Fuentes <ofv@wanadoo.es> writes:

> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
>
>> in d:/Emacs/emacs-28/admin/nt/dist-build/build-dep-zips.py there is 
>>
>>   SRC_REPO="https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources".
>>
>> But it seems that the Sources folder vanished (for the moment?)!  Do
>> you, or anybody else, know where else sources of MinGW are?
>
> Try
>
> https://repo.msys2.org/mingw/sources/

Yes, thank you!

modified   admin/nt/dist-build/build-dep-zips.py
@@ -121,7 +121,8 @@ def ntldd_munge(out):
 
 ## Currently no packages seem to require this!
 ARCH_PKGS=[]
-SRC_REPO="https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources"
+## SRC_REPO="https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources"
+SRC_REPO="https://repo.msys2.org/mingw/sources"
 
 
 def immediate_deps(pkg):
@@ -167,7 +168,7 @@ def download_source(tarball):
     if not os.path.exists("../emacs-src-cache/{}".format(tarball)):
         print("Downloading {}...".format(tarball))
         check_output_maybe(
-            "wget -a ../download.log -O ../emacs-src-cache/{} {}/{}/download"
+            "wget -a ../download.log -O ../emacs-src-cache/{} {}/{}"
             .format(tarball, SRC_REPO, tarball),
             shell=True
         )

> That directory contains tarballs with the exact sources used to build
> their respective binary packages.

Adjusted build-dep-zips.py and had to install some missing MSYS2
packages but now the script builds dpendencies.  And the other script
seems to build Emacs.  But I'm still unsecure about the naming.
E.g. why are snapshots of Emacs aren't named with the date but just
"emacs-29.0.50-snapshot"?

~/
|- emacs-build/
|	|- git/
|	|  |- emacs-$branch ?
|	|  |- master/ (for snapshots) -> emacs-29.0.50-snapshot
|	|  |- emacs-$version
|	|- deps/
|	|  |- libXpm/
|	|  |  |- libXpm-noX4.dll (cp from /bin, manually)
|       |  |- src-emacs (sources from dep py script)  
|       |  |- src-emacs-cache (from dep py script)  
|       |  |- x86_64 (DLLs from dep.py script)  
|	|- build/
|	|  |- $version
|	|- install/ (from script)
|	   |- emacs-29.0.50-snapshot/
|             |- Emacs' tree
|- emacs-upload/

   Dieter

Re: MinGW Sources, was: Windows Binaries Release

[Phillip Lord]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
>>
>>> in d:/Emacs/emacs-28/admin/nt/dist-build/build-dep-zips.py there is 
>>>
>>>   SRC_REPO="https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources".
>>>
>>> But it seems that the Sources folder vanished (for the moment?)!  Do
>>> you, or anybody else, know where else sources of MinGW are?
>>
>> Try
>>
>> https://repo.msys2.org/mingw/sources/
>
> Yes, thank you!
>
> modified   admin/nt/dist-build/build-dep-zips.py
> @@ -121,7 +121,8 @@ def ntldd_munge(out):
>  
>  ## Currently no packages seem to require this!
>  ARCH_PKGS=[]
> -SRC_REPO="https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources"
> +## SRC_REPO="https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources"
> +SRC_REPO="https://repo.msys2.org/mingw/sources"
>  
>  
>  def immediate_deps(pkg):
> @@ -167,7 +168,7 @@ def download_source(tarball):
>      if not os.path.exists("../emacs-src-cache/{}".format(tarball)):
>          print("Downloading {}...".format(tarball))
>          check_output_maybe(
> -            "wget -a ../download.log -O ../emacs-src-cache/{} {}/{}/download"
> +            "wget -a ../download.log -O ../emacs-src-cache/{} {}/{}"
>              .format(tarball, SRC_REPO, tarball),
>              shell=True
>          )
>

Do you have access to the Emacs git repo? Can you add this?



>> That directory contains tarballs with the exact sources used to build
>> their respective binary packages.
>
> Adjusted build-dep-zips.py and had to install some missing MSYS2
> packages but now the script builds dpendencies.  And the other script
> seems to build Emacs.  But I'm still unsecure about the naming.
> E.g. why are snapshots of Emacs aren't named with the date but just
> "emacs-29.0.50-snapshot"?
>
> ~/
> |- emacs-build/
> |	|- git/
> |	|  |- emacs-$branch ?
> |	|  |- master/ (for snapshots) -> emacs-29.0.50-snapshot
> |	|  |- emacs-$version
> |	|- deps/
> |	|  |- libXpm/
> |	|  |  |- libXpm-noX4.dll (cp from /bin, manually)
> |       |  |- src-emacs (sources from dep py script)  
> |       |  |- src-emacs-cache (from dep py script)  
> |       |  |- x86_64 (DLLs from dep.py script)  
> |	|- build/
> |	|  |- $version
> |	|- install/ (from script)
> |	   |- emacs-29.0.50-snapshot/
> |             |- Emacs' tree
> |- emacs-upload/




Right. When a release happens, we create a new worktree from the
tag. So, we currently have the emacs-28 release branch which will we
eventually worktree to give emacs-28.1.

The problem is that because we are making a build from a completely
unbuilt branch at this point we do a de novo build of everything,
including all the el->elc byte compilation (and some elc->eln for the
preloads). It takes time -- about 4-5 hours for a 64 and 32 bit build
(IIRC, we have ditched the i686 build now, but still).

That's fine for a release. But for snapshots, it a lot. So we might want
to build snapshots from the emacs-28 or master branch. For snapshots,
the risk of an non-clean, incremental build is perfectly justified, so
we do not make a new branch. As the elc files go into the source tree
(even for an "out-of-source" build) they will still be there; again,
IIRC, the scripts do not delete the "build" directory so even the C is
build incrementally; we do need to delete the install directory, because
building the zip file with dependencies, including the installer
version, alters the install tree, so make that clean every time -- not a
biggie as it's just a copy and takes very little time.

All of this, of course, is dependent on building shapshots. I did that
well for a while, but over the recent past, it has happened very, very
rarely. Be a different matter, perhaps, if it were fully automated, but
I never quite got there. Snapshots are not essential, though, and I have
no idea how many people were actually using them.

Cheers

Phil

Re: MinGW Sources, was: Windows Binaries Release

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Date: Mon, 08 Nov 2021 22:27:22 +0000
> Cc: Óscar Fuentes <ofv@wanadoo.es>, emacs-devel@gnu.org
> 
> The problem is that because we are making a build from a completely
> unbuilt branch at this point we do a de novo build of everything,
> including all the el->elc byte compilation (and some elc->eln for the
> preloads). It takes time -- about 4-5 hours for a 64 and 32 bit build
> (IIRC, we have ditched the i686 build now, but still).

Unless the VM where you build this is extremely slow and/or
resource-depleted, it should take much less than several hours.  I'd
expect less than 30 minutes, probably even 15 or 20, assuming you use
"make -j8" or more.

Re: MinGW Sources, was: Windows Binaries Release

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Date: Mon, 08 Nov 2021 22:27:22 +0000
>> Cc: Óscar Fuentes <ofv@wanadoo.es>, emacs-devel@gnu.org
>> 
>> The problem is that because we are making a build from a completely
>> unbuilt branch at this point we do a de novo build of everything,
>> including all the el->elc byte compilation (and some elc->eln for the
>> preloads). It takes time -- about 4-5 hours for a 64 and 32 bit build
>> (IIRC, we have ditched the i686 build now, but still).
>
> Unless the VM where you build this is extremely slow and/or
> resource-depleted, it should take much less than several hours.  I'd
> expect less than 30 minutes, probably even 15 or 20, assuming you use
> "make -j8" or more.

It's slower than that on some of my physical machines. But, yes,
throwing resource at it would make a massive difference to the
performance. And that in turn would simplify the build scripts.

Guess that's true with lots of the Emacs internals which were build when
"eight megabytes" was a resource hog.

Phil

Re: Windows Binaries Release: was The emacs-28 release branch

[Phillip Lord]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> Hello Phil
>
> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> They aren't anywhere accessible, because they are a bit specific to my
>> use.
>>
>> But for instance, I use one called "build-emacs27.sh" which essentially
>> looks like:
>>
>> rm -rf ~/emacs-build/build
>> rm -rf ~/emacs-build/install
>>
>> cd emacs-27/admin/nt/dist-build
>>
>>
>> ./build-zips.sh -g
>> ./build-zips.sh
>>
>
> I created such a script but my problem is that chmod seems not to work
> for me!
>
>    chmod u+x build-zips.sh
>
> doesn't change anything, so I can't run the scripts.  Do you know this
> problem?

I'd need an error message. build-zips.sh needs to be run in a mingw64
shell, and build-deps-zips.py in an msys2 shell (for obscure reasons
that I don't really understand).


>> Then I have "build-27-deps.sh" which again is just a launch script.
>>
>>
>> set -o errexit
>> if test -f emacs-src; then
>>     rsync -r emacs-src emacs-src-cache
>> fi
>> rm -rf i686 x86_64 emacs-src
>> ../git/emacs-27/admin/nt/dist-build/build-dep-zips.py -s 2>&1 | tee
>> build-deps.log
>> cp emacs-27* ~/emacs-upload
>
> I don't get here the "emacs-src" and "emacs-src-cache" folders, they
> aren't mention in README-scripts.  Do I have to create them by hand?

build-deps-zips should create them both at the same time. The cache is
useful because it means you don't need to download the source every
time. This step also tends to error and my script can't recover from
this, so the cache means it can pick up from where it left off.





> So I tried to run build-dep.zips.py -s by hand for (a snapshot) but I
> haven't had luck:
>
> Please have a look at the end and tell me which MSYS2 package contains
> /mingw64/bin/ntldd?  I don't see it in the system and in pacman
>
> Traceback (most recent call last):
>   File "/c/Users/uidg1626/emacs-build/deps/../git/master/admin/nt/dist-build/build-dep-zips.py", line 267, in <module>
>     gather_deps()
>   File "/c/Users/uidg1626/emacs-build/deps/../git/master/admin/nt/dist-build/build-dep-zips.py", line 71, in gather_deps
>     for dep in full_dll_dependency():
>   File "/c/Users/uidg1626/emacs-build/deps/../git/master/admin/nt/dist-build/build-dep-zips.py", line 83, in full_dll_dependency
>     deps = [dll_dependency(dep) for dep in DLL_REQ]
>   File "/c/Users/uidg1626/emacs-build/deps/../git/master/admin/nt/dist-build/build-dep-zips.py", line 83, in <listcomp>
>     deps = [dll_dependency(dep) for dep in DLL_REQ]
>   File "/c/Users/uidg1626/emacs-build/deps/../git/master/admin/nt/dist-build/build-dep-zips.py", line 88, in dll_dependency
>     output = check_output(["/mingw64/bin/ntldd", "--recursive",
>   File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
>     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
>   File "/usr/lib/python3.9/subprocess.py", line 505, in run
>     with Popen(*popenargs, **kwargs) as process:
>   File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
>     self._execute_child(args, executable, preexec_fn, close_fds,
>   File "/usr/lib/python3.9/subprocess.py", line 1821, in _execute_child
>     raise child_exception_type(errno_num, err_msg, err_filename)
> FileNotFoundError: [Errno 2] No such file or directory: '/mingw64/bin/ntldd' 
>
>
> Thank you for your assistance

It's a mingw64 package. Building requires the full toolchain.

mingw-w64-x86_64-ntldd-git - MSYS2 Packages

Phil

Re: Windows Binaries Release: was The emacs-28 release branch

[H. Dieter Wilhelm]

Hello Phil

I'm a bit unsecure about the naming of the git/ folders.

What do you mean by "git/emacs-$branch"? For snapshots, I think, it is
"git/master", not "git/emacs-master"?  This variable $branch pains me...

Then for the emacs-28 branch must it be named "git worktree
emacs-emacs-28"?  or is it just "emacs-28" but then wouldn't it clash
with the release branch emacs-28?  Sorry, it's still a bit confusing, I
think I've to fully understand the build scripts...

Anyway:

Phillip Lord <phillip.lord@russet.org.uk> writes:

> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
>
>> Hello Phil
>>
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>
>>> They aren't anywhere accessible, because they are a bit specific to my
>>> use.
>>>
>>> But for instance, I use one called "build-emacs27.sh" which essentially
>>> looks like:
>>>
>>> rm -rf ~/emacs-build/build
>>> rm -rf ~/emacs-build/install
>>>
>>> cd emacs-27/admin/nt/dist-build
>>>
>>>
>>> ./build-zips.sh -g
>>> ./build-zips.sh
>>>
>>
>> I created such a script but my problem is that chmod seems not to work
>> for me!
>>
>>    chmod u+x build-zips.sh
>>
>> doesn't change anything, so I can't run the scripts.  Do you know this
>> problem?
>
> I'd need an error message. build-zips.sh needs to be run in a mingw64
> shell, and build-deps-zips.py in an msys2 shell (for obscure reasons
> that I don't really understand).

Thanks.  I mistakenly tried to run the scripts in an *eshell*!

>>> Then I have "build-27-deps.sh" which again is just a launch script.
>>>
>>>
>>> set -o errexit
>>> if test -f emacs-src; then
>>>     rsync -r emacs-src emacs-src-cache
>>> fi
>>> rm -rf i686 x86_64 emacs-src
>>> ../git/emacs-27/admin/nt/dist-build/build-dep-zips.py -s 2>&1 | tee
>>> build-deps.log
>>> cp emacs-27* ~/emacs-upload
>>
>> I don't get here the "emacs-src" and "emacs-src-cache" folders, they
>> aren't mention in README-scripts.  Do I have to create them by hand?
>
> build-deps-zips should create them both at the same time. The cache is
> useful because it means you don't need to download the source every
> time. This step also tends to error and my script can't recover from
> this, so the cache means it can pick up from where it left off.

I see, thanks for the clarifiation.

>> So I tried to run build-dep.zips.py -s by hand for (a snapshot) but I
>> haven't had luck:
>>
>> Please have a look at the end and tell me which MSYS2 package contains
>> /mingw64/bin/ntldd?  I don't see it in the system and in pacman
>>
>> Traceback (most recent call last):
>>   File "/c/Users/uidg1626/emacs-build/deps/../git/master/admin/nt/dist-build/build-dep-zips.py", line 267, in <module>
>>     gather_deps()
>>   File "/c/Users/uidg1626/emacs-build/deps/../git/master/admin/nt/dist-build/build-dep-zips.py", line 71, in gather_deps
>>     for dep in full_dll_dependency():
>>   File "/c/Users/uidg1626/emacs-build/deps/../git/master/admin/nt/dist-build/build-dep-zips.py", line 83, in full_dll_dependency
>>     deps = [dll_dependency(dep) for dep in DLL_REQ]
>>   File "/c/Users/uidg1626/emacs-build/deps/../git/master/admin/nt/dist-build/build-dep-zips.py", line 83, in <listcomp>
>>     deps = [dll_dependency(dep) for dep in DLL_REQ]
>>   File "/c/Users/uidg1626/emacs-build/deps/../git/master/admin/nt/dist-build/build-dep-zips.py", line 88, in dll_dependency
>>     output = check_output(["/mingw64/bin/ntldd", "--recursive",
>>   File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
>>     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
>>   File "/usr/lib/python3.9/subprocess.py", line 505, in run
>>     with Popen(*popenargs, **kwargs) as process:
>>   File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
>>     self._execute_child(args, executable, preexec_fn, close_fds,
>>   File "/usr/lib/python3.9/subprocess.py", line 1821, in _execute_child
>>     raise child_exception_type(errno_num, err_msg, err_filename)
>> FileNotFoundError: [Errno 2] No such file or directory: '/mingw64/bin/ntldd' 
>>
>>
>> Thank you for your assistance
>
> It's a mingw64 package. Building requires the full toolchain.
>
> mingw-w64-x86_64-ntldd-git - MSYS2 Packages

Perfect, I'll install it!

Have a nice time

     Dieter

Re: Windows Binaries Release: was The emacs-28 release branch

[Phillip Lord]

Yeah, apologies I have been a bit overwhelmed recently, and didn't
reply! Might be easier to do something asynchronous?




Corwin Brust <corwin@bru.st> writes:

> Hi Dieter!
>
>
> On Sat, Oct 16, 2021, 05:04 H. Dieter Wilhelm <dieter@duenenhof-wilhelm.de>
> wrote:
>
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>
>> > Corwin Brust <corwin@bru.st> writes:
>> >
>> >>> As the release of Emacs-28 is soon, I would like to look for someone to
>> >>> take over the role of producing windows releases.
>> >>>
>> >>> If anyone is interested, let me know.
>> >>
>> >> I could be interested!
>>
>> Me too
>>
>
> Yay!
>
>
>> Would you mind to send me your recipes or shall I contact you
>> beforehand?
>>
>
>
> Phil and I have been trying to connect to walk through the process, also.
> I would welcome you, given we can eventually connect and  you are
> interested.
>
> I've suggested using the emacsconf BBB so we can easily make an informal
> recording.
>
>
>> Thank you
>>
>>       Dieter
>>
>>
>>

Re: Windows Binaries Release: was The emacs-28 release branch

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 1287 bytes --]

I'm in fairly good shape building, afaict.

I could use pointers on signing, packaging, and pushing.

On Wed, Oct 20, 2021, 10:16 Phillip Lord <phillip.lord@russet.org.uk> wrote:

>
> Yeah, apologies I have been a bit overwhelmed recently, and didn't
> reply! Might be easier to do something asynchronous?
>
>
>
>
> Corwin Brust <corwin@bru.st> writes:
>
> > Hi Dieter!
> >
> >
> > On Sat, Oct 16, 2021, 05:04 H. Dieter Wilhelm <
> dieter@duenenhof-wilhelm.de>
> > wrote:
> >
> >> Phillip Lord <phillip.lord@russet.org.uk> writes:
> >>
> >> > Corwin Brust <corwin@bru.st> writes:
> >> >
> >> >>> As the release of Emacs-28 is soon, I would like to look for
> someone to
> >> >>> take over the role of producing windows releases.
> >> >>>
> >> >>> If anyone is interested, let me know.
> >> >>
> >> >> I could be interested!
> >>
> >> Me too
> >>
> >
> > Yay!
> >
> >
> >> Would you mind to send me your recipes or shall I contact you
> >> beforehand?
> >>
> >
> >
> > Phil and I have been trying to connect to walk through the process, also.
> > I would welcome you, given we can eventually connect and  you are
> > interested.
> >
> > I've suggested using the emacsconf BBB so we can easily make an informal
> > recording.
> >
> >
> >> Thank you
> >>
> >>       Dieter
> >>
> >>
> >>
>

[-- Attachment #2: Type: text/html, Size: 2365 bytes --]

Re: Windows Binaries Release: was The emacs-28 release branch

[Phillip Lord]

All the packaging is done by "build-zips.sh" and "build-deps.py". I've
replied in more detail in an overlapping email to Dieter.


Signing and uploading is uses gnupload

https://www.gnu.org/prep/maintain/html_node/Automated-Upload-Procedure.html

For this, you need to get your GPG keys registered with the FSF sys
admin. But you can try it now, it's just the uploads will get deleted if
the keys aren't known.

Phil

Corwin Brust <corwin@bru.st> writes:

> I'm in fairly good shape building, afaict.
>
> I could use pointers on signing, packaging, and pushing.
>
> On Wed, Oct 20, 2021, 10:16 Phillip Lord <phillip.lord@russet.org.uk> wrote:
>
>>
>> Yeah, apologies I have been a bit overwhelmed recently, and didn't
>> reply! Might be easier to do something asynchronous?
>>
>>
>>
>>
>> Corwin Brust <corwin@bru.st> writes:
>>
>> > Hi Dieter!
>> >
>> >
>> > On Sat, Oct 16, 2021, 05:04 H. Dieter Wilhelm <
>> dieter@duenenhof-wilhelm.de>
>> > wrote:
>> >
>> >> Phillip Lord <phillip.lord@russet.org.uk> writes:
>> >>
>> >> > Corwin Brust <corwin@bru.st> writes:
>> >> >
>> >> >>> As the release of Emacs-28 is soon, I would like to look for
>> someone to
>> >> >>> take over the role of producing windows releases.
>> >> >>>
>> >> >>> If anyone is interested, let me know.
>> >> >>
>> >> >> I could be interested!
>> >>
>> >> Me too
>> >>
>> >
>> > Yay!
>> >
>> >
>> >> Would you mind to send me your recipes or shall I contact you
>> >> beforehand?
>> >>
>> >
>> >
>> > Phil and I have been trying to connect to walk through the process, also.
>> > I would welcome you, given we can eventually connect and  you are
>> > interested.
>> >
>> > I've suggested using the emacsconf BBB so we can easily make an informal
>> > recording.
>> >
>> >
>> >> Thank you
>> >>
>> >>       Dieter
>> >>
>> >>
>> >>
>>

Re: The emacs-28 release branch has been created

[Ken Brown]

On 9/30/2021 2:30 PM, Eli Zaretskii wrote:
> Barring any unexpected calamities, the first pretest of Emacs 28 will
> be produced from this branch in a few weeks' time.

I apologize in advance if this has already been answered somewhere, but I'm 
curious how building from a pretest tarball will work for people who want to 
test native compilation.  Release tarballs always contain *.elc files, which 
seem to inhibit the creation of the preloaded *.eln files.

I had assumed that commit 90655e4bc0 was intended to deal with this, but I'm not 
able to see how it will work.

Thanks.

Ken

Re: The emacs-28 release branch has been created

[Andreas Schwab]

On Okt 02 2021, Ken Brown wrote:

> I apologize in advance if this has already been answered somewhere, but
> I'm curious how building from a pretest tarball will work for people who
> want to test native compilation.  Release tarballs always contain *.elc
> files, which seem to inhibit the creation of the preloaded *.eln files.

You can always run make bootstrap.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 7578 EB47 D4E5 4D69 2510  2552 DF73 E780 A9DA AEC1
"And now for something completely different."

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> From: Ken Brown <kbrown@cornell.edu>
> Date: Sat, 2 Oct 2021 21:35:36 -0400
> 
> On 9/30/2021 2:30 PM, Eli Zaretskii wrote:
> > Barring any unexpected calamities, the first pretest of Emacs 28 will
> > be produced from this branch in a few weeks' time.
> 
> I apologize in advance if this has already been answered somewhere, but I'm 
> curious how building from a pretest tarball will work for people who want to 
> test native compilation.  Release tarballs always contain *.elc files, which 
> seem to inhibit the creation of the preloaded *.eln files.
> 
> I had assumed that commit 90655e4bc0 was intended to deal with this, but I'm not 
> able to see how it will work.

Good question.  That commit was indeed supposed to deal with this, but
I didn't yet have time to actually test it with a sample tarball, so
it could include bugs (I'm not too proficient with Make wizardry, and
the problem was not trivial to fix).  If you have time to produce a
tarball from the emacs-28 branch (the instructions are in
admin/make-tarball.txt) and then try building it, perhaps you could
test this and report any problems.  If not, I will get to that soon
enough.

The way it's supposed to work is that a special rule added in that
commit to src/Makefile.in should be triggered by the fact that the
native-lisp directory is missing in the build tree.  That special rule
is supposed to build the preloaded *.eln files (and only those), and
then re-dump Emacs.

Re: The emacs-28 release branch has been created

[Ken Brown]

On 10/3/2021 5:27 AM, Eli Zaretskii wrote:
>> From: Ken Brown <kbrown@cornell.edu>
>> Date: Sat, 2 Oct 2021 21:35:36 -0400
>>
>> On 9/30/2021 2:30 PM, Eli Zaretskii wrote:
>>> Barring any unexpected calamities, the first pretest of Emacs 28 will
>>> be produced from this branch in a few weeks' time.
>>
>> I apologize in advance if this has already been answered somewhere, but I'm
>> curious how building from a pretest tarball will work for people who want to
>> test native compilation.  Release tarballs always contain *.elc files, which
>> seem to inhibit the creation of the preloaded *.eln files.
>>
>> I had assumed that commit 90655e4bc0 was intended to deal with this, but I'm not
>> able to see how it will work.
> 
> Good question.  That commit was indeed supposed to deal with this, but
> I didn't yet have time to actually test it with a sample tarball, so
> it could include bugs (I'm not too proficient with Make wizardry, and
> the problem was not trivial to fix).  If you have time to produce a
> tarball from the emacs-28 branch (the instructions are in
> admin/make-tarball.txt) and then try building it, perhaps you could
> test this and report any problems.  If not, I will get to that soon
> enough.

I'll try that later today.  In the meantime, I did notice one bug: 
src/Makefile.in is missing

   HAVE_NATIVE_COMP = @HAVE_NATIVE_COMP@

Ken

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Ken Brown <kbrown@cornell.edu>
> Date: Sun, 3 Oct 2021 11:01:25 -0400
> 
> I'll try that later today.

Thanks!

> In the meantime, I did notice one bug: src/Makefile.in is missing
> 
>    HAVE_NATIVE_COMP = @HAVE_NATIVE_COMP@

I think it's supposed to get that from the top-level Makefile.
Doesn't it do that in your builds?

Re: The emacs-28 release branch has been created

[Ken Brown]

On 10/3/2021 11:17 AM, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Ken Brown <kbrown@cornell.edu>
>> Date: Sun, 3 Oct 2021 11:01:25 -0400
>>
>> I'll try that later today.
> 
> Thanks!
> 
>> In the meantime, I did notice one bug: src/Makefile.in is missing
>>
>>     HAVE_NATIVE_COMP = @HAVE_NATIVE_COMP@
> 
> I think it's supposed to get that from the top-level Makefile.
> Doesn't it do that in your builds?

No.  And both lib/Makefile.in and lisp/Makefile.in include that line.

Ken

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Ken Brown <kbrown@cornell.edu>
> Date: Sun, 3 Oct 2021 11:34:31 -0400
> 
> >>     HAVE_NATIVE_COMP = @HAVE_NATIVE_COMP@
> > 
> > I think it's supposed to get that from the top-level Makefile.
> > Doesn't it do that in your builds?
> 
> No.  And both lib/Makefile.in and lisp/Makefile.in include that line.

Oops!  Should be fixed now, thanks.

Re: The emacs-28 release branch has been created

[Ken Brown]

On 10/3/2021 12:11 PM, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Ken Brown <kbrown@cornell.edu>
>> Date: Sun, 3 Oct 2021 11:34:31 -0400
>>
>>>>      HAVE_NATIVE_COMP = @HAVE_NATIVE_COMP@
>>>
>>> I think it's supposed to get that from the top-level Makefile.
>>> Doesn't it do that in your builds?
>>
>> No.  And both lib/Makefile.in and lisp/Makefile.in include that line.
> 
> Oops!  Should be fixed now, thanks.

I built a tarball and started testing, and I found another bug, this one 
somewhat non-intuitive: In the recipe for ../native-lisp, "mkdir" needs to be 
replaced by "$(MKDIR_P)" (or even omitted).  The reason is that the native-lisp 
directory actually exists by the time that recipe is executed.

Suppose you run "make all" in src and the native-lisp directory doesn't exist. 
Make sees that the prerequisite "../native-lisp" of "all" doesn't exist, so it 
remembers that it will have to build it after building emacs$(EXEEXT), $(pdmp), 
and $(OTHER_FILES).  But by that time native-lisp exists because of "make 
compile-first" in the lisp directory.

Here's what I see in my build log before making the change:

./temacs --batch  -l loadup --temacs=pbootstrap \
	--bin-dest /usr/local/bin/ --eln-dest /usr/local/lib/emacs/28.0.60/
[...]
make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
make[2]: Entering directory '/tmp/emacs-28.0.60/lisp'
   ELC+ELN  emacs-lisp/comp.elc
   ELC+ELN  emacs-lisp/comp-cstr.elc
[...]
LC_ALL=C ./temacs -batch  -l loadup --temacs=pdump \
	--bin-dest /usr/local/bin/ --eln-dest /usr/local/lib/emacs/28.0.60/
[...]
/usr/bin/mkdir ../native-lisp && make --no-print-directory 
../lisp/emacs-lisp/autoload.eln ...

... and the mkdir command fails.

After making the change, the native compilation of the preloaded files proceeds, 
and I think everything will be OK.  I say "I think", because I ran into fork 
failures (even on 64-bit Cygwin!), so I'll need to insert rebase commands 
somewhere before I can test further.

Ken

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Ken Brown <kbrown@cornell.edu>
> Date: Sun, 3 Oct 2021 13:14:15 -0400
> 
> I built a tarball and started testing, and I found another bug, this one 
> somewhat non-intuitive: In the recipe for ../native-lisp, "mkdir" needs to be 
> replaced by "$(MKDIR_P)" (or even omitted).  The reason is that the native-lisp 
> directory actually exists by the time that recipe is executed.
> 
> Suppose you run "make all" in src and the native-lisp directory doesn't exist. 
> Make sees that the prerequisite "../native-lisp" of "all" doesn't exist, so it 
> remembers that it will have to build it after building emacs$(EXEEXT), $(pdmp), 
> and $(OTHER_FILES).  But by that time native-lisp exists because of "make 
> compile-first" in the lisp directory.

I think that's the problem to fix: compile-first isn't supposed to run
in this case, because all the *.elc files are already present in the
tarball.  Can you figure out why does compile-first run? which one of
the *.elc files it depends on is outdated?

> make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
> make[2]: Entering directory '/tmp/emacs-28.0.60/lisp'
>    ELC+ELN  emacs-lisp/comp.elc
>    ELC+ELN  emacs-lisp/comp-cstr.elc

This (also) regenerates *.elc files that are supposed to be in the
tarball, which is not what we want.

Thanks.

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> Date: Sun, 03 Oct 2021 20:33:00 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > Suppose you run "make all" in src and the native-lisp directory doesn't exist. 
> > Make sees that the prerequisite "../native-lisp" of "all" doesn't exist, so it 
> > remembers that it will have to build it after building emacs$(EXEEXT), $(pdmp), 
> > and $(OTHER_FILES).  But by that time native-lisp exists because of "make 
> > compile-first" in the lisp directory.
> 
> I think that's the problem to fix: compile-first isn't supposed to run
> in this case, because all the *.elc files are already present in the
> tarball.  Can you figure out why does compile-first run? which one of
> the *.elc files it depends on is outdated?

IOW, how is this different from the build without native-compilation,
where compile-first is invoked, but does nothing (because the *.elc
files are already built)?  In the native-compilation build, Make is
supposed to look at the same *.elc files, just at a longer list of
them, so how come it finds something outdated?

Re: The emacs-28 release branch has been created

[Ken Brown]

On 10/3/2021 1:33 PM, Eli Zaretskii wrote:
>> make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
>> make[2]: Entering directory '/tmp/emacs-28.0.60/lisp'
>>     ELC+ELN  emacs-lisp/comp.elc
>>     ELC+ELN  emacs-lisp/comp-cstr.elc
> 
> This (also) regenerates *.elc files that are supposed to be in the
> tarball, which is not what we want.

It turns out that those *.elc files are not in the tarball because of my own 
stupid mistake.  When I ran make-dist, I got a warning about that.  I didn't 
want to think about it, so I reran make-dist with --no-check rather than fixing 
the problem.

Sorry for the noise.

Ken

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Ken Brown <kbrown@cornell.edu>
> Date: Sun, 3 Oct 2021 13:56:56 -0400
> 
> On 10/3/2021 1:33 PM, Eli Zaretskii wrote:
> >> make -C ../lisp compile-first EMACS="../src/bootstrap-emacs.exe"
> >> make[2]: Entering directory '/tmp/emacs-28.0.60/lisp'
> >>     ELC+ELN  emacs-lisp/comp.elc
> >>     ELC+ELN  emacs-lisp/comp-cstr.elc
> > 
> > This (also) regenerates *.elc files that are supposed to be in the
> > tarball, which is not what we want.
> 
> It turns out that those *.elc files are not in the tarball because of my own 
> stupid mistake.  When I ran make-dist, I got a warning about that.  I didn't 
> want to think about it, so I reran make-dist with --no-check rather than fixing 
> the problem.

Ah, okay.  That explains why those rules did produce the *.elc+*.eln
files.

So I guess, once you rebase, you will retry and see if everything does
work?

Thanks.

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Ken Brown <kbrown@cornell.edu>
> Date: Sun, 3 Oct 2021 13:56:56 -0400
> 
> It turns out that those *.elc files are not in the tarball because of my own 
> stupid mistake.  When I ran make-dist, I got a warning about that.  I didn't 
> want to think about it, so I reran make-dist with --no-check rather than fixing 
> the problem.

Could it be that the *.elc files were not in the tarball because the
build failed at some point?

There's still a bug in src/Makefile.in: Make could try building
native-lisp even though it exists, because the rules that create that
directory don't tell Make the directory is created as a side effect.
If and when Make tries to rune the ../native-lisp: rule, it will fail
because mkdir will fail.

One way of solving this is to make the recipe test whether the
directory exists, and if so, do nothing, because we don't want to run
native-compilations if that directory exists.  Hmm...

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> Date: Sun, 03 Oct 2021 22:20:39 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> One way of solving this is to make the recipe test whether the
> directory exists, and if so, do nothing, because we don't want to run
> native-compilations if that directory exists.  Hmm...

OK, I installed something along these lines, but didn't yet have time
to test it in a build out of clean Git clone (which is where the
problem happened).

Re: The emacs-28 release branch has been created

[Ken Brown]

On 10/3/2021 3:20 PM, Eli Zaretskii wrote:
>> Cc: emacs-devel@gnu.org
>> From: Ken Brown <kbrown@cornell.edu>
>> Date: Sun, 3 Oct 2021 13:56:56 -0400
>>
>> It turns out that those *.elc files are not in the tarball because of my own
>> stupid mistake.  When I ran make-dist, I got a warning about that.  I didn't
>> want to think about it, so I reran make-dist with --no-check rather than fixing
>> the problem.
> 
> Could it be that the *.elc files were not in the tarball because the
> build failed at some point?

No, they weren't in the tarball because when I built emacs prior to running 
make-dist, I didn't specify --with-native-compilation.  Currently 
lisp/Makefile.in has

ifneq ($(HAVE_NATIVE_COMP),yes)
compile-targets: $(filter-out ./emacs-lisp/comp-cstr.elc,$(filter-out 
./emacs-lisp/comp.elc,$(TARGETS)))

That seems wrong to me.

> There's still a bug in src/Makefile.in: Make could try building
> native-lisp even though it exists, because the rules that create that
> directory don't tell Make the directory is created as a side effect.
> If and when Make tries to rune the ../native-lisp: rule, it will fail
> because mkdir will fail.

I see you've fixed that now.  I'll test it while building a new tarball.

Ken

Re: The emacs-28 release branch has been created

[Ken Brown]

On 10/3/2021 3:45 PM, Ken Brown wrote:
> On 10/3/2021 3:20 PM, Eli Zaretskii wrote:
>>> Cc: emacs-devel@gnu.org
>>> From: Ken Brown <kbrown@cornell.edu>
>>> Date: Sun, 3 Oct 2021 13:56:56 -0400
>>>
>>> It turns out that those *.elc files are not in the tarball because of my own
>>> stupid mistake.  When I ran make-dist, I got a warning about that.  I didn't
>>> want to think about it, so I reran make-dist with --no-check rather than fixing
>>> the problem.
>>
>> Could it be that the *.elc files were not in the tarball because the
>> build failed at some point?
> 
> No, they weren't in the tarball because when I built emacs prior to running 
> make-dist, I didn't specify --with-native-compilation.  Currently 
> lisp/Makefile.in has
> 
> ifneq ($(HAVE_NATIVE_COMP),yes)
> compile-targets: $(filter-out ./emacs-lisp/comp-cstr.elc,$(filter-out 
> ./emacs-lisp/comp.elc,$(TARGETS)))
> 
> That seems wrong to me.
> 
>> There's still a bug in src/Makefile.in: Make could try building
>> native-lisp even though it exists, because the rules that create that
>> directory don't tell Make the directory is created as a side effect.
>> If and when Make tries to rune the ../native-lisp: rule, it will fail
>> because mkdir will fail.
> 
> I see you've fixed that now.  I'll test it while building a new tarball.

That fix seems OK.  But there are still problems building from a tarball. 
First, there's an obvious typo (presumably a copy/paste error) in this part of 
the recipe for ../native-lisp:

   cp -f $@ $(bootstrap_pdmp)

I assume you want

   cp -f $(pdmp) $(bootstrap_pdmp)

if that's even needed at all.

Second, I get the following native-compilation error, which doesn't occur in an 
ordinary build (i.e., not from a tarball):

   ELN  ../lisp/disp-table.eln
Debugger entered--Lisp error: (native-compiler-error "../lisp/disp-table.el" 
"Debugger entered--Lisp error: (invalid-read-syntax...")
   signal(native-compiler-error ("../lisp/disp-table.el" "Debugger entered--Lisp 
error: (invalid-read-syntax..."))
   comp--native-compile("../lisp/disp-table.el")
   batch-native-compile(t)
   eval((batch-native-compile t) t)
   command-line-1(("--eval" "(setq load-prefer-newer t)" "-l" "comp" "-f" 
"byte-compile-refresh-preloaded" "--eval" "(batch-native-compile t)" 
"../lisp/disp-table.el"))
   command-line()
   normal-top-level()

I'll retry with make -k to see if there are any more such errors, but I might 
not get to it until tomorrow.

Ken

Re: The emacs-28 release branch has been created

[Ken Brown]

On 10/3/2021 5:21 PM, Ken Brown wrote:
> On 10/3/2021 3:45 PM, Ken Brown wrote:
>> On 10/3/2021 3:20 PM, Eli Zaretskii wrote:
>>>> Cc: emacs-devel@gnu.org
>>>> From: Ken Brown <kbrown@cornell.edu>
>>>> Date: Sun, 3 Oct 2021 13:56:56 -0400
>>>>
>>>> It turns out that those *.elc files are not in the tarball because of my own
>>>> stupid mistake.  When I ran make-dist, I got a warning about that.  I didn't
>>>> want to think about it, so I reran make-dist with --no-check rather than fixing
>>>> the problem.
>>>
>>> Could it be that the *.elc files were not in the tarball because the
>>> build failed at some point?
>>
>> No, they weren't in the tarball because when I built emacs prior to running 
>> make-dist, I didn't specify --with-native-compilation.  Currently 
>> lisp/Makefile.in has
>>
>> ifneq ($(HAVE_NATIVE_COMP),yes)
>> compile-targets: $(filter-out ./emacs-lisp/comp-cstr.elc,$(filter-out 
>> ./emacs-lisp/comp.elc,$(TARGETS)))
>>
>> That seems wrong to me.
>>
>>> There's still a bug in src/Makefile.in: Make could try building
>>> native-lisp even though it exists, because the rules that create that
>>> directory don't tell Make the directory is created as a side effect.
>>> If and when Make tries to rune the ../native-lisp: rule, it will fail
>>> because mkdir will fail.
>>
>> I see you've fixed that now.  I'll test it while building a new tarball.
> 
> That fix seems OK.  But there are still problems building from a tarball. First, 
> there's an obvious typo (presumably a copy/paste error) in this part of the 
> recipe for ../native-lisp:
> 
>    cp -f $@ $(bootstrap_pdmp)
> 
> I assume you want
> 
>    cp -f $(pdmp) $(bootstrap_pdmp)
> 
> if that's even needed at all.
> 
> Second, I get the following native-compilation error, which doesn't occur in an 
> ordinary build (i.e., not from a tarball):
> 
>    ELN  ../lisp/disp-table.eln
> Debugger entered--Lisp error: (native-compiler-error "../lisp/disp-table.el" 
> "Debugger entered--Lisp error: (invalid-read-syntax...")
>    signal(native-compiler-error ("../lisp/disp-table.el" "Debugger entered--Lisp 
> error: (invalid-read-syntax..."))
>    comp--native-compile("../lisp/disp-table.el")
>    batch-native-compile(t)
>    eval((batch-native-compile t) t)
>    command-line-1(("--eval" "(setq load-prefer-newer t)" "-l" "comp" "-f" 
> "byte-compile-refresh-preloaded" "--eval" "(batch-native-compile t)" 
> "../lisp/disp-table.el"))
>    command-line()
>    normal-top-level()
> 
> I'll retry with make -k to see if there are any more such errors

There were no more compilation errors, and the build seemed to finish correctly 
except for what I've already mentioned.

The rebase problem on Cygwin is fixed by the following:

diff --git a/src/Makefile.in b/src/Makefile.in
index 25c7865d4a..01ae7356b5 100644
--- a/src/Makefile.in
+++ b/src/Makefile.in
@@ -806,6 +806,9 @@ elnlisp := $(addprefix ${lispsource}/,${elnlisp}) $(lisp:
  ../native-lisp: | $(pdmp)
         if test ! -d $@; then \
           mkdir $@ && $(MAKE) $(AM_V_NO_PD) $(elnlisp); \
+         if test $(SYSTEM_TYPE) = cygwin; then \
+           find $@ -name '*.eln' | rebase -v -O -T -; \
+         fi; \
           LC_ALL=C $(RUN_TEMACS) -batch $(BUILD_DETAILS) -l loadup --temacs=pdump \
                 --bin-dest $(BIN_DESTDIR) --eln-dest $(ELN_DESTDIR); \
           cp -f $@ $(bootstrap_pdmp); \

OK to apply this bandaid?

Ken

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> From: Ken Brown <kbrown@cornell.edu>
> Cc: emacs-devel@gnu.org
> Date: Sun, 3 Oct 2021 18:40:11 -0400
> 
> diff --git a/src/Makefile.in b/src/Makefile.in
> index 25c7865d4a..01ae7356b5 100644
> --- a/src/Makefile.in
> +++ b/src/Makefile.in
> @@ -806,6 +806,9 @@ elnlisp := $(addprefix ${lispsource}/,${elnlisp}) $(lisp:
>   ../native-lisp: | $(pdmp)
>          if test ! -d $@; then \
>            mkdir $@ && $(MAKE) $(AM_V_NO_PD) $(elnlisp); \
> +         if test $(SYSTEM_TYPE) = cygwin; then \
> +           find $@ -name '*.eln' | rebase -v -O -T -; \
> +         fi; \
>            LC_ALL=C $(RUN_TEMACS) -batch $(BUILD_DETAILS) -l loadup --temacs=pdump \
>                  --bin-dest $(BIN_DESTDIR) --eln-dest $(ELN_DESTDIR); \
>            cp -f $@ $(bootstrap_pdmp); \
> 
> OK to apply this bandaid?

Yes, of course.  But note that I changed that part today.

Re: The emacs-28 release branch has been created

[Ken Brown]

On 10/3/2021 5:21 PM, Ken Brown wrote:
> Second, I get the following native-compilation error, which doesn't occur in an 
> ordinary build (i.e., not from a tarball):
> 
>    ELN  ../lisp/disp-table.eln
> Debugger entered--Lisp error: (native-compiler-error "../lisp/disp-table.el" 
> "Debugger entered--Lisp error: (invalid-read-syntax...")
>    signal(native-compiler-error ("../lisp/disp-table.el" "Debugger entered--Lisp 
> error: (invalid-read-syntax..."))
>    comp--native-compile("../lisp/disp-table.el")
>    batch-native-compile(t)
>    eval((batch-native-compile t) t)
>    command-line-1(("--eval" "(setq load-prefer-newer t)" "-l" "comp" "-f" 
> "byte-compile-refresh-preloaded" "--eval" "(batch-native-compile t)" 
> "../lisp/disp-table.el"))
>    command-line()
>    normal-top-level()

This has nothing to do with building from a tarball, of course.  I can reproduce 
the problem as follows, starting from a git checkout:

1. ./configure --with-native-compilation && make

2. make clean

3. make

Ken

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> From: Ken Brown <kbrown@cornell.edu>
> Cc: emacs-devel@gnu.org
> Date: Mon, 4 Oct 2021 09:31:25 -0400
> 
> >    ELN  ../lisp/disp-table.eln
> > Debugger entered--Lisp error: (native-compiler-error "../lisp/disp-table.el" 
> > "Debugger entered--Lisp error: (invalid-read-syntax...")
> >    signal(native-compiler-error ("../lisp/disp-table.el" "Debugger entered--Lisp 
> > error: (invalid-read-syntax..."))
> >    comp--native-compile("../lisp/disp-table.el")
> >    batch-native-compile(t)
> >    eval((batch-native-compile t) t)
> >    command-line-1(("--eval" "(setq load-prefer-newer t)" "-l" "comp" "-f" 
> > "byte-compile-refresh-preloaded" "--eval" "(batch-native-compile t)" 
> > "../lisp/disp-table.el"))
> >    command-line()
> >    normal-top-level()
> 
> This has nothing to do with building from a tarball, of course.  I can reproduce 
> the problem as follows, starting from a git checkout:
> 
> 1. ./configure --with-native-compilation && make
> 
> 2. make clean
> 
> 3. make

Right.  Andrea, could you please look into this as soon as you can?
This currently blocks the pretest, because the build from the source
tarball fails.

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> Date: Mon, 04 Oct 2021 17:25:44 +0300
> From: Eli Zaretskii <eliz@gnu.org>
> Cc: emacs-devel@gnu.org
> 
> > 1. ./configure --with-native-compilation && make
> > 
> > 2. make clean
> > 
> > 3. make
> 
> Right.  Andrea, could you please look into this as soon as you can?
> This currently blocks the pretest, because the build from the source
> tarball fails.

To make reproduction easier, this command succeeds, when issued from
the lisp/ directory:

  $ make disp-table.elc THEFILE=disp-table.el V=1 LISP_PRELOADED=disp-table.el
  EMACSLOADPATH= '../src/emacs' -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)'  \
	  -l comp -f byte-compile-refresh-preloaded \
	  -f batch-byte+native-compile disp-table.el

while this command fails:

  $ make disp-table.eln THEFILE=disp-table.el V=1
  EMACSLOADPATH= '../src/emacs' -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)'  \
	  -l comp -f byte-compile-refresh-preloaded \
	  --eval '(batch-native-compile t)' disp-table.el
  Debugger entered--Lisp error: (native-compiler-error "disp-table.el" "Debugger entered--Lisp error: (invalid-read-syntax...")
    signal(native-compiler-error ("disp-table.el" "Debugger entered--Lisp error: (invalid-read-syntax..."))
    comp--native-compile("disp-table.el")
    batch-native-compile(t)
    eval((batch-native-compile t) t)
    command-line-1(("--eval" "(setq load-prefer-newer t)" "-l" "comp" "-f" "byte-compile-refresh-preloaded" "--eval" "(batch-native-compile t)" "disp-table.el"))
    command-line()
    normal-top-level()

Re: The emacs-28 release branch has been created

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Mon, 04 Oct 2021 17:25:44 +0300
>> From: Eli Zaretskii <eliz@gnu.org>
>> Cc: emacs-devel@gnu.org
>> 
>> > 1. ./configure --with-native-compilation && make
>> > 
>> > 2. make clean
>> > 
>> > 3. make
>> 
>> Right.  Andrea, could you please look into this as soon as you can?
>> This currently blocks the pretest, because the build from the source
>> tarball fails.
>
> To make reproduction easier, this command succeeds, when issued from
> the lisp/ directory:
>
>   $ make disp-table.elc THEFILE=disp-table.el V=1 LISP_PRELOADED=disp-table.el
>   EMACSLOADPATH= '../src/emacs' -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)'  \
> 	  -l comp -f byte-compile-refresh-preloaded \
> 	  -f batch-byte+native-compile disp-table.el
>
> while this command fails:
>
>   $ make disp-table.eln THEFILE=disp-table.el V=1
>   EMACSLOADPATH= '../src/emacs' -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)'  \
> 	  -l comp -f byte-compile-refresh-preloaded \
> 	  --eval '(batch-native-compile t)' disp-table.el
>   Debugger entered--Lisp error: (native-compiler-error "disp-table.el" "Debugger entered--Lisp error: (invalid-read-syntax...")

[...]

thanks for the reduced reproducer, I will start having a look this
evening.

BR

  Andrea

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
> Date: Mon, 04 Oct 2021 14:45:21 +0000
> From:  Andrea Corallo via "Emacs development discussions." <emacs-devel@gnu.org>
> 
> thanks for the reduced reproducer, I will start having a look this
> evening.

Thanks.  To clarify: this is in the emacs-28 branch, not on master
(although on master things should be very similar for now).

Re: The emacs-28 release branch has been created

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
>> Date: Mon, 04 Oct 2021 14:45:21 +0000
>> From:  Andrea Corallo via "Emacs development discussions." <emacs-devel@gnu.org>
>>
>> thanks for the reduced reproducer, I will start having a look this
>> evening.
>
> Thanks.  To clarify: this is in the emacs-28 branch, not on master
> (although on master things should be very similar for now).

Yes, I'm on emacs-28 and I can reproduce.

Quick hint this is a reduced reproducer for disp-table.el

==============
(defun foo ()
  "\e(0")
==============

akrl@xxx:~/emacs2/lisp$ ../src/emacs -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l comp -f byte-compile-refresh-preloaded -f ba\
tch-byte+native-compile disp-table.el

akrl@xxx:~/emacs2/lisp$ ../src/emacs -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l comp -f byte-compile-refresh-preloaded --eva\
l '(batch-native-compile t)' disp-table.el
Debugger entered--Lisp error: (native-compiler-error "disp-table.el" "Debugger entered--Lisp error: (end-of-file \"/tmp/e...")
  signal(native-compiler-error ("disp-table.el" "Debugger entered--Lisp error: (end-of-file \"/tmp/e..."))
  comp--native-compile("disp-table.el")
  batch-native-compile(t)
  command-line-1(("--eval" "(setq load-prefer-newer t)" "-l" "comp" "-f" "byte-compile-refresh-preloaded" "--eval" "(batch-native-compile t)" "disp-table.el"))
  command-line()
  normal-top-level()
  
Are we are doing something wrong in setting-up the reader when using it
with second command?

Perhaps this is already evident to someone, will look into it further.

  Andrea

Re: The emacs-28 release branch has been created

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
>> Date: Mon, 04 Oct 2021 14:45:21 +0000
>> From:  Andrea Corallo via "Emacs development discussions." <emacs-devel@gnu.org>
>> 
>> thanks for the reduced reproducer, I will start having a look this
>> evening.
>
> Thanks.  To clarify: this is in the emacs-28 branch, not on master
> (although on master things should be very similar for now).
>

Okay I see what's the issue.  In `comp-final' we spawn a child process
to run the final pass or not discriminating on the `byte+native-compile'
var.  This is wrong cause this is not bound when using
`batch-native-compile'.

When spawning the sub-process we print the limple dump in a file and this
gets in this case truncated (still to be understood why) leading to the
error.

Will come with patch after dinner.

BR

  Andrea

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
> Date: Mon, 04 Oct 2021 16:15:35 +0000
> 
> Okay I see what's the issue.  In `comp-final' we spawn a child process
> to run the final pass or not discriminating on the `byte+native-compile'
> var.  This is wrong cause this is not bound when using
> `batch-native-compile'.
> 
> When spawning the sub-process we print the limple dump in a file and this
> gets in this case truncated (still to be understood why) leading to the
> error.
> 
> Will come with patch after dinner.

Thanks.

I'd appreciate if you could later describe how you debugged this
problem and how did you arrive to the root cause.  We desperately need
to become more proficient in debugging problems with
native-compilation, and perhaps develop tools and techniques to
support that.

Re: The emacs-28 release branch has been created

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
>> Date: Mon, 04 Oct 2021 16:15:35 +0000
>> 
>> Okay I see what's the issue.  In `comp-final' we spawn a child process
>> to run the final pass or not discriminating on the `byte+native-compile'
>> var.  This is wrong cause this is not bound when using
>> `batch-native-compile'.
>> 
>> When spawning the sub-process we print the limple dump in a file and this
>> gets in this case truncated (still to be understood why) leading to the
>> error.
>> 
>> Will come with patch after dinner.
>
> Thanks.

Attached.  Ok for emacs-28?

> I'd appreciate if you could later describe how you debugged this
> problem and how did you arrive to the root cause.  We desperately need
> to become more proficient in debugging problems with
> native-compilation, and perhaps develop tools and techniques to
> support that.

Sure even tho in this case the process was not much sophisticated :)

- I reduced the reproducer searching manually by bissection in the file
  for the function causing the ICE

- Once identified the function I removed pieces of it to make it the
  smallest function still ICing the compiler

- I read carefully the stack trace of the compiler (after having posted
  it here :/ ) and saw that the reader error was not on file being
  compiled but on the file that we use to drive the native compilation
  happening in the subprocess (the one generated by `comp-final').

- I looked into one of these files and I saw clearly in the limple dump
  a constant vector that is truncated (not sure why maybe we have to
  bind some other print-something var in comp final?).

- Wondered why we are trying to spawn a child process for running the
  final pass if we are doing a batch compilation and found the issue
  reading the code.


Regards

  Andrea

Re: The emacs-28 release branch has been created

[Andrea Corallo via Emacs development discussions.]

[-- Attachment #1: Type: text/plain, Size: 800 bytes --]

Andrea Corallo via "Emacs development discussions."
<emacs-devel@gnu.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Andrea Corallo <akrl@sdf.org>
>>> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
>>> Date: Mon, 04 Oct 2021 16:15:35 +0000
>>> 
>>> Okay I see what's the issue.  In `comp-final' we spawn a child process
>>> to run the final pass or not discriminating on the `byte+native-compile'
>>> var.  This is wrong cause this is not bound when using
>>> `batch-native-compile'.
>>> 
>>> When spawning the sub-process we print the limple dump in a file and this
>>> gets in this case truncated (still to be understood why) leading to the
>>> error.
>>> 
>>> Will come with patch after dinner.
>>
>> Thanks.
>
> Attached.  Ok for emacs-28?

Attached for real...  Apologies

Andrea


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: 0001-Fix-batch-native-compile-not-to-spawn-a-subprocess.patch --]
[-- Type: text/x-diff, Size: 2193 bytes --]

From d3f96619660000ff37663c8f9717ff2f620ab2a6 Mon Sep 17 00:00:00 2001
From: Andrea Corallo <akrl@sdf.org>
Date: Mon, 4 Oct 2021 21:15:02 +0200
Subject: [PATCH] * Fix `batch-native-compile' not to spawn a subprocess

* lisp/emacs-lisp/comp.el (comp-running-batch-compilation): New var.
(comp-final): Use it.
(batch-native-compile): Bind `comp-running-batch-compilation' it.
---
 lisp/emacs-lisp/comp.el | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/lisp/emacs-lisp/comp.el b/lisp/emacs-lisp/comp.el
index 94753ec3d9..63d4a74b54 100644
--- a/lisp/emacs-lisp/comp.el
+++ b/lisp/emacs-lisp/comp.el
@@ -3653,6 +3653,9 @@ comp-final1
 (defvar comp-async-compilation nil
   "Non-nil while executing an asynchronous native compilation.")
 
+(defvar comp-running-batch-compilation nil
+  "Non-nil when compilation is driven by any `batch-*-compile' function.")
+
 (defun comp-final (_)
   "Final pass driving the C back-end for code emission."
   (maphash #'comp-compute-function-type (comp-ctxt-funcs-h comp-ctxt))
@@ -3661,7 +3664,7 @@ comp-final
     ;; unless during bootstrap or async compilation (bug#45056).  GCC
     ;; leaks memory but also interfere with the ability of Emacs to
     ;; detect when a sub-process completes (TODO understand why).
-    (if (or byte+native-compile comp-async-compilation)
+    (if (or comp-running-batch-compilation comp-async-compilation)
 	(comp-final1)
       ;; Call comp-final1 in a child process.
       (let* ((output (comp-ctxt-output comp-ctxt))
@@ -4202,9 +4205,10 @@ batch-native-compile
 will be placed under the native-lisp/ directory (actually, in the
 last directory in `native-comp-eln-load-path')."
   (comp-ensure-native-compiler)
-  (let ((native-compile-target-directory
-         (if for-tarball
-             (car (last native-comp-eln-load-path)))))
+  (let ((comp-running-batch-compilation t)
+        (native-compile-target-directory
+            (if for-tarball
+                (car (last native-comp-eln-load-path)))))
     (cl-loop for file in command-line-args-left
              if (or (null byte+native-compile)
                     (cl-notany (lambda (re) (string-match re file))
-- 
2.20.1

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Eli Zaretskii <eliz@gnu.org>, kbrown@cornell.edu
> Date: Mon, 04 Oct 2021 19:40:50 +0000
> 
> >>> Will come with patch after dinner.
> >>
> >> Thanks.
> >
> > Attached.  Ok for emacs-28?

Yes, thanks.

Thanks for the description of the debugging, I will read it soon.

Re: The emacs-28 release branch has been created

[Ken Brown]

On 10/4/2021 3:40 PM, Andrea Corallo wrote:
> Attached for real... Apologies

That fixes it.  Thanks.

Ken

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
> Date: Mon, 04 Oct 2021 19:38:43 +0000
> 
> - I reduced the reproducer searching manually by bissection in the file
>   for the function causing the ICE

So the problem caused an ICE in libgccjit, is that right?  Is that
always so when the error message says

  Lisp error: (native-compiler-error ...

?  Because the error message then said "(invalid-read-syntax...",
which sounds like a message coming from Emacs, not from the compiler.

Also, we probably want to avoid the ellipsis in batch invocations,
because (unlike in interactive usage) they cannot be expanded.  Is
that something that comp.el controls, or is that a general "feature"
of Emacs?

> - I read carefully the stack trace of the compiler (after having posted
>   it here :/ ) and saw that the reader error was not on file being
>   compiled but on the file that we use to drive the native compilation
>   happening in the subprocess (the one generated by `comp-final').

How did you see that?  The backtrace says:

> akrl@xxx:~/emacs2/lisp$ ../src/emacs -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l comp -f byte-compile-refresh-preloaded --eva\
> l '(batch-native-compile t)' disp-table.el
> Debugger entered--Lisp error: (native-compiler-error "disp-table.el" "Debugger entered--Lisp error: (end-of-file \"/tmp/e...")
>   signal(native-compiler-error ("disp-table.el" "Debugger entered--Lisp error: (end-of-file \"/tmp/e..."))
>   comp--native-compile("disp-table.el")
>   batch-native-compile(t)
>   command-line-1(("--eval" "(setq load-prefer-newer t)" "-l" "comp" "-f" "byte-compile-refresh-preloaded" "--eval" "(batch-native-compile t)" "disp-table.el"))
>   command-line()
>   normal-top-level()

So it only says "end of file", and the file name is not spelled out.

And btw, this backtrace is different from what I saw originally: that
one didn't say "end of file".  Why this one did?

> - I looked into one of these files and I saw clearly in the limple dump
>   a constant vector that is truncated (not sure why maybe we have to
>   bind some other print-something var in comp final?).

I'd like to understand why stuff gets truncated in that case.

Thanks.

Re: The emacs-28 release branch has been created

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
>> Date: Mon, 04 Oct 2021 19:38:43 +0000
>> 
>> - I reduced the reproducer searching manually by bissection in the file
>>   for the function causing the ICE
>
> So the problem caused an ICE in libgccjit, is that right?  Is that
> always so when the error message says
>
>   Lisp error: (native-compiler-error ...
>
> ?  Because the error message then said "(invalid-read-syntax...",
> which sounds like a message coming from Emacs, not from the compiler.

No we ICE in the lisp side of the native compiler.  Actually we ICEd in
the sub process when this tried to read the input lisp file we generate
(as truncated).

> Also, we probably want to avoid the ellipsis in batch invocations,
> because (unlike in interactive usage) they cannot be expanded.  Is
> that something that comp.el controls, or is that a general "feature"
> of Emacs?

The second.  Agree on the benefit of avoiding the ellipsis or making the
information longer befor truncation.

>> - I read carefully the stack trace of the compiler (after having posted
>>   it here :/ ) and saw that the reader error was not on file being
>>   compiled but on the file that we use to drive the native compilation
>>   happening in the subprocess (the one generated by `comp-final').
>
> How did you see that?  The backtrace says:
>
>> akrl@xxx:~/emacs2/lisp$ ../src/emacs -batch --no-site-file --no-site-lisp --eval '(setq load-prefer-newer t)' -l comp -f byte-compile-refresh-preloaded --eva\
>> l '(batch-native-compile t)' disp-table.el
>> Debugger entered--Lisp error: (native-compiler-error "disp-table.el" "Debugger entered--Lisp error: (end-of-file \"/tmp/e...")
>>   signal(native-compiler-error ("disp-table.el" "Debugger entered--Lisp error: (end-of-file \"/tmp/e..."))
>>   comp--native-compile("disp-table.el")
>>   batch-native-compile(t)
>>   command-line-1(("--eval" "(setq load-prefer-newer t)" "-l" "comp" "-f" "byte-compile-refresh-preloaded" "--eval" "(batch-native-compile t)" "disp-table.el"))
>>   command-line()
>>   normal-top-level()
>
> So it only says "end of file", and the file name is not spelled out.

As name it reports /tmp/e... so it really looked like one of the
temporary input files we generate for child compilations and certainly
not the file I was trying to compile.

> And btw, this backtrace is different from what I saw originally: that
> one didn't say "end of file".  Why this one did?

Dunno for sure but I just guess the truncation happend in a different
point of the buffer.

>> - I looked into one of these files and I saw clearly in the limple dump
>>   a constant vector that is truncated (not sure why maybe we have to
>>   bind some other print-something var in comp final?).
>
> I'd like to understand why stuff gets truncated in that case.

Sure that's still the open question.

Best Regards

  Andrea

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
> Date: Mon, 04 Oct 2021 16:15:35 +0000
> 
> Okay I see what's the issue.  In `comp-final' we spawn a child process
> to run the final pass or not discriminating on the `byte+native-compile'
> var.  This is wrong cause this is not bound when using
> `batch-native-compile'.

Can you explain (or guess) why this caused the specific problem it did
(i.e. truncation of some temporary file, AFAIU), and why only in that
single .el file?

Also, your fix introduces a new special variable for that, so I guess
it means the problem is not specific to all batch native-compilations,
only to some?  If so, what is special in byte+native-compile and
batch-native-compile that they require not to spawn a child process?

Thanks.

Re: The emacs-28 release branch has been created

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
>> Date: Mon, 04 Oct 2021 16:15:35 +0000
>> 
>> Okay I see what's the issue.  In `comp-final' we spawn a child process
>> to run the final pass or not discriminating on the `byte+native-compile'
>> var.  This is wrong cause this is not bound when using
>> `batch-native-compile'.
>
> Can you explain (or guess) why this caused the specific problem it did
> (i.e. truncation of some temporary file, AFAIU), and why only in that
> single .el file?

No idea so far, needs further investigation.

> Also, your fix introduces a new special variable for that, so I guess
> it means the problem is not specific to all batch native-compilations,
> only to some?

I guess the problem affects all compilations where we spawn a child
process to compile and where we compile something like the reproducer
where we have (probably) a very long const vector(?).  I'll report when
I have more details.

> If so, what is special in byte+native-compile and
> batch-native-compile that they require not to spawn a child process?

Nothing special, we used to discriminate if we wanted to spawn a child
process using `byte+native-compile', but this worked only for
byte+native-compile' and not for `batch-native-compile'.  For both these
two function we do not want to spawn a child process as its really not
necessary, so my fix.

Best Regards

  Andrea

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
> Date: Tue, 05 Oct 2021 15:37:15 +0000
> 
> > If so, what is special in byte+native-compile and
> > batch-native-compile that they require not to spawn a child process?
> 
> Nothing special, we used to discriminate if we wanted to spawn a child
> process using `byte+native-compile', but this worked only for
> byte+native-compile' and not for `batch-native-compile'.  For both these
> two function we do not want to spawn a child process as its really not
> necessary, so my fix.

I guess I'm asking whether all batch-mode native-compilations need
this, or only these two functions?  If it's true for any batch-mode
compilation, then we already have a variable to test for that.

Re: The emacs-28 release branch has been created

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Andrea Corallo <akrl@sdf.org>
>> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
>> Date: Tue, 05 Oct 2021 15:37:15 +0000
>> 
>> > If so, what is special in byte+native-compile and
>> > batch-native-compile that they require not to spawn a child process?
>> 
>> Nothing special, we used to discriminate if we wanted to spawn a child
>> process using `byte+native-compile', but this worked only for
>> byte+native-compile' and not for `batch-native-compile'.  For both these
>> two function we do not want to spawn a child process as its really not
>> necessary, so my fix.
>
> I guess I'm asking whether all batch-mode native-compilations need
> this, or only these two functions?

The rationale behind is that when these two function are used we
(likely) know that the session will not be a long standing one as Emacs
is run to perform a compilation and die, therefore we can accept some
memory leakage from GCC.  Not sure we want to do that for any
compilation running in batch mode.  WDYT?

Thanks

  Andrea

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: kbrown@cornell.edu, emacs-devel@gnu.org
> Date: Tue, 05 Oct 2021 16:52:14 +0000
> 
> > I guess I'm asking whether all batch-mode native-compilations need
> > this, or only these two functions?
> 
> The rationale behind is that when these two function are used we
> (likely) know that the session will not be a long standing one as Emacs
> is run to perform a compilation and die, therefore we can accept some
> memory leakage from GCC.  Not sure we want to do that for any
> compilation running in batch mode.  WDYT?

Sounds good enough for now.  We can always change this later, as we
gain more experience.

Thanks.

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Ken Brown <kbrown@cornell.edu>
> Date: Sun, 3 Oct 2021 15:45:43 -0400
> 
> > Could it be that the *.elc files were not in the tarball because the
> > build failed at some point?
> 
> No, they weren't in the tarball because when I built emacs prior to running 
> make-dist, I didn't specify --with-native-compilation.  Currently 
> lisp/Makefile.in has
> 
> ifneq ($(HAVE_NATIVE_COMP),yes)
> compile-targets: $(filter-out ./emacs-lisp/comp-cstr.elc,$(filter-out 
> ./emacs-lisp/comp.elc,$(TARGETS)))
> 
> That seems wrong to me.

Why do you think it's wrong?  The comment before it explains the
reason:

  # TARGETS is set dynamically in the recursive call from 'compile-main'.
  # Do not build comp.el unless necessary not to exceed max-specpdl-size and
  # max-lisp-eval-depth in normal builds.

Re: The emacs-28 release branch has been created

[Ken Brown]

On 10/4/2021 7:37 AM, Eli Zaretskii wrote:
>> No, they weren't in the tarball because when I built emacs prior to running
>> make-dist, I didn't specify --with-native-compilation.  Currently
>> lisp/Makefile.in has
>>
>> ifneq ($(HAVE_NATIVE_COMP),yes)
>> compile-targets: $(filter-out ./emacs-lisp/comp-cstr.elc,$(filter-out
>> ./emacs-lisp/comp.elc,$(TARGETS)))
>>
>> That seems wrong to me.
> 
> Why do you think it's wrong?  The comment before it explains the
> reason:
> 
>    # TARGETS is set dynamically in the recursive call from 'compile-main'.
>    # Do not build comp.el unless necessary not to exceed max-specpdl-size and
>    # max-lisp-eval-depth in normal builds.

Maybe I'm misunderstanding the comment, but if byte-compiling comp.el and 
comp-cstr.el causes max-specpdl-size and max-lisp-eval-depth to be exceeded, 
isn't that a problem for builds with native compilation that needs to be fixed?

If I'm wrong, that's fine, but then it means that the instructions for making a 
tarball have to be changed to ensure that those two files get byte-compiled for 
the tarball.

Ken

Re: The emacs-28 release branch has been created

[Eli Zaretskii]

> Cc: emacs-devel@gnu.org
> From: Ken Brown <kbrown@cornell.edu>
> Date: Mon, 4 Oct 2021 09:11:35 -0400
> 
> >    # TARGETS is set dynamically in the recursive call from 'compile-main'.
> >    # Do not build comp.el unless necessary not to exceed max-specpdl-size and
> >    # max-lisp-eval-depth in normal builds.
> 
> Maybe I'm misunderstanding the comment, but if byte-compiling comp.el and 
> comp-cstr.el causes max-specpdl-size and max-lisp-eval-depth to be exceeded, 
> isn't that a problem for builds with native compilation that needs to be fixed?

No, because the problem doesn't happen in builds with native
compilation, AFAIK.

> If I'm wrong, that's fine, but then it means that the instructions for making a 
> tarball have to be changed to ensure that those two files get byte-compiled for 
> the tarball.

The tarball should be built from Emacs configured with native
compilation, that's true.  Something to mention in make-tarball.txt, I
guess.