Bootstrap Compilation Speed

Bootstrap Compilation Speed

[phillip.lord]

Has anyone else seen a regression in bootstrap compilation speed?

I am currently building Emacs-28 trunk. I am still on the 
bootstrap-emacs phase after half an hour or so and am getting no more 
than 2 or 3 ELC+ELN steps complete per minute, with a parallel build.

I haven't been doing bootstrap builds consistently, so I do not know if 
this has changed recently or whether I am just seeing it for the first 
time, so this is not a very consistent "bug report", rather just to see 
if anyone else is seeing something similar.

Phil

Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> Date: Fri, 14 Jan 2022 11:36:26 +0000
> From: phillip.lord@russet.org.uk
> 
> Has anyone else seen a regression in bootstrap compilation speed?
> 
> I am currently building Emacs-28 trunk. I am still on the 
> bootstrap-emacs phase after half an hour or so and am getting no more 
> than 2 or 3 ELC+ELN steps complete per minute, with a parallel build.

How many parallel jobs did you allow Make to have?  And on what OS is
this?

Also, do you see this slow compilation only for the first few files,
before comp.el is natively-compiled, or even after that?

Re: Bootstrap Compilation Speed

[phillip.lord]

On 2022-01-14 13:21, Eli Zaretskii wrote:
>> Date: Fri, 14 Jan 2022 11:36:26 +0000
>> From: phillip.lord@russet.org.uk
>> 
>> Has anyone else seen a regression in bootstrap compilation speed?
>> 
>> I am currently building Emacs-28 trunk. I am still on the
>> bootstrap-emacs phase after half an hour or so and am getting no more
>> than 2 or 3 ELC+ELN steps complete per minute, with a parallel build.
> 
> How many parallel jobs did you allow Make to have?  And on what OS is
> this?
> 
> Also, do you see this slow compilation only for the first few files,
> before comp.el is natively-compiled, or even after that?


Ubuntu 20.04. I tried both single threaded (so I could see how long 
individual files take) and my normal build (-j 20). Again, based on my 
ad hoc experience rather than any careful measurement, it was for all 
the elc+eln using bootstrap-emacs. Once emacs proper was generated, 
things went rather more quickly and upto my expectations.

It's probably something local to me. I thought to mention it here 
because I think we still do not have good metrics set up for bootstrap 
build speed performance.

Phil

Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> Date: Fri, 14 Jan 2022 11:36:26 +0000
> From: phillip.lord@russet.org.uk
> 
> Has anyone else seen a regression in bootstrap compilation speed?
> 
> I am currently building Emacs-28 trunk. I am still on the 
> bootstrap-emacs phase after half an hour or so and am getting no more 
> than 2 or 3 ELC+ELN steps complete per minute, with a parallel build.

With "make -j4" recompiling natively all the preloaded files and their
dependencies on the emacs-28 branch, I see this:

  . compiling the 16 COMPILE_FIRST files takes 22 min, with comp.el
    being the worst offender: about 20 minutes
  . compiling all the rest of the preloaded files takes an additional
    25 minutes

If someone can compare this with past versions, maybe we could see if
the slowdown is real.

Thanks.

Re: Bootstrap Compilation Speed

[phillip.lord]

On 2022-01-15 09:34, Eli Zaretskii wrote:
>> Date: Fri, 14 Jan 2022 11:36:26 +0000
>> From: phillip.lord@russet.org.uk
>> 
>> Has anyone else seen a regression in bootstrap compilation speed?
>> 
>> I am currently building Emacs-28 trunk. I am still on the
>> bootstrap-emacs phase after half an hour or so and am getting no more
>> than 2 or 3 ELC+ELN steps complete per minute, with a parallel build.
> 
> With "make -j4" recompiling natively all the preloaded files and their
> dependencies on the emacs-28 branch, I see this:
> 
>   . compiling the 16 COMPILE_FIRST files takes 22 min, with comp.el
>     being the worst offender: about 20 minutes
>   . compiling all the rest of the preloaded files takes an additional
>     25 minutes
> 
> If someone can compare this with past versions, maybe we could see if
> the slowdown is real.
> 
> Thanks.

Okay, that's similar to what I am seeing.

I can probably do a little testing over the weekend (while my computer 
is otherwise inactive) to see if it has slowed down.

Phil

Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Hello Phil :-)

phillip.lord@russet.org.uk writes:

> Has anyone else seen a regression in bootstrap compilation speed?
>
> I am currently building Emacs-28 trunk. I am still on the
> bootstrap-emacs phase after half an hour or so and am getting no more
> than 2 or 3 ELC+ELN steps complete per minute, with a parallel build.

Are you doing these builds for Windows?

    Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Bootstrap Compilation Speed

[phillip.lord]

On 2022-01-15 10:29, H. Dieter Wilhelm wrote:
> Hello Phil :-)
> 
> phillip.lord@russet.org.uk writes:
> 
>> Has anyone else seen a regression in bootstrap compilation speed?
>> 
>> I am currently building Emacs-28 trunk. I am still on the
>> bootstrap-emacs phase after half an hour or so and am getting no more
>> than 2 or 3 ELC+ELN steps complete per minute, with a parallel build.
> 
> Are you doing these builds for Windows?
> 


No, I am afraid not.

The performance that I got on Windows builds when I did do it was highly 
variable (between half an hour and three), but I think this was mostly 
the shared CPU infrastructure I was using. I never worked out how to get 
native comp working in a way that I could package it for windows, so the 
builds were all without it.

Phil

Re: Bootstrap Compilation Speed

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 736 bytes --]

Hi :)

On Sat, Jan 15, 2022, 05:32 <phillip.lord@russet.org.uk> wrote:

> On 2022-01-15 10:29, H. Dieter Wilhelm wrote:
> > Hello Phil :-)
> >
> > phillip.lord@russet.org.uk writes:
> >
>
> >
> > Are you doing these builds for Windows?
> >
>
>
> No, I am afraid not.
>
> The performance that I got on Windows builds when I did do it was highly
> variable (between half an hour and three), but I think this was mostly
> the shared CPU infrastructure I was using. I never worked out how to get
> native comp working in a way that I could package it for windows, so the
> builds were all without it.
>

This is my present situation also.  I can build with native comp but not
(yet) found a way to package that version.

Regards,
Corwin

>

[-- Attachment #2: Type: text/html, Size: 1463 bytes --]

Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Corwin Brust <corwin@bru.st> writes:
>  > phillip.lord@russet.org.uk writes:
>  The performance that I got on Windows builds when I did do it was highly 
>  variable (between half an hour and three), but I think this was mostly 
>  the shared CPU infrastructure I was using. I never worked out how to get 
>  native comp working in a way that I could package it for windows, so the 
>  builds were all without it.
>
> This is my present situation also.  I can build with native comp but
> not (yet) found a way to package that version.

I don't understand yet the packaging requirements, is it not possible to
copy additionally the native-lisp/ folder to the package?

So far I could compile (locally) the current pretest 28.0.91 version
--with-native-compilation.

Then I tried to build (with build.zips.sh) a snapshot of the
masterbranch --with-native-compilation on the same system.  And now
./configure is complaining:

  configure: error: Elisp native compiler was requested, but libgccjit was not found.

Does Emacs-29 has different requirements?

   Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Date: Sat, 15 Jan 2022 20:06:18 +0100
> Cc: Emacs developers <emacs-devel@gnu.org>,
>  Phillip Lord <phillip.lord@russet.org.uk>
> 
> Then I tried to build (with build.zips.sh) a snapshot of the
> masterbranch --with-native-compilation on the same system.

I'm not familiar with that script, so I don't think I understand the
implications.

> And now ./configure is complaining:
> 
>   configure: error: Elisp native compiler was requested, but libgccjit was not found.
> 
> Does Emacs-29 has different requirements?

No.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Date: Sat, 15 Jan 2022 20:06:18 +0100

>> Then I tried to build (with build.zips.sh) a snapshot of the
>> masterbranch --with-native-compilation on the same system.
>
> I'm not familiar with that script, so I don't think I understand the
> implications.

Sorry, I meant the script admin/nt/dist-build/build-zips.sh

   $ build-zips.sh -s

>> And now ./configure is complaining:
>>   configure: error: Elisp native compiler was requested, but libgccjit was not found.
>> Does Emacs-29 has different requirements?
>
> No.

OK thank you, for the moment I'll focus on emacs-28

    Dieter

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
> Sorry, I meant the script admin/nt/dist-build/build-zips.sh

I could build Gnu-Emacs-28.0.91 when I used the archived sources

  $ ./configure --with-native-compilation

The difference to the build-zips.sh seems to me some more options from
the script (and that is build from the git tree):

        echo [build] Configuring Emacs
        $REPO_DIR/$BRANCH/configure \
            --without-dbus \
            --without-compress-install \
            --with-native-compilation \
            $CACHE \
            CFLAGS="$CFLAGS"

Could it be that these additional options are spoiling the build?
(CACHE=-C , I think.)

      Dieter

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: corwin@bru.st,  emacs-devel@gnu.org,  phillip.lord@russet.org.uk
> Date: Sun, 16 Jan 2022 12:47:24 +0100
> 
> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
> > Sorry, I meant the script admin/nt/dist-build/build-zips.sh
> 
> I could build Gnu-Emacs-28.0.91 when I used the archived sources
> 
>   $ ./configure --with-native-compilation
> 
> The difference to the build-zips.sh seems to me some more options from
> the script (and that is build from the git tree):
> 
>         echo [build] Configuring Emacs
>         $REPO_DIR/$BRANCH/configure \
>             --without-dbus \
>             --without-compress-install \
>             --with-native-compilation \
>             $CACHE \
>             CFLAGS="$CFLAGS"
> 
> Could it be that these additional options are spoiling the build?
> (CACHE=-C , I think.)

I'd drop the CACHE part, so that the configure script starts from a
clean state.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> I could build Gnu-Emacs-28.0.91 when I used the archived sources
>> 
>>   $ ./configure --with-native-compilation
>> 
>> The difference to the build-zips.sh seems to me some more options from
>> the script (and that is build from the git tree):
>> 
>>         echo [build] Configuring Emacs
>>         $REPO_DIR/$BRANCH/configure \
>>             --without-dbus \
>>             --without-compress-install \
>>             --with-native-compilation \
>>             $CACHE \
>>             CFLAGS="$CFLAGS"
>> 
>> Could it be that these additional options are spoiling the build?
>> (CACHE=-C , I think.)
>
> I'd drop the CACHE part, so that the configure script starts from a
> clean state.

I wouldn't!

CACHE=-C is used only on a snapshot build. These were always built
incrementally from the previous version because I judged that the risk
of a broken build was small compared to the significant time and cost
saving that the incremental build provided. A full build is easy to
achieve by simply deleting the build tree, as build-zips does an
out-of-tree. Reconfiguring requires deleting the config.cache as well.

Full release builds are built from a clean tree. The time saving is less
important there.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>,  corwin@bru.st,
>   emacs-devel@gnu.org
> Date: Mon, 17 Jan 2022 09:39:34 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> I could build Gnu-Emacs-28.0.91 when I used the archived sources
> >> 
> >>   $ ./configure --with-native-compilation
> >> 
> >> The difference to the build-zips.sh seems to me some more options from
> >> the script (and that is build from the git tree):
> >> 
> >>         echo [build] Configuring Emacs
> >>         $REPO_DIR/$BRANCH/configure \
> >>             --without-dbus \
> >>             --without-compress-install \
> >>             --with-native-compilation \
> >>             $CACHE \
> >>             CFLAGS="$CFLAGS"
> >> 
> >> Could it be that these additional options are spoiling the build?
> >> (CACHE=-C , I think.)
> >
> > I'd drop the CACHE part, so that the configure script starts from a
> > clean state.
> 
> I wouldn't!
> 
> CACHE=-C is used only on a snapshot build. These were always built
> incrementally from the previous version because I judged that the risk
> of a broken build was small compared to the significant time and cost
> saving that the incremental build provided.

AFAIU, Dieter said he did more than one build in the same tree for
several different configurations, so I suggested to eliminate the
cache as a possible factor for the problems he saw in the second
build.  Apologies if I misunderstood.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Cc: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>,  corwin@bru.st,
>>   emacs-devel@gnu.org
>> Date: Mon, 17 Jan 2022 09:39:34 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> I could build Gnu-Emacs-28.0.91 when I used the archived sources
>> >> 
>> >>   $ ./configure --with-native-compilation
>> >> 
>> >> The difference to the build-zips.sh seems to me some more options from
>> >> the script (and that is build from the git tree):
>> >> 
>> >>         echo [build] Configuring Emacs
>> >>         $REPO_DIR/$BRANCH/configure \
>> >>             --without-dbus \
>> >>             --without-compress-install \
>> >>             --with-native-compilation \
>> >>             $CACHE \
>> >>             CFLAGS="$CFLAGS"
>> >> 
>> >> Could it be that these additional options are spoiling the build?
>> >> (CACHE=-C , I think.)
>> >
>> > I'd drop the CACHE part, so that the configure script starts from a
>> > clean state.
>> 
>> I wouldn't!
>> 
>> CACHE=-C is used only on a snapshot build. These were always built
>> incrementally from the previous version because I judged that the risk
>> of a broken build was small compared to the significant time and cost
>> saving that the incremental build provided.
>
> AFAIU, Dieter said he did more than one build in the same tree for
> several different configurations, so I suggested to eliminate the
> cache as a possible factor for the problems he saw in the second
> build.  Apologies if I misunderstood.


Oh, sure, that's fine. Just saying that the `CACHE=-C` is there for
mostly good reasons!

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: dieter@duenenhof-wilhelm.de,  corwin@bru.st,  emacs-devel@gnu.org
> Date: Mon, 17 Jan 2022 14:07:29 +0000
> 
> >> CACHE=-C is used only on a snapshot build. These were always built
> >> incrementally from the previous version because I judged that the risk
> >> of a broken build was small compared to the significant time and cost
> >> saving that the incremental build provided.
> >
> > AFAIU, Dieter said he did more than one build in the same tree for
> > several different configurations, so I suggested to eliminate the
> > cache as a possible factor for the problems he saw in the second
> > build.  Apologies if I misunderstood.
> 
> 
> Oh, sure, that's fine. Just saying that the `CACHE=-C` is there for
> mostly good reasons!

Of course.  I'm a happy user of -C myself.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: Phillip Lord <phillip.lord@russet.org.uk>
>>> Cc: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>,  corwin@bru.st,
>>>   emacs-devel@gnu.org
>>> Date: Mon, 17 Jan 2022 09:39:34 +0000
>>> 
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>> 
>>> >> I could build Gnu-Emacs-28.0.91 when I used the archived sources
>>> >> 
>>> >>   $ ./configure --with-native-compilation
>>> >> 
>>> >> The difference to the build-zips.sh seems to me some more options from
>>> >> the script (and that is build from the git tree):
>>> >> 
>>> >>         echo [build] Configuring Emacs
>>> >>         $REPO_DIR/$BRANCH/configure \
>>> >>             --without-dbus \
>>> >>             --without-compress-install \
>>> >>             --with-native-compilation \
>>> >>             $CACHE \
>>> >>             CFLAGS="$CFLAGS"
>>> >> 
>>> >> Could it be that these additional options are spoiling the build?
>>> >> (CACHE=-C , I think.)
>>> >
>>> > I'd drop the CACHE part, so that the configure script starts from a
>>> > clean state.
>>> 
>>> I wouldn't!
>>> 
>>> CACHE=-C is used only on a snapshot build. These were always built
>>> incrementally from the previous version because I judged that the risk
>>> of a broken build was small compared to the significant time and cost
>>> saving that the incremental build provided.
>>
>> AFAIU, Dieter said he did more than one build in the same tree for
>> several different configurations, so I suggested to eliminate the
>> cache as a possible factor for the problems he saw in the second
>> build.  Apologies if I misunderstood.
>
>
> Oh, sure, that's fine. Just saying that the `CACHE=-C` is there for
> mostly good reasons!

I'm too inexperienced to judge the options.

But, for the moment, it seems to me when the $CACHE line is active then
there is NO native compilation going on for my builds.  (No idea why,
I'll try to confirm this.)

The current idea is to build 28.0.91 with build-zips.sh - in the usual
way but --with-native compilation - and then copy the lisp-native/
folder into the archive.  Then checking if such a build is running and
actually loading the .eln files on a Windows system without MSYS/MinGW.

Hmm, how can I test if Emacs is actually loading some .eln and not the
.elc files?

       Dieter


-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 496 bytes --]

On Mon, Jan 17, 2022, 10:25 H. Dieter Wilhelm <dieter@duenenhof-wilhelm.de>
wrote:

>
> Hmm, how can I test if Emacs is actually loading some .eln and not the
> .elc files?
>


I have been using describe-function (C-h f) for this  It can tell us if a
given command has been natively compiled (or byte-compiled or neither of
these) for the present session.

Aside: I would first check `native-comp-available-p' to confirm the build
has the native compiler and that the Emacs instance is using it.

[-- Attachment #2: Type: text/html, Size: 938 bytes --]

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Date: Mon, 17 Jan 2022 17:25:00 +0100
> Cc: Eli Zaretskii <eliz@gnu.org>, corwin@bru.st, emacs-devel@gnu.org
> 
> Hmm, how can I test if Emacs is actually loading some .eln and not the
> .elc files?

Start Process Explorer, and look at the DLLs loaded by Emacs: you
should see there all the *.eln files Emacs has currently loaded into
the session.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

>>>> I wouldn't!
>>>> 
>>>> CACHE=-C is used only on a snapshot build. These were always built
>>>> incrementally from the previous version because I judged that the risk
>>>> of a broken build was small compared to the significant time and cost
>>>> saving that the incremental build provided.
>>>
>>> AFAIU, Dieter said he did more than one build in the same tree for
>>> several different configurations, so I suggested to eliminate the
>>> cache as a possible factor for the problems he saw in the second
>>> build.  Apologies if I misunderstood.
>>
>>
>> Oh, sure, that's fine. Just saying that the `CACHE=-C` is there for
>> mostly good reasons!
>
> I'm too inexperienced to judge the options.
>
> But, for the moment, it seems to me when the $CACHE line is active then
> there is NO native compilation going on for my builds.  (No idea why,
> I'll try to confirm this.)
>
> The current idea is to build 28.0.91 with build-zips.sh - in the usual
> way but --with-native compilation - and then copy the lisp-native/
> folder into the archive.  Then checking if such a build is running and
> actually loading the .eln files on a Windows system without MSYS/MinGW.
>
> Hmm, how can I test if Emacs is actually loading some .eln and not the
> .elc files?


If you change the configure options in build-zips.sh, I would go for a
full clean build for sure. "make extraclean" I think will do the trick,
then run build-zips.sh with modified options.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

>> The current idea is to build 28.0.91 with build-zips.sh - in the usual
>> way but --with-native compilation - and then copy the lisp-native/
>> folder into the archive.  Then checking if such a build is running and
>> actually loading the .eln files on a Windows system without MSYS/MinGW.
>>
>> Hmm, how can I test if Emacs is actually loading some .eln and not the
>> .elc files?
>
>
> If you change the configure options in build-zips.sh, I would go for a
> full clean build for sure. "make extraclean" I think will do the trick,
> then run build-zips.sh with modified options.

Thank you Phil

And when we are at it, please tell me your recipe how to build a
snapshot of the master branch.  I think my build tree is damaged, I
don't manage to build a snapshot of emacs-29 with a date included..

Shouldn't it be just:

$ build-zips.sh -g
$ build-zips.sh -s

      Dieter

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

>> If you change the configure options in build-zips.sh, I would go for a
>> full clean build for sure. "make extraclean" I think will do the trick,
>> then run build-zips.sh with modified options.
>
> Thank you Phil
>
> And when we are at it, please tell me your recipe how to build a
> snapshot of the master branch.  I think my build tree is damaged, I
> don't manage to build a snapshot of emacs-29 with a date included..
>
> Shouldn't it be just:
>
> $ build-zips.sh -g
> $ build-zips.sh -s
>


Yes, I think that should do it.

Then, the I build the source like so.

./make-dist --snapshot --no-check --no-changelog
rename .tar _`date +%Y-%m-%d`.tar emacs*.tar.gz
mv emacs-*.tar.gz ~/emacs-upload

I don't think I ever automated this bit.

The model of build-zips.sh is probably a bit complicated and could be
simplified. The addition of snapshots and the ability to build from a
branch makes it much more so and the documentation isn't always well
done. I have been trying to remember why I called the variable
"OF_VERSION" so, or what "OF" stands for, and have failed!

Apologies for this.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
>
>>> If you change the configure options in build-zips.sh, I would go for a
>>> full clean build for sure. "make extraclean" I think will do the trick,
>>> then run build-zips.sh with modified options.
>>
>> Thank you Phil
>>
>> And when we are at it, please tell me your recipe how to build a
>> snapshot of the master branch.  I think my build tree is damaged, I
>> don't manage to build a snapshot of emacs-29 with a date included..
>>
>> Shouldn't it be just:
>>
>> $ build-zips.sh -g
>> $ build-zips.sh -s
>
> Yes, I think that should do it.
>
> Then, the I build the source like so.
>
> ./make-dist --snapshot --no-check --no-changelog
> rename .tar _`date +%Y-%m-%d`.tar emacs*.tar.gz
> mv emacs-*.tar.gz ~/emacs-upload
>
> I don't think I ever automated this bit.

Uups, I thought this - or similar - would be done by below part of
build-zips.sh?


    make -j 4 $INSTALL_TARGET \
         prefix=$HOME/emacs-build/install/emacs-$VERSION
    cd $HOME/emacs-build/install/emacs-$VERSION
    zip -r -9 emacs-$OF_VERSION-no-deps.zip *
    mv emacs-$OF_VERSION-no-deps.zip $HOME/emacs-upload

    if [ -z $SNAPSHOT ];
    then
        DEPS_FILE=$HOME/emacs-build/deps/emacs-$MAJOR_VERSION-deps.zip
    else
        ## Pick the most recent snapshot whatever that is
        DEPS_FILE=`ls $HOME/emacs-build/deps/emacs-$MAJOR_VERSION-*-deps.zip | tail -n 1`
    fi

    echo [build] Using $DEPS_FILE
    unzip -d bin $DEPS_FILE

    zip -r -9 emacs-$OF_VERSION.zip *
    mv emacs-$OF_VERSION.zip ~/emacs-upload

> The model of build-zips.sh is probably a bit complicated and could be
> simplified. The addition of snapshots and the ability to build from a
> branch makes it much more so and the documentation isn't always well
> done. I have been trying to remember why I called the variable
> "OF_VERSION" so, or what "OF" stands for, and have failed!

Not only above is confusing, but also why is

    INSTALL_TARGET="install-strip"

in the script?  (Back to pretest building) I've got an error for

$ build-zips.sh -V28

in the emacs-28 worktree, maybe, that is a wrong usage of the scrip?
Anyway, "install-strip" seems not to be a valid target for make!?

> Apologies for this.

No reason to apologise, it was working for you.  Rather, thank you for
your work. :-)

   Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  corwin@bru.st,  emacs-devel@gnu.org
> Date: Tue, 18 Jan 2022 20:37:53 +0100
> 
> Anyway, "install-strip" seems not to be a valid target for make!?

It is not?  How do you see that?

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  corwin@bru.st,  emacs-devel@gnu.org
>> Date: Tue, 18 Jan 2022 20:37:53 +0100
>> 
>> Anyway, "install-strip" seems not to be a valid target for make!?
>
> It is not?  How do you see that?

The console returns an error:

   make: *** No rule to make target 'install-strip'. Stop

But only on a certain machine, on the other I don't see this error.
Need to start from scratch with a new build tree, I think..

On a positive note, but mysteriously enough for me, I found in a package
archive - build with build-zips.sh - a native-lisp/ folder without my
ado (under 'lib/emacs/28.0.91/)!?  And I checked an Emacs from such a
build on an "MSYS Windows system" - with the Process Explorer - loading
.eln files. :-)

Unfortunately this build didn't work on another Windows computer.  It
seems that there are not yet all necessary dependencies in the distro
archive (for example libgmp-10.dll is missing).  I hope Corwin can bring
some light into the build system.

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

On Tue, Jan 18, 2022 at 3:36 PM H. Dieter Wilhelm
<dieter@duenenhof-wilhelm.de> wrote:
> Unfortunately this build didn't work on another Windows computer.  It
> seems that there are not yet all necessary dependencies in the distro
> archive (for example libgmp-10.dll is missing).  I hope Corwin can bring
> some light into the build system.

We shall see >:-]

I was able make ./build-deps.zip create a version that (appears as if
it) supports native-comp.

New versions are uploaded to my same Sourcehut repo in a new folder
called "with-native-compilation":

  https://git.sr.ht/~mplscorwin/emacs-w64/tree/master/item/with-native-compilation

I'd be quite interested to learn how these work (or don't) for others.

The hackery I found "successful" is presented at the end of this
message.  In brief, what solved the problem for me was remove -static
from CFLAGS.  You can ignore the swapping of $CACHE and CFLAGS; this
simply made it easier to comment out $CACHE while debugging.

Before removing -static I was running into this:

  #./configure --with-native-compilation CFLAGS="-O2 -static"
  [...snip.. lots of the usual ...]
  checking for gcc_jit_context_acquire in -lgccjit... no
  configure: error: ELisp native compiler was requested, but libgccjit
was not found.
  Please try installing libgccjit or a similar package.
  If you are sure you want Emacs be compiled without ELisp native compiler,
  pass the --without-native-compilation option to configure.

After removing -static build-zips.sh works fine.

I have to assume that "just dumping" the -static flag isn't TRT, so
trying to understand the above error message, code producing it, etc.
will be my next move.  Once I understand that I hope we (I, but feel
free) can work up patches to make building with native compilation the
default when building windows distributables, get the msys pathing
corrected once and for all, and (assuming anything is necessary) fix
configure &ct. to make -static work under msys when
--with-native-compilation is specified.  Notwithstanding -static,
assuming that files linked above are not Just Exactly Perfect then I
am probably understating the remaining work to get back to a "turn
key" windows packaging process.

Here is my "fix" for build-zips:

diff --git a/admin/nt/dist-build/build-zips.sh
b/admin/nt/dist-build/build-zips.sh
index 77d20a5a7b..30131b5202 100755
--- a/admin/nt/dist-build/build-zips.sh
+++ b/admin/nt/dist-build/build-zips.sh
@@ -48,12 +48,13 @@
         echo [build] Configuring Emacs
         $REPO_DIR/$BRANCH/configure \
             --without-dbus \
+            --with-native-compilation \
             --without-compress-install \
-            $CACHE \
-            CFLAGS="$CFLAGS"
+            CFLAGS="$CFLAGS" \
+            $CACHE
     fi

-    make -j 4 $INSTALL_TARGET \
+    make -j 20 $INSTALL_TARGET \
          prefix=$HOME/emacs-build/install/emacs-$VERSION
     cd $HOME/emacs-build/install/emacs-$VERSION
     zip -r -9 emacs-$OF_VERSION-no-deps.zip *
@@ -96,7 +97,8 @@ BUILD=
 BUILD_64=1
 GIT_UP=0
 CONFIG=1
-CFLAGS="-O2 -static"
+#CFLAGS="-O2 -static"
+CFLAGS="-O2"
 INSTALL_TARGET="install-strip"

 ## The location of the git repo

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Tue, 18 Jan 2022 18:02:21 -0600
> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs developers <emacs-devel@gnu.org>, 
> 	Phillip Lord <phillip.lord@russet.org.uk>
> 
> The hackery I found "successful" is presented at the end of this
> message.  In brief, what solved the problem for me was remove -static
> from CFLAGS.  You can ignore the swapping of $CACHE and CFLAGS; this
> simply made it easier to comment out $CACHE while debugging.

The -static flag was there for a reason, so you cannot just remove it,
you will probably need to do something else in addition.

AFAIR, the -static switch was used due to the libwinpthreads
dependency, and is related to the no-deps package.  We should resolve
this issue now, because using -static is IMO a time bomb, and it
sounds like it just went off on us.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

On Wed, Jan 19, 2022 at 1:46 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Corwin Brust <corwin@bru.st>
> > Date: Tue, 18 Jan 2022 18:02:21 -0600
> > Cc: Eli Zaretskii <eliz@gnu.org>, Emacs developers <emacs-devel@gnu.org>,
> >       Phillip Lord <phillip.lord@russet.org.uk>
> >
> > The hackery I found "successful" is presented at the end of this
> > message.  In brief, what solved the problem for me was remove -static
> > from CFLAGS.  You can ignore the swapping of $CACHE and CFLAGS; this
> > simply made it easier to comment out $CACHE while debugging.
>
> The -static flag was there for a reason, so you cannot just remove it,
> you will probably need to do something else in addition.

I expected as much.

> AFAIR, the -static switch was used due to the libwinpthreads
> dependency, and is related to the no-deps package.  We should resolve
> this issue now, because using -static is IMO a time bomb, and it
> sounds like it just went off on us.

Makes sense.   How can I help?

One other thing I found is that -installer.exe requires libgmp-10.dll
but either ..
1. only when Emacs is build --with-native-compilation or else
2. only when -static isn't present
I have no idea which, so far.

Looking forward to reviewing/researching further after work today.

As Dieter said (in the other thread) -- I too would very much
appreciate any tips you can share WRT your preferred/personal approach
to crafting the windows binaries/zips.  Thanks so much for that offer.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Wed, 19 Jan 2022 08:26:20 -0600
> Cc: Phillip Lord <phillip.lord@russet.org.uk>, 
> 	"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>, Emacs developers <emacs-devel@gnu.org>
> 
> > AFAIR, the -static switch was used due to the libwinpthreads
> > dependency, and is related to the no-deps package.  We should resolve
> > this issue now, because using -static is IMO a time bomb, and it
> > sounds like it just went off on us.
> 
> Makes sense.   How can I help?

I hope Phillip will chime in and tell us the details of that
conundrum.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Corwin Brust <corwin@bru.st>
>> Date: Tue, 18 Jan 2022 18:02:21 -0600
>> Cc: Eli Zaretskii <eliz@gnu.org>, Emacs developers <emacs-devel@gnu.org>, 
>> 	Phillip Lord <phillip.lord@russet.org.uk>
>> 
>> The hackery I found "successful" is presented at the end of this
>> message.  In brief, what solved the problem for me was remove -static
>> from CFLAGS.  You can ignore the swapping of $CACHE and CFLAGS; this
>> simply made it easier to comment out $CACHE while debugging.
>
> The -static flag was there for a reason, so you cannot just remove it,
> you will probably need to do something else in addition.
>
> AFAIR, the -static switch was used due to the libwinpthreads
> dependency, and is related to the no-deps package.  We should resolve
> this issue now, because using -static is IMO a time bomb, and it
> sounds like it just went off on us.


Yep, libwinpthreads was the problem.

It only affected the no-dependency version of Emacs which has other than
libXpm, no dependencies. Without -static it broke complaining about
libwinpthreads.

Nasty hack. Well worth checking whether it is still a problem (it comes
from msys2 and that's changed lots). And if it is, worth wondering
whether no-deps is still needed.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Hi Corwin 

Corwin Brust <corwin@bru.st> writes:
> New versions are uploaded to my same Sourcehut repo in a new folder
> called "with-native-compilation":
>
>   https://git.sr.ht/~mplscorwin/emacs-w64/tree/master/item/with-native-compilation

Great, I'll test it later on a "clean" Windows machine (and check your
recipes).

Thanks a lot and till later

       Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Hi Corwin

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> Hi Corwin 
>
> Corwin Brust <corwin@bru.st> writes:
>> New versions are uploaded to my same Sourcehut repo in a new folder
>> called "with-native-compilation":
>>
>>   https://git.sr.ht/~mplscorwin/emacs-w64/tree/master/item/with-native-compilation
>
> Great, I'll test it later on a "clean" Windows machine

First I checked your signature.

At first I couldn't find your key on a keyserver, then I found a hint on
your sourcehut site.  Didn't you made your key public?  And is it
necessary to limit the key duration?

On an MSYS Machine I could start Emacs from your archive.  But there are
constant warnings:

  Warning (comp): e:\Corwing\emacs-28.0.91_eln\bin\libgccjit-0.dll: error: error invoking gcc driver. Disable showing Disable logging
  Warning (comp): e:/Corwing/emacs-28.0.91_eln/share/emacs/28.0.91/lisp/url/url-vars.el: Error: Internal native compiler error failed to compile. Disable showing Disable logging
  Warning (comp): e:\Corwing\emacs-28.0.91_eln\bin\libgccjit-0.dll: error: error invoking gcc driver Disable showing Disable logging
  Warning (comp): e:/Corwing/emacs-28.0.91_eln/share/emacs/28.0.91/lisp/password-cache.el: Error: Internal native compiler error failed to compile Disable showing Disable logging
  Warning (comp): e:\Corwing\emacs-28.0.91_eln\bin\libgccjit-0.dll: error: error invoking gcc driver Disable showing Disable logging
  Warning (comp): e:/Corwing/emacs-28.0.91_eln/share/emacs/28.0.91/lisp/emacs-lisp/eieio.el: Error: Internal native compiler error failed to compile Disable showing Disable logging
  ...

Sadly on a "pristine" Windows machine I couldn't start Emacs from the
unzipped archive!  Your build has the same problem than mine.

  libgmp-10.dll

and probably more, are missing in the tree.  It seems that either
something is wrong with build-dep-zips.py. or not all .DLLs are copied
into the archive.  (Your installer I didn't check..)

        Dieter

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

On Wed, Jan 19, 2022 at 12:59 PM H. Dieter Wilhelm
<dieter@duenenhof-wilhelm.de> wrote:
>
> First I checked your signature.
>
> At first I couldn't find your key on a keyserver, then I found a hint on
> your sourcehut site.  Didn't you made your key public?  And is it
> necessary to limit the key duration?

My key is on Savannah, as I assume you've discovered.  I think this is
the expected/required to upload to (either) FTP location.  AFAIK
having *some* duration is required but I haven't looked into that; I
simply accepted the defaults when creating it some time back.

> On an MSYS Machine I could start Emacs from your archive.  But there are
[snip, oops; my bad]

> Sadly on a "pristine" Windows machine I couldn't start Emacs from the
> unzipped archive!  Your build has the same problem than mine.
>
>   libgmp-10.dll
>
> and probably more, are missing in the tree.  It seems that either
> something is wrong with build-dep-zips.py. or not all .DLLs are copied

I believe we just need to add this to the make-deps-zips.py (see the
big lists at the top of that program); I'm not sure yet which of these
to edit.  In any case, I'm /fairly/ confident this is the only "gap".

> into the archive.  (Your installer I didn't check..)

Apologies; only the installer is likely to work because that one I
re-created (build-zips.sh -i) after adding manually adding
llibgmp-10.dll into the deps zip file.

I'll fix up make-deps-zips.py once I can confirm that the
("hand-corrected") installer actually works for someone.  I'm so glad
you have a clean windows environment to use for this testing, and
thanks so much for checking my work.  I wish I could return the favor;
my windows environment isn't anything like sanitary.

Can I ask you to give my install.exe a whirl and confirm that libgmp
was (as I believe) the only issue?  I'll rebuild the full set
(including correcting build-deps-zips.py) as soon as I have that
confirmation.

Yay for progress Deeter; can't wait to work more on this in my evening!

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Corwin Brust <corwin@bru.st> writes:
> Can I ask you to give my install.exe a whirl and confirm that libgmp
> was (as I believe) the only issue?  I'll rebuild the full set
> (including correcting build-deps-zips.py) as soon as I have that
> confirmation.

Ah I didn't expect that your installer had some "extra", nice. :-) Sure,
I'll give it a try this evening.

Please tell me if the uninstaller can remove all remnants of an Emacs
installation, or do I have to clean some parts of the registry or
whatever?

Thank you

      Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

On Thu, Jan 20, 2022 at 1:12 AM H. Dieter Wilhelm
<dieter@duenenhof-wilhelm.de> wrote:
>
> Corwin Brust <corwin@bru.st> writes:
> > Can I ask you to give my install.exe a whirl and confirm that libgmp
> > was (as I believe) the only issue?  I'll rebuild the full set
> > (including correcting build-deps-zips.py) as soon as I have that
> > confirmation.
>
> Ah I didn't expect that your installer had some "extra", nice. :-) Sure,
> I'll give it a try this evening.

Thanks so much Dieter!

> Please tell me if the uninstaller can remove all remnants of an Emacs
> installation, or do I have to clean some parts of the registry or
> whatever?

If we find it isn't removing something it creates I think we should
open bug-reports for that specifically.

I'd been forgetting my daughter's birthday yesterday morning when
planning my day/evening; tonight I have other commitments. I look
forward to getting back into this and a few other things Emacsian
Friday or over the weekend.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Corwin Brust <corwin@bru.st> writes:

> On Thu, Jan 20, 2022 at 1:12 AM H. Dieter Wilhelm
> <dieter@duenenhof-wilhelm.de> wrote:
>>
>> Corwin Brust <corwin@bru.st> writes:
>> > Can I ask you to give my install.exe a whirl and confirm that libgmp
>> > was (as I believe) the only issue?  I'll rebuild the full set
>> > (including correcting build-deps-zips.py) as soon as I have that
>> > confirmation.

I'm sorry to say but the problem with the missing libgmp-10.dll remains
also in your installer.

I checked the Emacs-27.2 installer as well as the zip archive and both
are working as expected on this machine.

>> Please tell me if the uninstaller can remove all remnants of an Emacs
>> installation, or do I have to clean some parts of the registry or
>> whatever?
>
> If we find it isn't removing something it creates I think we should
> open bug-reports for that specifically.

Some minor observiations:

1. The uninstaller isn't removing a link to runemacs.exe.

2. The installer has the Gnu(s)-Emacs image aspect ratio rather
   distorted.

3. The installer default is "c:\Program Files\Emacs" I think it would be
   cleaner when the installation tree includes the version number.


   Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Emacs developers <emacs-devel@gnu.org>,
>   Phillip Lord <phillip.lord@russet.org.uk>
> Date: Thu, 20 Jan 2022 23:56:01 +0100
> 
> 3. The installer default is "c:\Program Files\Emacs" I think it would be
>    cleaner when the installation tree includes the version number.

Why?  The Emacs installation tree is designed to accommodate more than
a single Emacs version simultaneously.  Using versioned directories
would mean users need to tweak their PATH each time they install a new
version, which is undesirable and unnecessary.

What are your reasons for using versioned installation trees?

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  Emacs developers <emacs-devel@gnu.org>,
>>   Phillip Lord <phillip.lord@russet.org.uk>
>> Date: Thu, 20 Jan 2022 23:56:01 +0100
>> 
>> 3. The installer default is "c:\Program Files\Emacs" I think it would be
>>    cleaner when the installation tree includes the version number.
>
> Why?  The Emacs installation tree is designed to accommodate more than
> a single Emacs version simultaneously.  Using versioned directories
> would mean users need to tweak their PATH each time they install a new
> version, which is undesirable and unnecessary.

Stupid me, I meant Emacs' release numbers not "version" numbers!

At the moment Emacs will be installed per default into

c:\
  |- Program Files\
     |- Emacs

If you intent to install another Emacs release and forget to change the
installation directory "Emacs" then - without warning - the new release
will be installed over the old one.  Is this supposed to work for major
release numbers?

Wouldn't it be better to install Emacs in a tree with the release
numbers?

c:\
  |- Program Files\
     |- Emacs-27.1
     |- Emacs-27.2
     |- Emacs-28.0.91


-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: corwin@bru.st,  emacs-devel@gnu.org,  phillip.lord@russet.org.uk
> Date: Fri, 21 Jan 2022 12:07:40 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> 3. The installer default is "c:\Program Files\Emacs" I think it would be
> >>    cleaner when the installation tree includes the version number.
> >
> > Why?  The Emacs installation tree is designed to accommodate more than
> > a single Emacs version simultaneously.  Using versioned directories
> > would mean users need to tweak their PATH each time they install a new
> > version, which is undesirable and unnecessary.
> 
> Stupid me, I meant Emacs' release numbers not "version" numbers!
> 
> At the moment Emacs will be installed per default into
> 
> c:\
>   |- Program Files\
>      |- Emacs
> 
> If you intent to install another Emacs release and forget to change the
> installation directory "Emacs" then - without warning - the new release
> will be installed over the old one.

No, it won't be installed "over the old one", it will be installed
_in_addition_to_ the old one.  the old Emacs binary will still be
available, and all the support files will still be there.

Why did you say "over the old one", and what did you think would
happen when such an installation is done?  I think you might be having
an inaccurate mental model of what happens when a new Emacs version is
installed "over the old one".

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: corwin@bru.st,  emacs-devel@gnu.org,  phillip.lord@russet.org.uk
>> Date: Fri, 21 Jan 2022 12:07:40 +0100
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> 3. The installer default is "c:\Program Files\Emacs" I think it would be
>> >>    cleaner when the installation tree includes the version number.
>> >
>> > Why?  The Emacs installation tree is designed to accommodate more than
>> > a single Emacs version simultaneously.  Using versioned directories
>> > would mean users need to tweak their PATH each time they install a new
>> > version, which is undesirable and unnecessary.
>> 
>> Stupid me, I meant Emacs' release numbers not "version" numbers!
>> 
>> At the moment Emacs will be installed per default into
>> 
>> c:\
>>   |- Program Files\
>>      |- Emacs
>> 
>> If you intent to install another Emacs release and forget to change the
>> installation directory "Emacs" then - without warning - the new release
>> will be installed over the old one.
>
> No, it won't be installed "over the old one", it will be installed
> _in_addition_to_ the old one.  the old Emacs binary will still be

Ah, I was always wondering why there is an emacs.exe AND an
emacs-27.2.exe. :-)

> available, and all the support files will still be there.

It'll work for the libexec\ and share\ branches.

But I'm afraid, currently, not for all files in the bin\ branch, they'll
be overwritten after a default (side-by-side?)  installation.  For
example runemacs.exe and emacsclient.exe are then only working for the
last installed release.

For Emacs-27 there are more than 150 .exe files, moreover .dll and
scripts in bin\.  Maybe this will work if one is installing ever newer
releases but in the opposite case, I'm not sure..

(But I guess few Windows users intend to run multiple Emacs releases,
just using the latest and greatest.)

By the way.  Would you mind to tell me why an emacsclientw.exe together
with emacsclient.exe is needed and what emacs.pdmp is for?

> Why did you say "over the old one", and what did you think would
> happen when such an installation is done?  I think you might be having
> an inaccurate mental model of what happens when a new Emacs version is
> installed "over the old one".

I admit I had a wrong mental model of Emacs' tree organisation.  I think
all software I'm installing on Windows is using a different
approach. :-) Every release is installed in a separate tree.  So I'm
looking forward to learn the advantages of Emacs' organisation.

Thanks a lot

    Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: corwin@bru.st,  phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Fri, 21 Jan 2022 20:22:09 +0100
> 
> It'll work for the libexec\ and share\ branches.
> 
> But I'm afraid, currently, not for all files in the bin\ branch, they'll
> be overwritten after a default (side-by-side?)  installation.  For
> example runemacs.exe and emacsclient.exe are then only working for the
> last installed release.

Why is that a problem?  the old versions can still be invoked, just
not via the default methods.  Which is exactly the expected result
when installing a newer version.

> For Emacs-27 there are more than 150 .exe files, moreover .dll and
> scripts in bin\.  Maybe this will work if one is installing ever newer
> releases but in the opposite case, I'm not sure..

Be sure.  I'm doing this on my system all the time.

> (But I guess few Windows users intend to run multiple Emacs releases,
> just using the latest and greatest.)

Most of them, yes.  But there's no reason to force them to remove the
old version.

> By the way.  Would you mind to tell me why an emacsclientw.exe together
> with emacsclient.exe is needed

That's explained in nt/README.

> and what emacs.pdmp is for?

That's the preloaded packages that are dumped when Emacs is built and
loaded when Emacs is started.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: corwin@bru.st,  phillip.lord@russet.org.uk,  emacs-devel@gnu.org
>> Date: Fri, 21 Jan 2022 20:22:09 +0100
>> 
>> It'll work for the libexec\ and share\ branches.
>> 
>> But I'm afraid, currently, not for all files in the bin\ branch, they'll
>> be overwritten after a default (side-by-side?)  installation.  For
>> example runemacs.exe and emacsclient.exe are then only working for the
>> last installed release.
>
> Why is that a problem?  the old versions can still be invoked, just
> not via the default methods.  Which is exactly the expected result
> when installing a newer version.

True it's possible to run the Emacs-27.1 executable after installing
Emacs-27.2 (checked it).

But I see no way to run an 27.1 release of emacsclientw.exe or
runemacs.exe!  (I don't like this unsightly and useless terminal window
when not starting from runemacs.exe.)

Wouldn't it make sense to provide also emacsclientw-27.1.exe and
consorts?

>> For Emacs-27 there are more than 150 .exe files, moreover .dll and
>> scripts in bin\.  Maybe this will work if one is installing ever newer
>> releases but in the opposite case, I'm not sure..
>
> Be sure.  I'm doing this on my system all the time.

Just copied an Emacs-26 tree "over" Emacs-27.

Emacs-27.2.exe still seems to work but I've got a bad feeling replacing
newer "support stuff" with older..

By the way, with Emacs-26 and 27 there's an issue with the network
manager when running M-x list-packages (certificate expired or not
verified).

After that, I've take the last Emacs snapshot 28.0.50 for Windows and
copied it "over" above tree.  Now the certificate issue is gone.

By implication, this means if I'm "ironing" Emacs-27 over Emacs-28 I'll
get a problem.  As I wanted to say, it is probably safer doing
side-by-side installations from older to newer releases and never the
other way around.

>> (But I guess few Windows users intend to run multiple Emacs releases,
>> just using the latest and greatest.)
>
> Most of them, yes.  But there's no reason to force them to remove the
> old version.

I agree

>> By the way.  Would you mind to tell me why an emacsclientw.exe together
>> with emacsclient.exe is needed
>
> That's explained in nt/README.

Thank you, but I need to run emacsclient on Windows to understand it
fully.

>> and what emacs.pdmp is for?
>
> That's the preloaded packages that are dumped when Emacs is built and
> loaded when Emacs is started.

I see, not all necessary data must be packed into one executable, for
example emacs.exe, thanks.

  Dieter

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: corwin@bru.st,  phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Sat, 22 Jan 2022 11:30:29 +0100
> 
> But I see no way to run an 27.1 release of emacsclientw.exe or
> runemacs.exe!  (I don't like this unsightly and useless terminal window
> when not starting from runemacs.exe.)
> 
> Wouldn't it make sense to provide also emacsclientw-27.1.exe and
> consorts?

I don't see any reason, because we don't change the server protocol in
backward-incompatible ways.

> >> For Emacs-27 there are more than 150 .exe files, moreover .dll and
> >> scripts in bin\.  Maybe this will work if one is installing ever newer
> >> releases but in the opposite case, I'm not sure..
> >
> > Be sure.  I'm doing this on my system all the time.
> 
> Just copied an Emacs-26 tree "over" Emacs-27.
> 
> Emacs-27.2.exe still seems to work but I've got a bad feeling replacing
> newer "support stuff" with older..

IME, there's no real reason for you to feel bad.

> By implication, this means if I'm "ironing" Emacs-27 over Emacs-28 I'll
> get a problem.

If you downgrade Emacs, then yes, you get to live with old problems
that are already fixed in a newer Emacs.  Why is this a surprise?  You
also get back all the bugs that were fixed between Emacs 27 and Emacs
28, right?  Would you call that a problem with the installation
procedure or with the structure of the installation tree?

> As I wanted to say, it is probably safer doing side-by-side
> installations from older to newer releases and never the other way
> around.

Like I said, there's no reason to think so.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>>> By the way.  Would you mind to tell me why an emacsclientw.exe together
>>> with emacsclient.exe is needed
>>
>> That's explained in nt/README.
>
> Thank you, but I need to run emacsclient on Windows to understand it
> fully.

When using the executables to open files, it seems, emacsclientw.exe is
to emacsclient.exe what runemacs.exe is to emacs.exe (with the former no
additional terminal windows are popping up :-) ).

Thanks again

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: corwin@bru.st,  emacs-devel@gnu.org,  phillip.lord@russet.org.uk
> Date: Tue, 25 Jan 2022 20:45:33 +0100
> 
> When using the executables to open files, it seems, emacsclientw.exe is
> to emacsclient.exe what runemacs.exe is to emacs.exe (with the former no
> additional terminal windows are popping up :-) ).

Not exactly.  emacsclientw is the no-wait variant of emacsclient, not
a don't-show-console variant.  IOW, emacsclientw is appropriate for
using from GUI programs that don't (and cannot) wait for Emacs.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: corwin@bru.st,  emacs-devel@gnu.org,  phillip.lord@russet.org.uk
>> Date: Tue, 25 Jan 2022 20:45:33 +0100
>> 
>> When using the executables to open files, it seems, emacsclientw.exe is
>> to emacsclient.exe what runemacs.exe is to emacs.exe (with the former no
>> additional terminal windows are popping up :-) ).
>
> Not exactly.  emacsclientw is the no-wait variant of emacsclient, not
> a don't-show-console variant.  IOW, emacsclientw is appropriate for
> using from GUI programs that don't (and cannot) wait for Emacs.

Thank you for the details. (From nt/README I understood it only as a
don't-show-console.)

  + emacsclientw.exe - A version of emacsclient that does not open
    a command-line window.

--
    Dieter

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

>> Why?  The Emacs installation tree is designed to accommodate more than
>> a single Emacs version simultaneously.  Using versioned directories
>> would mean users need to tweak their PATH each time they install a new
>> version, which is undesirable and unnecessary.
>
> Stupid me, I meant Emacs' release numbers not "version" numbers!
>
> At the moment Emacs will be installed per default into
>
> c:\
>   |- Program Files\
>      |- Emacs
>
> If you intent to install another Emacs release and forget to change the
> installation directory "Emacs" then - without warning - the new release
> will be installed over the old one.  Is this supposed to work for major
> release numbers?
>
> Wouldn't it be better to install Emacs in a tree with the release
> numbers?
>
> c:\
>   |- Program Files\
>      |- Emacs-27.1
>      |- Emacs-27.2
>      |- Emacs-28.0.91


This is the NSIS installer I wrote? It should include a version number,
but I think with a common top level directory (I can't test it at the
moment).

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
>
>>> Why?  The Emacs installation tree is designed to accommodate more than
>>> a single Emacs version simultaneously.  Using versioned directories
>>> would mean users need to tweak their PATH each time they install a new
>>> version, which is undesirable and unnecessary.
>>
>> Stupid me, I meant Emacs' release numbers not "version" numbers!
>>
>> At the moment Emacs will be installed per default into
>>
>> c:\
>>   |- Program Files\
>>      |- Emacs
>>
>> If you intent to install another Emacs release and forget to change the
>> installation directory "Emacs" then - without warning - the new release
>> will be installed over the old one.  Is this supposed to work for major
>> release numbers?
>>
>> Wouldn't it be better to install Emacs in a tree with the release
>> numbers?
>>
>> c:\
>>   |- Program Files\
>>      |- Emacs-27.1
>>      |- Emacs-27.2
>>      |- Emacs-28.0.91
>
>
> This is the NSIS installer I wrote? It should include a version number,
> but I think with a common top level directory (I can't test it at the
> moment).

Yes Phil, I checked it with 27.1 and 27.1. Both releases are installed
into the same tree:

c:\
  |- Program Files\
     |- Emacs\

(without manual intervention).

The release number appears under the share\ folder.

c:\
  |- Program Files\
     |- Emacs\
        |- x86_64\
           |- share\
              |- emacs\
                 |- 27.1
                 |- 27.2

                 Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  corwin@bru.st,  emacs-devel@gnu.org
> Date: Fri, 21 Jan 2022 13:51:28 +0100
> 
> The release number appears under the share\ folder.
> 
> c:\
>   |- Program Files\
>      |- Emacs\
>         |- x86_64\
>            |- share\
>               |- emacs\
>                  |- 27.1
>                  |- 27.2

You should also see a versioned subdirectory under libexec.  (And with
Emacs 28, also under lib.)

This is normal, and has nothing to do with the Windows installer.
It's how the Emacs installation tree looks on all platforms.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[phillip.lord]

On 2022-01-21 12:59, Eli Zaretskii wrote:
>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  corwin@bru.st,  emacs-devel@gnu.org
>> Date: Fri, 21 Jan 2022 13:51:28 +0100
>> 
>> The release number appears under the share\ folder.
>> 
>> c:\
>>   |- Program Files\
>>      |- Emacs\
>>         |- x86_64\
>>            |- share\
>>               |- emacs\
>>                  |- 27.1
>>                  |- 27.2
> 
> You should also see a versioned subdirectory under libexec.  (And with
> Emacs 28, also under lib.)
> 
> This is normal, and has nothing to do with the Windows installer.
> It's how the Emacs installation tree looks on all platforms.

Hmmm.

As you say, installing multiple versions side-by-side works fine and 
does not need to have
a version number in the install directory. ("C:\Program Files\Emacs"). 
Actually, the "x86_64" is
not needed now, but I guess I left that in for backward compatibility.

But, the uninstall section looks like this:

Section "Uninstall"

   # Always delete uninstaller first
   Delete "$INSTDIR\Uninstall.exe"

   # now delete installed directory
   RMDir /r "$INSTDIR"
   RMDir "$INSTDIR"

   !insertmacro MUI_STARTMENU_GETFOLDER Application $StartMenuFolder

   Delete "$SMPROGRAMS\$StartMenuFolder\Uninstall.lnk"
   RMDir "$SMPROGRAMS\$StartMenuFolder"
SectionEnd



So, I think there will only be a single uninstaller. Uninstalling will 
remove all versions of Emacs because it just deletes the INSTDIR. 
Likewise, there is a single start menu item which will overwrite the 
last installed Emacs.I guess this makes sense. In windows land, 
installing Emacs-28 would be expected to remove Emacs-27. If all older 
versions of Emacs get left behind, that would generally be considered a 
bug. I don't remember whether NSIS runs the previous uninstaller first, 
that might be checked.

So, side-by-side installation will work but it's not by design and while 
the installer supports it, the uninstaller will not respect it.

My feeling is, therefore, the current set up is correct. If you want to 
run multiple versions of Emacs, use the zip file.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> Date: Fri, 21 Jan 2022 17:48:31 +0000
> From: phillip.lord@russet.org.uk
> Cc: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>, corwin@bru.st,
>  emacs-devel@gnu.org
> 
> Actually, the "x86_64" is
> not needed now, but I guess I left that in for backward compatibility.

Why do you think the architecture subdirectory is not needed?

> But, the uninstall section looks like this:
> 
> Section "Uninstall"
> 
>    # Always delete uninstaller first
>    Delete "$INSTDIR\Uninstall.exe"
> 
>    # now delete installed directory
>    RMDir /r "$INSTDIR"
>    RMDir "$INSTDIR"
> 
>    !insertmacro MUI_STARTMENU_GETFOLDER Application $StartMenuFolder
> 
>    Delete "$SMPROGRAMS\$StartMenuFolder\Uninstall.lnk"
>    RMDir "$SMPROGRAMS\$StartMenuFolder"
> SectionEnd
> 
> So, I think there will only be a single uninstaller. Uninstalling will 
> remove all versions of Emacs because it just deletes the INSTDIR. 

Then that is the problem to fix: how Emacs is uninstalled.

Or maybe it's a feature that it uninstalls all the versions?

In general, I see no reason for anyone to ever uninstall Emacs.  But
that's me.

> In windows land, installing Emacs-28 would be expected to remove
> Emacs-27.

We are in Emacs-land, not in Windows-land.  Emacs on Windows tries to
behave like on any other platform, from this POV.

> If all older versions of Emacs get left behind, that would generally
> be considered a bug.

It is not a bug, it's a feature: old versions are still available
after installing new ones.  If someone reports this as a bug, we will
explain why it isn't.

> So, side-by-side installation will work but it's not by design and while 
> the installer supports it, the uninstaller will not respect it.

Then the uninstaller needs to be fixed.

> My feeling is, therefore, the current set up is correct. If you want to 
> run multiple versions of Emacs, use the zip file.

I disagree.  I think it will be confusing to have different
installation expectations depending on whether a zip file or an
installer were used.  They should both yield the same results.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

On Fri, Jan 21, 2022 at 1:36 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > Date: Fri, 21 Jan 2022 17:48:31 +0000
> > From: phillip.lord@russet.org.uk
> > So, side-by-side installation will work but it's not by design and while
> > the installer supports it, the uninstaller will not respect it.
>
> Then the uninstaller needs to be fixed.

Should the uninstaller detect when several versions are present in the
installation path and then prompt the user to select which ones to
remove?   I assume we would then make removing the installation folder
conditional on the user selected "Remove all versions" or something to
this effect.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Fri, 21 Jan 2022 13:47:26 -0600
> Cc: Phillip Lord <phillip.lord@russet.org.uk>, 
> 	"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>, Emacs developers <emacs-devel@gnu.org>
> 
> > > So, side-by-side installation will work but it's not by design and while
> > > the installer supports it, the uninstaller will not respect it.
> >
> > Then the uninstaller needs to be fixed.
> 
> Should the uninstaller detect when several versions are present in the
> installation path and then prompt the user to select which ones to
> remove?   I assume we would then make removing the installation folder
> conditional on the user selected "Remove all versions" or something to
> this effect.

Ideally, the uninstaller should only remove the files that the
installer installed, it should not remove entire directories if those
directories include files not installed by the installer.

"Remove all version" should be a separate feature, IMO.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Fri, 21 Jan 2022 17:48:31 +0000
>> From: phillip.lord@russet.org.uk
>> So, I think there will only be a single uninstaller. Uninstalling will 
>> remove all versions of Emacs because it just deletes the INSTDIR. 

I can confirm this (for Emacs-27 releases, side-by-side) there's then
only one uninstaller (left?) and it will remove the complete Emacs tree
per default (optionally, I think, it offers to remove parts of the
tree).


-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> Date: Fri, 21 Jan 2022 17:48:31 +0000
>>> From: phillip.lord@russet.org.uk
>>> So, I think there will only be a single uninstaller. Uninstalling will 
>>> remove all versions of Emacs because it just deletes the INSTDIR. 
>
> I can confirm this (for Emacs-27 releases, side-by-side) there's then
> only one uninstaller (left?) and it will remove the complete Emacs tree
> per default (optionally, I think, it offers to remove parts of the
> tree).


Currently, I think, the installer doesn't offer to uninstall the
previous version, although I have only tested it on wine. I would have
assumed it did.

So, we have an half-and-half world at the moment. It doesn't install
multiple versions cleanly, nor does the "update to a new version"
functionality work cleanly. You have to uninstall the old one first.

None of this will result in an unfunctional Emacs, so it's not a
disaster. But I think it should do or the other.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> Actually, the "x86_64" is
>> not needed now, but I guess I left that in for backward compatibility.
>
> Why do you think the architecture subdirectory is not needed?


Because there is no i686 any more. You could install them both together.


>> In windows land, installing Emacs-28 would be expected to remove
>> Emacs-27.
>
> We are in Emacs-land, not in Windows-land.  Emacs on Windows tries to
> behave like on any other platform, from this POV.

Yes, that's true, but Emacs does not ship an installer on any other
platform. The Windows installer has to behave uniquely for that reasons.

Although, I would say that most of the binary packaged versions of Emacs
do the same thing. The debian .deb is a single version (27.1). A new
release will remove the old one.



>> If all older versions of Emacs get left behind, that would generally
>> be considered a bug.
>
> It is not a bug, it's a feature: old versions are still available
> after installing new ones.  If someone reports this as a bug, we will
> explain why it isn't.

You could:-) I'd agree with the bug report. But, as I have stepping back
from making the windows version, I'm easy with which ever way it goes.



>> So, side-by-side installation will work but it's not by design and while 
>> the installer supports it, the uninstaller will not respect it.
>
> Then the uninstaller needs to be fixed.


It doable, although with the directory layout as it is, it is slightly
more complex. If we want to do this, I would add a top level-directory
with the version number in. The uninstaller just deletes this. The start
menu items will need updating also, so there is not just one.


>> My feeling is, therefore, the current set up is correct. If you want to 
>> run multiple versions of Emacs, use the zip file.
>
> I disagree.  I think it will be confusing to have different
> installation expectations depending on whether a zip file or an
> installer were used.  They should both yield the same results.


The zip file doesn't bring any expectations. It just unpacks where every
you want it, and doesn't install any short cuts. You get the same Emacs
but all the directories are clearly different. In that sense, the zip
file is equivalent to a portable app. The installer version is not. The
latter is what most people would be expecting.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: dieter@duenenhof-wilhelm.de,  corwin@bru.st,  emacs-devel@gnu.org
> Date: Sat, 22 Jan 2022 12:40:10 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> >> 
> >> Actually, the "x86_64" is
> >> not needed now, but I guess I left that in for backward compatibility.
> >
> > Why do you think the architecture subdirectory is not needed?
> 
> Because there is no i686 any more. You could install them both together.

In the "official" binaries, there's no i686, because MinGW64 basically
tossed support for that.  But mingw.org's MinGW still supports it, so
a user could theoretically build such an Emacs, or ask someone else to
build it, and then install it alongside the 64-bit one.

And then there could be a Cygwin build, which will be installed into a
separate subdirectory of libexec, even though it's a 64-bit build.

So I think there's a good reason to keep the architecture
subdirectory.

> The zip file doesn't bring any expectations. It just unpacks where every
> you want it, and doesn't install any short cuts. You get the same Emacs
> but all the directories are clearly different. In that sense, the zip
> file is equivalent to a portable app. The installer version is not. The
> latter is what most people would be expecting.

I provided my opinions, but since I'm not the one who will do the
work, feel free to disregard them.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Cc: dieter@duenenhof-wilhelm.de, corwin@bru.st, emacs-devel@gnu.org
>> Date: Sat, 22 Jan 2022 12:40:10 +0000
>> 
>> Because there is no i686 any more. You could install them both
>> together.
>
> In the "official" binaries, there's no i686, because MinGW64 basically
> tossed support for that.  But mingw.org's MinGW still supports it, so
> a user could theoretically build such an Emacs, or ask someone else to
> build it, and then install it alongside the 64-bit one.
>
> And then there could be a Cygwin build, which will be installed into a
> separate subdirectory of libexec, even though it's a 64-bit build.
>
> So I think there's a good reason to keep the architecture
> subdirectory.

If there were such a thing, my guess is that they would install it
somewhere entirely different. But it adds no complexity, so I am easy
either way.


>> The zip file doesn't bring any expectations. It just unpacks where
>> every
>> you want it, and doesn't install any short cuts. You get the same
>> Emacs
>> but all the directories are clearly different. In that sense, the
>> zip
>> file is equivalent to a portable app. The installer version is
>> not. The
>> latter is what most people would be expecting.
>
> I provided my opinions, but since I'm not the one who will do the
> work, feel free to disregard them.


Likewise! I think we have had a good discussion of the pros and
cons. The final decision can rest with Dieter and Corwin.


Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 1950 bytes --]

On Thu, Jan 20, 2022 at 4:56 PM H. Dieter Wilhelm
<dieter@duenenhof-wilhelm.de> wrote:
> Corwin Brust <corwin@bru.st> writes:
> > On Thu, Jan 20, 2022 at 1:12 AM H. Dieter Wilhelm
> > <dieter@duenenhof-wilhelm.de> wrote:
> >> Corwin Brust <corwin@bru.st> writes:

> I'm sorry to say but the problem with the missing libgmp-10.dll remains
> also in your installer.

I found myself up triaging some things for another project I support,
so kicked off a new build.  These include pushes to the emacs-28
branch though today up to approximately 10pm US/Central.  I think
these new versions may do the trick.   A patch-set is attached for
changes to the admin/nt/build-*zips* scripts.  You will see that I'm
still building without -static; I haven't had a chance to dig into
that yet.

  https://git.sr.ht/~mplscorwin/emacs-w64/tree/master/item/with-native-compilation

Would you be able to try again with this version of emacs-28.0.91-installer.exe?
In fact, all of the packages in the linked folder should work now AFAIK.

> >> Please tell me if the uninstaller can remove all remnants of an Emacs
> >> installation, or do I have to clean some parts of the registry or
> >> whatever?
> >
> > If we find it isn't removing something it creates I think we should
> > open bug-reports for that specifically.
>
> Some minor observiations:
>
> 1. The uninstaller isn't removing a link to runemacs.exe.

I think this should get a bug report (assuming it's still happening
with the above-linked versions).

> 2. The installer has the Gnu(s)-Emacs image aspect ratio rather
>    distorted.

This feels like a lower priority but maybe it too should be tracked
with a separate bug report?

> 3. The installer default is "c:\Program Files\Emacs" I think it would be
>    cleaner when the installation tree includes the version number.

I see you are already discussing this one with Eli; I'll leave it to
the two of you as Eli's thoughts appear to echo my own here.

[-- Attachment #2: emacs-28_nt-dist-build-scripts.patch --]
[-- Type: application/octet-stream, Size: 2091 bytes --]

diff --git a/admin/nt/dist-build/build-dep-zips.py b/admin/nt/dist-build/build-dep-zips.py
index c27045a001..974bf456fb 100755
--- a/admin/nt/dist-build/build-dep-zips.py
+++ b/admin/nt/dist-build/build-dep-zips.py
@@ -37,6 +37,7 @@
 mingw-w64-x86_64-librsvg
 mingw-w64-x86_64-libtiff
 mingw-w64-x86_64-libxml2
+mingw-w64-x86_64-gmp
 mingw-w64-x86_64-xpm-nox'''.split()
 
 DLL_REQ='''libgif
@@ -49,6 +50,7 @@
 librsvg
 libtiff
 libxml
+libgmp
 libXpm'''.split()
 
 
@@ -121,7 +123,7 @@ def ntldd_munge(out):
 
 ## Currently no packages seem to require this!
 ARCH_PKGS=[]
-SRC_REPO="https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources"
+SRC_REPO="https://repo.msys2.org/mingw/sources"
 
 
 def immediate_deps(pkg):
@@ -167,7 +169,7 @@ def download_source(tarball):
     if not os.path.exists("../emacs-src-cache/{}".format(tarball)):
         print("Downloading {}...".format(tarball))
         check_output_maybe(
-            "wget -a ../download.log -O ../emacs-src-cache/{} {}/{}/download"
+            "wget -a ../download.log -O ../emacs-src-cache/{} {}/{}"
             .format(tarball, SRC_REPO, tarball),
             shell=True
         )
diff --git a/admin/nt/dist-build/build-zips.sh b/admin/nt/dist-build/build-zips.sh
index 77d20a5a7b..30131b5202 100755
--- a/admin/nt/dist-build/build-zips.sh
+++ b/admin/nt/dist-build/build-zips.sh
@@ -48,12 +48,13 @@
         echo [build] Configuring Emacs
         $REPO_DIR/$BRANCH/configure \
             --without-dbus \
+            --with-native-compilation \
             --without-compress-install \
-            $CACHE \
-            CFLAGS="$CFLAGS"
+            CFLAGS="$CFLAGS" \
+            $CACHE
     fi
 
-    make -j 4 $INSTALL_TARGET \
+    make -j 20 $INSTALL_TARGET \
          prefix=$HOME/emacs-build/install/emacs-$VERSION
     cd $HOME/emacs-build/install/emacs-$VERSION
     zip -r -9 emacs-$OF_VERSION-no-deps.zip *
@@ -96,7 +97,8 @@ BUILD=
 BUILD_64=1
 GIT_UP=0
 CONFIG=1
-CFLAGS="-O2 -static"
+#CFLAGS="-O2 -static"
+CFLAGS="-O2"
 INSTALL_TARGET="install-strip"
 
 ## The location of the git repo

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Hi Corwin

Corwin Brust <corwin@bru.st> writes:

>   https://git.sr.ht/~mplscorwin/emacs-w64/tree/master/item/with-native-compilation
>
> Would you be able to try again with this version of emacs-28.0.91-installer.exe?
> In fact, all of the packages in the linked folder should work now AFAIK.

I can't find libgmp-10.dll in your emacs-28.0.91.zip archive and when I
run the installer - on a system without MSYS/MinGW - it still bails out
because of the missing libgmp-10.dll.


        Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

On Sat, Jan 22, 2022 at 3:51 PM H. Dieter Wilhelm
<dieter@duenenhof-wilhelm.de> wrote:
>
> Hi Corwin

Hi Dieter!

> Corwin Brust <corwin@bru.st> writes:
>
> >   https://git.sr.ht/~mplscorwin/emacs-w64/tree/master/item/with-native-compilation
> >
> > Would you be able to try again with this version of emacs-28.0.91-installer.exe?
> > In fact, all of the packages in the linked folder should work now AFAIK.
>
> I can't find libgmp-10.dll in your emacs-28.0.91.zip archive and when I
> run the installer - on a system without MSYS/MinGW - it still bails out
> because of the missing libgmp-10.dll.

I've pushed new versions.  I was able to test the new installer.exe on
my son's gaming rig, so I'm rather confident that, at least, works.

I hope someone with a working msys can confirm that this supports
native-compilation when msys+libgccjit is available.

Regards,
Corwin

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Hi Corwin :-)

Corwin Brust <corwin@bru.st> writes:

> On Sat, Jan 22, 2022 at 3:51 PM H. Dieter Wilhelm
> <dieter@duenenhof-wilhelm.de> wrote:
>> Corwin Brust <corwin@bru.st> writes:
>> >   https://git.sr.ht/~mplscorwin/emacs-w64/tree/master/item/with-native-compilation
>> >
>> > Would you be able to try again with this version of emacs-28.0.91-installer.exe?
>> > In fact, all of the packages in the linked folder should work now AFAIK.
>>
>> I can't find libgmp-10.dll in your emacs-28.0.91.zip archive and when I
>> run the installer - on a system without MSYS/MinGW - it still bails out
>> because of the missing libgmp-10.dll.
>
> I've pushed new versions.  I was able to test the new installer.exe on
> my son's gaming rig, so I'm rather confident that, at least, works.

So you installed Doom-Emacs? ;-)

In earnest: I could start Emacs from the installer, great. :-)

Then I checked it with M-x list-packages and received the following
*Messages*:

  gnutls.c: [1] (Emacs) GnuTLS library not found [4 times]
  Package refresh done

I don't know how important this is, though.  I'm wondering if we could
automate such build testing with existing ert tests?

Some more observations (no criticism).

Signing: When I check the downloads gpg says:

  Good signature. Warning: This key is not certified with a trusted
  signature!

Would it help, be appear more trustful, when I sign your key?

Size: Your installer and emacs-28.0.91 builds are smaller than Phil's.
I assume you didn't copy the source code?

Installation tree: I've seen that you skipped the x86_64 branch.

> I hope someone with a working msys can confirm that this supports
> native-compilation when msys+libgccjit is available.

I'm sorry, forgot to check it on my MSYS machine, will do tomorrow.

And now for something else: Would you expect it to be problematic when
we build on Windows10 for Windows11 or vice versa?

Have a nice evening

     Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Phillip Lord
>  <phillip.lord@russet.org.uk>,  Emacs developers <emacs-devel@gnu.org>
> Date: Sun, 23 Jan 2022 22:16:00 +0100
> 
> And now for something else: Would you expect it to be problematic when
> we build on Windows10 for Windows11 or vice versa?

No, there should be no problem like that.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

Thanks so much for testing Dieter!

On Sun, Jan 23, 2022 at 3:16 PM H. Dieter Wilhelm
<dieter@duenenhof-wilhelm.de> wrote:
>
> So you installed Doom-Emacs? ;-)

hahahahahahahaha
No.

> Then I checked it with M-x list-packages and received the following
> *Messages*:
>
>   gnutls.c: [1] (Emacs) GnuTLS library not found [4 times]
>   Package refresh done

I'm having trouble replicating this.  I see libgnutls-30.dll (and also
libgnutls-openssl-27.dll?) in both the -deps.zip and inside
with-native-compilation\emacs-28.0.91.zip

Any thoughts on what could be up here?

> I don't know how important this is, though.  I'm wondering if we could
> automate such build testing with existing ert tests?
>
> Some more observations (no criticism).
>
> Signing: When I check the downloads gpg says:
>
>   Good signature. Warning: This key is not certified with a trusted
>   signature!

Hmm.  I've uploaded my key a few places (in addition to posting on
Savannah).  I can look for other places to publish it -- any
suggestions?

>
> Would it help, be appear more trustful, when I sign your key?
>

I don't think I quite understand this suggestion -- not a rejection.
Likely, you are wiser than I in the ways of GPG.

> Size: Your installer and emacs-28.0.91 builds are smaller than Phil's.
> I assume you didn't copy the source code?

I'm fairly sure the binary release packages have never included
source.  I suspect the size difference is accounted for by omitting
-static while compiling.

>
> Installation tree: I've seen that you skipped the x86_64 branch.
>

Can you elaborate?  Is this a difference between my prior versions and
the current, between what Philliip was producing and what I've been
making, or somehow .. both?

> > I hope someone with a working msys can confirm that this supports
> > native-compilation when msys+libgccjit is available.
>
> I'm sorry, forgot to check it on my MSYS machine, will do tomorrow.

Quite alright -- as I mentioned in the new thread[1] from today, I was
able to test the installing to a machine with a working msys (on path)
does enable native compilation to work, so I'm feeling a bit less
pressure on that score.  Don't get me wrong: I'll be very happy when
you are able to take a look.

> And now for something else: Would you expect it to be problematic when
> we build on Windows10 for Windows11 or vice versa?

No, I'm not expecting any differences between Windows 10 and Windows
11 that we'll notice WRT the packaging process we are using.   That's
not a super expert opinion, to be sure.  I can likely give the
installer a try under Windows 11 next week sometime.  I'll report back
directly in case of any supprises.

>
> Have a nice evening

And you!

[1] new thread, summary/TODOs:
https://lists.gnu.org/archive/html/emacs-devel/2022-01/msg01467.html

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Corwin Brust <corwin@bru.st> writes:
> Thanks so much for testing Dieter!

Pleasure :-)

> On Sun, Jan 23, 2022 at 3:16 PM H. Dieter Wilhelm
> <dieter@duenenhof-wilhelm.de> wrote:
>> Signing: When I check the downloads gpg says:
>>
>>   Good signature. Warning: This key is not certified with a trusted
>>   signature!
>
> Hmm.  I've uploaded my key a few places (in addition to posting on
> Savannah).  I can look for other places to publish it -- any
> suggestions?

Thank you 

I think it would be helpful for (some interested) users either to have
it directly at the ftp site (in the gnu-keyring?) or to make / announce
your key know to public keyservers.

It's probably not yet spread on public keyservers.

  $ gpg --keyserver hkp://keyserver.ubuntu.com --search-key corwin@bru.st

You could announce your key on keyservers thus

  $ gpg --keyserver hkp://keyserver.ubuntu.com --send-key corwin@bru.st

I guess only few users will actually check if their download is valid
with the .sig file...  But I remember how annoying it was to search for
a key, when the old key of the developer was expired.

>> Would it help, be appear more trustful, when I sign your key?

> I don't think I quite understand this suggestion -- not a rejection.
> Likely, you are wiser than I in the ways of GPG.

I'm not wiser and most likely this signing of keys isn't worth the
trouble since it seems that Emacs developers just include their keys
into the gnu-keyring and don't build a Web of Trust around it, like, for
example, the Debian project.

>> Size: Your installer and emacs-28.0.91 builds are smaller than Phil's.
>> I assume you didn't copy the source code?
>
> I'm fairly sure the binary release packages have never included
> source.  I suspect the size difference is accounted for by omitting
> -static while compiling.

Ah OK.

>> Installation tree: I've seen that you skipped the x86_64 branch.
>>
>
> Can you elaborate?  Is this a difference between my prior versions and
> the current, between what Philliip was producing and what I've been
> making, or somehow .. both?

Phil's installer (and archives) with a default installation:

c:\
|- Program Files\
   |- Emacs\
      | x86_64\
        |- bin\
        |- ...

Your installer:
c:\
|- Program Files\
   |- Emacs\
      |- bin\
      |- ...


>> > I hope someone with a working msys can confirm that this supports
>> > native-compilation when msys+libgccjit is available.
>>
>> I'm sorry, forgot to check it on my MSYS machine, will do tomorrow.
>
> Quite alright -- as I mentioned in the new thread[1] from today, I was
> able to test the installing to a machine with a working msys (on path)
> does enable native compilation to work, so I'm feeling a bit less
> pressure on that score.  Don't get me wrong: I'll be very happy when
> you are able to take a look.

I didn't comment yet on [1] because I think, in summary, your focus is
well thought out and you don't let yourself be too distracted from my
"firing" on the side-line. :-)

Some observations regarding the native compilation:

After runing Emacs native compilation started immediately till all
el. files were compiled to .eln.  I check with the Process Explorer that
these files are loaded as well.

It seems that you've configured Emacs to do the compilation
"as-soon-as-possible" and not "just-in-time" :-)

Then, unfortunately, for each .eln file (96) there remains an
accompanying .eln.tmp file in the eln-cache!?

When updating some packages (M-x list-packages) these were then native
compiled and respective .eln.tmp files vanished.  (But still not the
.eln file which were compiled in the beginning!)

Bye

  Dieter

> [1] new thread, summary/TODOs:
> https://lists.gnu.org/archive/html/emacs-devel/2022-01/msg01467.html

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Emacs developers <emacs-devel@gnu.org>,
>   Phillip Lord <phillip.lord@russet.org.uk>
> Date: Mon, 24 Jan 2022 16:43:15 +0100
> 
> After runing Emacs native compilation started immediately till all
> el. files were compiled to .eln.

What do you mean by "all" here?

> It seems that you've configured Emacs to do the compilation
> "as-soon-as-possible" and not "just-in-time" :-)

No such configuration.

> Then, unfortunately, for each .eln file (96) there remains an
> accompanying .eln.tmp file in the eln-cache!?

Doesn't happen here: those .eln.tmp files are renamed to .eln at the
end of the compilation.  What version of libgccjit do you have
installed?

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  Emacs developers <emacs-devel@gnu.org>,
>>   Phillip Lord <phillip.lord@russet.org.uk>
>> Date: Mon, 24 Jan 2022 16:43:15 +0100
>> 
>> After runing Emacs native compilation started immediately till all
>> el. files were compiled to .eln.
>
> What do you mean by "all" here?

Since I forgot to check with emacs -Q, I assume (now) all lisp files
which are loaded by my init.el.

When doing emacs -Q nothing is compiled at first, the eln-cache remains
empty.  After loading init.el Emacs is beavering away for some minutes
(4 cores, with virus scanners active). :-/

I can't reproduce the zero size .eln.tmp files anymore.  Maybe that was
an issue with org-mode from an old org-elpa package which I changed in
the meantime from the gnu archive?  That's the only idea which comes to
mind...

When updating packages the native compilation works as well.

So for the moment, everything looks promising! :-)

>> Then, unfortunately, for each .eln file (96) there remains an
>> accompanying .eln.tmp file in the eln-cache!?
>
> Doesn't happen here: those .eln.tmp files are renamed to .eln at the
> end of the compilation.  What version of libgccjit do you have
> installed?

I updated the system quite recently.

$ pacman -Ss libgccjit

mingw64/mingw-w64-x86_64-ligccjit 11.2.0-6 (mingw-w64-86_64-toolchain)
[installed: 10.3.0-2]

            Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: corwin@bru.st,  emacs-devel@gnu.org,  phillip.lord@russet.org.uk
> Date: Mon, 24 Jan 2022 19:28:57 +0100
> 
> >> After runing Emacs native compilation started immediately till all
> >> el. files were compiled to .eln.
> >
> > What do you mean by "all" here?
> 
> Since I forgot to check with emacs -Q, I assume (now) all lisp files
> which are loaded by my init.el.
> 
> When doing emacs -Q nothing is compiled at first, the eln-cache remains
> empty.  After loading init.el Emacs is beavering away for some minutes
> (4 cores, with virus scanners active). :-/

This is normal and expected.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

> [Why personal email?]

There was no purpose behind it, except saving traffic to this long
thread.  Initially I only wanted to confirm your statement.  Then I came
up with the cores vs. logical processors questions.

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Date: Mon, 24 Jan 2022 23:01:04 +0100
>> 
>> >> When doing emacs -Q nothing is compiled at first, the eln-cache remains
>> >> empty.  After loading init.el Emacs is beavering away for some minutes
>> >> (4 cores, with virus scanners active). :-/
>> >
>> > This is normal and expected.
>> 
>> Yes, I was not sceptical about the process, only a bit impatient, this
>> system is "stuffed" to the neck from our corporate IT.  Wouldn't it more
>> "polite" for Emacs if only half the available cores would be taken on
>> Intel systems by default?
>
> The notion of "available cores" is not well defined on a modern
> system.  Emacs uses half of the cores present on the system, and you
> can customize that via native-comp-async-jobs-number, if that's not
> good enough for your system.

Could you please tell me how to measure the compilation time of Emacs
without a stopwatch, how would you do it?  I'm interested in how
compilation time is scaling when I'm using all available logical
processors..

Thank you
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: emacs-devel@gnu.org
> Date: Tue, 25 Jan 2022 11:34:09 +0100
> 
> Could you please tell me how to measure the compilation time of Emacs
> without a stopwatch, how would you do it?  I'm interested in how
> compilation time is scaling when I'm using all available logical
> processors..

What do you mean by "compilation time of Emacs"?  What is the
compilation that you want to time?

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: emacs-devel@gnu.org
>> Date: Tue, 25 Jan 2022 11:34:09 +0100
>> 
>> Could you please tell me how to measure the compilation time of Emacs
>> without a stopwatch, how would you do it?  I'm interested in how
>> compilation time is scaling when I'm using all available logical
>> processors..
>
> What do you mean by "compilation time of Emacs"?  What is the
> compilation that you want to time?

Oh right, I could check the time compiling Emacs simply with

 $ time make -j8

In the thread's context I meant the time it takes with an empty
eln-cache to compile all .eln files after loading the init file.

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: emacs-devel@gnu.org
> Date: Tue, 25 Jan 2022 14:46:57 +0100
> 
> In the thread's context I meant the time it takes with an empty
> eln-cache to compile all .eln files after loading the init file.

I'm not aware of any facilities, except noting the time when the
compilation started (i.e. when you start Emacs), and then looking at
the time stamps of the produced *.eln files.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: emacs-devel@gnu.org
>> Date: Tue, 25 Jan 2022 14:46:57 +0100
>> 
>> In the thread's context I meant the time it takes with an empty
>> eln-cache to compile all .eln files after loading the init file.
>
> I'm not aware of any facilities, except noting the time when the
> compilation started (i.e. when you start Emacs), and then looking at
> the time stamps of the produced *.eln files.

Thanks a lot for the hint!

  $ rm -r eln-cache/
  $ date && emacs
  $ date -r latest/eln-cache/file

That worked for me doing a quick check:

I'm curious about the hyperthreading potential on my 4 core Intel laptop
for doing compilations. And checked Emacs-28.0.91 from Corwin with (setq
native-comp-async-jobs-number 8) vs 4 jobs.  My init.el forces Emacs to
compile over 200 .eln files and it took about 5'40" with 8 logical
processors and 7'7" with 4 logical processors.

--
        Dieter

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: emacs-devel@gnu.org
> Date: Sun, 30 Jan 2022 12:57:04 +0100
> 
>   $ rm -r eln-cache/
>   $ date && emacs
>   $ date -r latest/eln-cache/file
> 
> That worked for me doing a quick check:
> 
> I'm curious about the hyperthreading potential on my 4 core Intel laptop
> for doing compilations. And checked Emacs-28.0.91 from Corwin with (setq
> native-comp-async-jobs-number 8) vs 4 jobs.  My init.el forces Emacs to
> compile over 200 .eln files and it took about 5'40" with 8 logical
> processors and 7'7" with 4 logical processors.

How many execution units you actually have there?

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

[-- Attachment #1: Type: text/plain, Size: 895 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: emacs-devel@gnu.org
>> Date: Sun, 30 Jan 2022 12:57:04 +0100
>> 
>>   $ rm -r eln-cache/
>>   $ date && emacs
>>   $ date -r latest/eln-cache/file
>> 
>> That worked for me doing a quick check:
>> 
>> I'm curious about the hyperthreading potential on my 4 core Intel laptop
>> for doing compilations. And checked Emacs-28.0.91 from Corwin with (setq
>> native-comp-async-jobs-number 8) vs 4 jobs.  My init.el forces Emacs to
>> compile over 200 .eln files and it took about 5'40" with 8 logical
>> processors and 7'7" with 4 logical processors.
>
> How many execution units you actually have there?

I think you mean with "execution units" what the task manager calls
"cores".  This computer has 4 cores and 8 logical processors as you can
see in the attached image from it's task manager.


[-- Attachment #2: task_manager_cores_2022-01-30.png --]
[-- Type: image/png, Size: 2545 bytes --]

[-- Attachment #3: Type: text/plain, Size: 20 bytes --]



--
        Dieter

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: emacs-devel@gnu.org
> Date: Sun, 30 Jan 2022 19:12:25 +0100
> 
> >> I'm curious about the hyperthreading potential on my 4 core Intel laptop
> >> for doing compilations. And checked Emacs-28.0.91 from Corwin with (setq
> >> native-comp-async-jobs-number 8) vs 4 jobs.  My init.el forces Emacs to
> >> compile over 200 .eln files and it took about 5'40" with 8 logical
> >> processors and 7'7" with 4 logical processors.
> >
> > How many execution units you actually have there?
> 
> I think you mean with "execution units" what the task manager calls
> "cores".  This computer has 4 cores and 8 logical processors as you can
> see in the attached image from it's task manager.

If all you have is 8 execution units, using them all for compilation
will indeed not show twice the speed as with 4 jobs, because the OS
needs some of the CPU resources to do its job.

I generally find the default of half the execution units used for
background native compilation as adequate: it lets me work in Emacs
with almost no visible slowdown while JIT compilation goes on.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: emacs-devel@gnu.org
>> Date: Sun, 30 Jan 2022 19:12:25 +0100
>> 
>> >> I'm curious about the hyperthreading potential on my 4 core Intel laptop
>> >> for doing compilations. And checked Emacs-28.0.91 from Corwin with (setq
>> >> native-comp-async-jobs-number 8) vs 4 jobs.  My init.el forces Emacs to
>> >> compile over 200 .eln files and it took about 5'40" with 8 logical
>> >> processors and 7'7" with 4 logical processors.
>> >
>> > How many execution units you actually have there?
>> 
>> I think you mean with "execution units" what the task manager calls
>> "cores".  This computer has 4 cores and 8 logical processors as you can
>> see in the attached image from it's task manager.
>
> If all you have is 8 execution units, using them all for compilation
> will indeed not show twice the speed as with 4 jobs, because the OS
> needs some of the CPU resources to do its job.

Yes, it's true that the OS needs resources, but it's not the major
effect on machines with Intel CPUs (hyperthreading enabled, I don't have
experience with AMD processors.)

Here's the same test for 6 and 3 native-comp-ansync-jobs (4 core CPU)

  6 jobs : 6'12"
  3 jobs : 8'23"

That's roughly what I experience with simulation software as well.  The
heavy lifting is done by cores, the number of "logical processors"
doesn't scale.  (Usually we are doing long running simulation jobs with
the number of cores - 1.)

> I generally find the default of half the execution units used for
> background native compilation as adequate: it lets me work in Emacs
> with almost no visible slowdown while JIT compilation goes on.

I assume for editing tasks it doesn't matter if all cores are used by
compilation and the system load is low.

--
        Dieter

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

Hi Corwin

Made a mistake!

> Phil's installer (and archives) with a default installation:
>
> c:\
> |- Program Files\
>    |- Emacs\
>       | x86_64\
>         |- bin\
>         |- ...

Here's the corrected tree from your installer
> Your installer:
> c:\
  |- Program Files\
     |- Emacs\
        |- emacs-28.0.91\
           |- bin\
           |- ...

(The uninstaler is under Emacs\.)

     Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
>
> Made a mistake!

What's that like?  Never tried it, myself :P

> > Phil's installer (and archives) with a default installation:
> >
> > c:\
> > |- Program Files\
> >    |- Emacs\
> >       | x86_64\
> >         |- bin\
> >         |- ...
>
> Here's the corrected tree from your installer
> > Your installer:
> > c:\
>   |- Program Files\
>      |- Emacs\
>         |- emacs-28.0.91\
>            |- bin\
>            |- ...
>
> (The uninstaler is under Emacs\.)

I'm not sure why this is.  Could you be comparing packages from
Phillip for a full release vs mine for a pre-release?   Other than
that the only thing I can think is that perhaps I'm not executing one
or the other script from the expected folder/CWD.

Phil,  thoughts/theories?

In any case, apart from the pending/on-going discussion of what
changes to make to installer/uninstaller (and then, how critical such
changes should be considered), I feel this is probably not a show
stopper:  So far, I still think we can push the prerelease to alpha
as-is and I'd lean toward not making any change around this prior to
28.1.  WDYT?

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Corwin Brust <corwin@bru.st> writes:

> On Tue, Jan 18, 2022 at 3:36 PM H. Dieter Wilhelm
> <dieter@duenenhof-wilhelm.de> wrote:
>> Unfortunately this build didn't work on another Windows computer.  It
>> seems that there are not yet all necessary dependencies in the distro
>> archive (for example libgmp-10.dll is missing).  I hope Corwin can bring
>> some light into the build system.
>
> We shall see >:-]
>
> I was able make ./build-deps.zip create a version that (appears as if
> it) supports native-comp.
>
> New versions are uploaded to my same Sourcehut repo in a new folder
> called "with-native-compilation":
>
>   https://git.sr.ht/~mplscorwin/emacs-w64/tree/master/item/with-native-compilation
>
> I'd be quite interested to learn how these work (or don't) for others.


In case you haven't see it, there in etc/w32-features.el there are a set
of tests for various features. It's in etc/w32-features.el rather than
the normal test location so that it is present in the windows binary
release. It's a nice way to check whether the build has worked (I use
.emacs on my test machine to automatically load it).

Anyway, you might want to add a test for native comp working to
there. It will make your life easier!

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: corwin@bru.st,  emacs-devel@gnu.org,  phillip.lord@russet.org.uk
> Date: Tue, 18 Jan 2022 22:36:48 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> >> Cc: Eli Zaretskii <eliz@gnu.org>,  corwin@bru.st,  emacs-devel@gnu.org
> >> Date: Tue, 18 Jan 2022 20:37:53 +0100
> >> 
> >> Anyway, "install-strip" seems not to be a valid target for make!?
> >
> > It is not?  How do you see that?
> 
> The console returns an error:
> 
>    make: *** No rule to make target 'install-strip'. Stop

The top-level Makefile.in has that target, so I wonder how did that
happen for you.

> On a positive note, but mysteriously enough for me, I found in a package
> archive - build with build-zips.sh - a native-lisp/ folder without my
> ado (under 'lib/emacs/28.0.91/)!?  And I checked an Emacs from such a
> build on an "MSYS Windows system" - with the Process Explorer - loading
> .eln files. :-)
> 
> Unfortunately this build didn't work on another Windows computer.  It
> seems that there are not yet all necessary dependencies in the distro
> archive (for example libgmp-10.dll is missing).  I hope Corwin can bring
> some light into the build system.

I hate to maybe take you back some steps, but why exactly do you need
those build-zips.sh etc. scripts?  If the problem is to produce a zip
archive with the products of a build (and nothing else), then I can
propose a much simpler solution, which I use for all the ezwinports
binaries.  Interested?

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> The console returns an error:
>> 
>>    make: *** No rule to make target 'install-strip'. Stop
>
> The top-level Makefile.in has that target, so I wonder how did that
> happen for you.

OK, I need to cleanup the whole build tree..

>> archive (for example libgmp-10.dll is missing).  I hope Corwin can bring
>> some light into the build system.
>
> I hate to maybe take you back some steps, but why exactly do you need
> those build-zips.sh etc. scripts?  If the problem is to produce a zip

Because I thought that this is the OFFICIALL way of doing Windows
builds, since this script is under admin/nt/dist-build!  But don't
worry, I want to learn a robust and simple way of distributing
Emacs. :-)

I see myself as a backup of Corwin: First I'd like to build a zip file
with Emacs and its necessary dependencies.

Later configured --with-native-compilation.

And later building an installer out of it.

> archive with the products of a build (and nothing else), then I can
> propose a much simpler solution, which I use for all the ezwinports
> binaries.  Interested?

Yes please.  A step by step approach to building Emacs is probably more
successful in the end.

           Thanks     

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: corwin@bru.st,  phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Wed, 19 Jan 2022 14:57:44 +0100
> 
> > archive with the products of a build (and nothing else), then I can
> > propose a much simpler solution, which I use for all the ezwinports
> > binaries.  Interested?
> 
> Yes please.  A step by step approach to building Emacs is probably more
> successful in the end.

1. First, I build the package (in this case, Emacs) as usual:

  $ ./configure --prefix=...
  $ make

2. Then I install it in a special place:

  $ make install --prefix=/some/directory

3. Then to produce a zip file with all the installed files, I go to
   /some/directory and invoke 'zip':

  $ zip -9r ./bin ./include ./lib ./libexec ./share ./var

(be sure to mention all the top-level directories).

Now, the tricky part is between steps 2 and 3.  To make the zip file
self-contained, you need to populate /some/directory tree with the
dependencies.  If those are DLLs, simply copy all of them into
/some/directory/bin/.  If there are other dependencies, they should go
in their respective subdirectories (but I don't expect to see any
files outside of bin/ and perhaps lib/).

I think part of the complexity of dist-build scripts is that Phillip
included many more dependencies than strictly needed to run Emacs, so
he needed to put there many programs and files not directly related to
Emacs.  I'm not sure that's justified, but YMMV.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: corwin@bru.st,  phillip.lord@russet.org.uk,  emacs-devel@gnu.org
>> Date: Wed, 19 Jan 2022 14:57:44 +0100
>> 
>> > archive with the products of a build (and nothing else), then I can
>> > propose a much simpler solution, which I use for all the ezwinports
>> > binaries.  Interested?
>> 
>> Yes please.  A step by step approach to building Emacs is probably more
>> successful in the end.
>
>
> I think part of the complexity of dist-build scripts is that Phillip
> included many more dependencies than strictly needed to run Emacs, so
> he needed to put there many programs and files not directly related to
> Emacs.  I'm not sure that's justified, but YMMV.


No, it's not that. The dependencies are now fairly small and now all
essential. Besides the dependencies were added simply by unzipping the
dependencies bundle.

The script that produces the dependencies bundle is rather simpler than
it used to be.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

> 1. First, I build the package (in this case, Emacs) as usual:
>   $ ./configure --prefix=...
>   $ make
> 2. Then I install it in a special place:
>   $ make install --prefix=/some/directory
> 3. Then to produce a zip file with all the installed files, I go to
>    /some/directory and invoke 'zip':
>   $ zip -9r ./bin ./include ./lib ./libexec ./share ./var

Why is the folder ./var necessary?

(var/ wasn't installed on my system (28.0.91).)

> (be sure to mention all the top-level directories).  Now, the tricky
> part is between steps 2 and 3.  To make the zip file self-contained,
> you need to populate /some/directory tree with the dependencies.  If
> those are DLLs, simply copy all of them into /some/directory/bin/.  If
> there are other dependencies, they should go

It seems that only

   libgmp-10.dll

is a prerequisite to get Emacs (28.0.91) running on a Windows system
without MSYS and

   libXpm-noX4.dll

provides a nicer logo and icons.  (I didn't test much functionality
yet.)

In /nt/README.W32 is a list of further dependencies provided in MSYS2
packages:

    mingw-w64-x86_64-giflib
    mingw-w64-x86_64-gnutls
    mingw-w64-x86_64-libjpeg-turbo
    mingw-w64-x86_64-libpng
    mingw-w64-x86_64-librsvg
    mingw-w64-x86_64-libtiff
    mingw-w64-x86_64-libxml2
    mingw-w64-x86_64-xpm-nox
    mingw-w64-x86_64-lcms2

are above dependencies considered as the absolut minimal set for a state
of the art Emacs under Windows? 

--
Thank you

      Dieter

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

On Sat, Jan 29, 2022 at 3:32 PM H. Dieter Wilhelm
<dieter@duenenhof-wilhelm.de> wrote:
>
>
>    libXpm-noX4.dll
>
> provides a nicer logo and icons.  (I didn't test much functionality
> yet.)

Thanks! I'll see if that could be a gap somewhere in the build
process.  The windows release builds seem to have a pretty ugly logo
on the splash.

>
> In /nt/README.W32 is a list of further dependencies provided in MSYS2
> packages:
>
>     mingw-w64-x86_64-giflib
>     mingw-w64-x86_64-gnutls
>     mingw-w64-x86_64-libjpeg-turbo
>     mingw-w64-x86_64-libpng
>     mingw-w64-x86_64-librsvg
>     mingw-w64-x86_64-libtiff
>     mingw-w64-x86_64-libxml2
>     mingw-w64-x86_64-xpm-nox
>     mingw-w64-x86_64-lcms2
>

Here is the list from build-deps-zips.  I'm fairly sure at least
harfbuzz and jansson should be added, but I strongly suspect the list
in the script is a good one (at least, I like the emacs release
packages Phillip has been creating).

mingw-w64-x86_64-giflib
mingw-w64-x86_64-gnutls
mingw-w64-x86_64-harfbuzz
mingw-w64-x86_64-jansson
mingw-w64-x86_64-lcms2
mingw-w64-x86_64-libjpeg-turbo
mingw-w64-x86_64-libpng
mingw-w64-x86_64-librsvg
mingw-w64-x86_64-libtiff
mingw-w64-x86_64-libxml2
mingw-w64-x86_64-xpm-nox

Removing the intersection of these gives me:

mingw-w64-x86_64-harfbuzz
mingw-w64-x86_64-jansson

However, when I look a little further down the script at the DLLs that
we'll bundle, I note that xpm-nox is missing from the latter list, so
that probably confirms the change needed per comments above.

I will test that I confirm back.

> are above dependencies considered as the absolut minimal set for a state
> of the art Emacs under Windows?

Do you want to patch the documentation while I further mangle Phillips scripts?

PS, not intending to side track the "simpler path" conversation you
are having with Eli, here, but I do think we should carry what we
learn back to the scripts that appear to do the job now.   In all
events, I suspect jannsson and harbuzz should be documented in
README.W32 in *some* fashion.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Corwin Brust <corwin@bru.st> writes:

> On Sat, Jan 29, 2022 at 3:32 PM H. Dieter Wilhelm
> <dieter@duenenhof-wilhelm.de> wrote:
>>
>>
>>    libXpm-noX4.dll
>>
>> provides a nicer logo and icons.  (I didn't test much functionality
>> yet.)
>
> Thanks! I'll see if that could be a gap somewhere in the build
> process.  The windows release builds seem to have a pretty ugly logo
> on the splash.

Stange, the ugly logo didn't occurred to me with Emacs from your
installer?

(And Phil mentioned this dependency explicitly in
admin/nt/dist-build/README-scripts.)


>> In /nt/README.W32 is a list of further dependencies provided in MSYS2
>> packages:
>>
>>     mingw-w64-x86_64-giflib
>>     mingw-w64-x86_64-gnutls
>>     mingw-w64-x86_64-libjpeg-turbo
>>     mingw-w64-x86_64-libpng
>>     mingw-w64-x86_64-librsvg
>>     mingw-w64-x86_64-libtiff
>>     mingw-w64-x86_64-libxml2
>>     mingw-w64-x86_64-xpm-nox
>>     mingw-w64-x86_64-lcms2
>>
>
> Here is the list from build-deps-zips.  I'm fairly sure at least
> harfbuzz and jansson should be added, but I strongly suspect the list
> in the script is a good one (at least, I like the emacs release
> packages Phillip has been creating).
>
> mingw-w64-x86_64-giflib
> mingw-w64-x86_64-gnutls
> mingw-w64-x86_64-harfbuzz
> mingw-w64-x86_64-jansson
> mingw-w64-x86_64-lcms2
> mingw-w64-x86_64-libjpeg-turbo
> mingw-w64-x86_64-libpng
> mingw-w64-x86_64-librsvg
> mingw-w64-x86_64-libtiff
> mingw-w64-x86_64-libxml2
> mingw-w64-x86_64-xpm-nox
>
> Removing the intersection of these gives me:
>
> mingw-w64-x86_64-harfbuzz
> mingw-w64-x86_64-jansson

Thanks, do you know what jannson is needed for?

> However, when I look a little further down the script at the DLLs that
> we'll bundle, I note that xpm-nox is missing from the latter list, so
> that probably confirms the change needed per comments above.
>
> I will test that I confirm back.
>
>> are above dependencies considered as the absolut minimal set for a state
>> of the art Emacs under Windows?
>
> Do you want to patch the documentation while I further mangle Phillips scripts?
>
> PS, not intending to side track the "simpler path" conversation you
> are having with Eli, here, but I do think we should carry what we
> learn back to the scripts that appear to do the job now.   In all
> events, I suspect jannsson and harbuzz should be documented in
> README.W32 in *some* fashion.

It would be good if you document your changes and when I'm ready I can
proofread them.

I don't see your files on
   https://alpha.gnu.org/gnu/emacs/pretest/windows/emacs-28/

didn't the GNU administration respond to your request yet?

--
       Dieter

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 1571 bytes --]

On Sat, Jan 29, 2022 at 4:23 PM H. Dieter Wilhelm
<dieter@duenenhof-wilhelm.de> wrote:
>
> Corwin Brust <corwin@bru.st> writes:
>
> > On Sat, Jan 29, 2022 at 3:32 PM H. Dieter Wilhelm
> > <dieter@duenenhof-wilhelm.de> wrote:
> >>
> >>
> >>    libXpm-noX4.dll
> >>

> Stange, the ugly logo didn't occurred to me with Emacs from your
> installer?

It's pretty obvious here, but maybe because I have an unusually
colored background in my default theme: the lack of transparency for
the image on the splash page is pretty unsightly.  But maybe I'm
overreacting.  See attached.

>
> (And Phil mentioned this dependency explicitly in
> admin/nt/dist-build/README-scripts.)
>

I seem to have missed that. Thanks, indeed.

> >
> > Do you want to patch the documentation while I further mangle Phillips scripts?
> >
> > PS, not intending to side track the "simpler path" conversation you
> > are having with Eli, here, but I do think we should carry what we
> > learn back to the scripts that appear to do the job now.   In all
> > events, I suspect jannsson and harbuzz should be documented in
> > README.W32 in *some* fashion.
>
> It would be good if you document your changes and when I'm ready I can
> proofread them.

Sure, I'll get to it then.  I'll file a bug and CC you in expressly
when I have a patch.

>
> I don't see your files on
>    https://alpha.gnu.org/gnu/emacs/pretest/windows/emacs-28/
>
> didn't the GNU administration respond to your request yet?

I'm awaiting confirmation that I've done the needful (correctly) and
that my upload permissions are in place.

[-- Attachment #2: GfiUlEUcwj.png --]
[-- Type: image/png, Size: 128587 bytes --]

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Corwin Brust <corwin@bru.st> writes:

> On Sat, Jan 29, 2022 at 4:23 PM H. Dieter Wilhelm
> <dieter@duenenhof-wilhelm.de> wrote:
>>
>> Corwin Brust <corwin@bru.st> writes:
>>
>> > On Sat, Jan 29, 2022 at 3:32 PM H. Dieter Wilhelm
>> > <dieter@duenenhof-wilhelm.de> wrote:
>> >>
>> >>    libXpm-noX4.dll

>> Stange, the ugly logo didn't occurred to me with Emacs from your
>> installer?
>
> It's pretty obvious here, but maybe because I have an unusually
> colored background in my default theme: the lack of transparency for
> the image on the splash page is pretty unsightly.  But maybe I'm
> overreacting.  See attached.

I see, yes, could be nicer if the png were transparent.

(Without libXpm-noX4.dll the image is drawn only in black & white.)


An aside regarding dependencies: When I'm starting Emacs from your
current build from an MSYS2 shell I'm getting this warning:

  gnutls.c: [1] (Emacs) GnuTLS library not found

Even though libgnutls-30.dll, libgnutls-openssl-27.dll and
libgnutlsxx-28.dll are in Emacs' bin/ directory.

This warning doesn't appear, when starting Emacs from an Mingw64 shell.
Sigh, it seems when it was build under MinGW64 it has to be started from
such a shell?

--
        Dieter

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  Phillip Lord
>  <phillip.lord@russet.org.uk>,  Emacs developers <emacs-devel@gnu.org>
> Date: Sun, 30 Jan 2022 11:13:00 +0100
> 
> An aside regarding dependencies: When I'm starting Emacs from your
> current build from an MSYS2 shell I'm getting this warning:
> 
>   gnutls.c: [1] (Emacs) GnuTLS library not found
> 
> Even though libgnutls-30.dll, libgnutls-openssl-27.dll and
> libgnutlsxx-28.dll are in Emacs' bin/ directory.

Check the dependencies of libgnutls-30.dll, perhaps they are not on
PATH.  (They should ideally be in the same directory as
libgnutls-30.dll itself.)

> This warning doesn't appear, when starting Emacs from an Mingw64 shell.
> Sigh, it seems when it was build under MinGW64 it has to be started from
> such a shell?

No, it just means your PATH is different in MinGW64 shell.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Arash Esbati]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> An aside regarding dependencies: When I'm starting Emacs from your
> current build from an MSYS2 shell I'm getting this warning:
>
>   gnutls.c: [1] (Emacs) GnuTLS library not found
>
> Even though libgnutls-30.dll, libgnutls-openssl-27.dll and
> libgnutlsxx-28.dll are in Emacs' bin/ directory.

I'm not tracking this development, so please bear with me.  Is the plan
to unpack the file emacs-28.0.91-no-deps.zip into, say C:\emacs and
unpack the .dll from emacs-28-deps.zip[1] into C:\emacs\bin?  If so, I'm
afraid that the compiled features will not work since dependencies are
missing in emacs-28-deps.zip, e.g.:

1)  libgnutls-30.dll has these dependencies:
Found: C:\path\to\msys64\mingw64\bin\libgnutls-30.dll
C:\path\to\msys64\mingw64\bin\libgnutls-30.dll
  C:\path\to\msys64\mingw64\bin\libwinpthread-1.dll
  C:\path\to\msys64\mingw64\bin\libgcc_s_seh-1.dll
  C:\path\to\msys64\mingw64\bin\libgmp-10.dll
  C:\path\to\msys64\mingw64\bin\libhogweed-6.dll
    C:\path\to\msys64\mingw64\bin\libnettle-8.dll
  C:\path\to\msys64\mingw64\bin\libidn2-0.dll
    C:\path\to\msys64\mingw64\bin\libiconv-2.dll
    C:\path\to\msys64\mingw64\bin\libintl-8.dll
    C:\path\to\msys64\mingw64\bin\libunistring-2.dll
  C:\path\to\msys64\mingw64\bin\libp11-kit-0.dll
    C:\path\to\msys64\mingw64\bin\libffi-7.dll
  C:\path\to\msys64\mingw64\bin\libtasn1-6.dll

2)  libharfbuzz-0.dll has these dependencies:
Found: C:\path\to\msys64\mingw64\bin\libharfbuzz-0.dll
C:\path\to\msys64\mingw64\bin\libharfbuzz-0.dll
  C:\path\to\msys64\mingw64\bin\libgcc_s_seh-1.dll
    C:\path\to\msys64\mingw64\bin\libwinpthread-1.dll
  C:\path\to\msys64\mingw64\bin\libstdc++-6.dll
  C:\path\to\msys64\mingw64\bin\libfreetype-6.dll
    C:\path\to\msys64\mingw64\bin\libbz2-1.dll
    C:\path\to\msys64\mingw64\bin\libbrotlidec.dll
      C:\path\to\msys64\mingw64\bin\libbrotlicommon.dll
    C:\path\to\msys64\mingw64\bin\libharfbuzz-0.dll
      C:\path\to\msys64\mingw64\bin\libglib-2.0-0.dll
        C:\path\to\msys64\mingw64\bin\libintl-8.dll
          C:\path\to\msys64\mingw64\bin\libiconv-2.dll
        C:\path\to\msys64\mingw64\bin\libpcre-1.dll
      C:\path\to\msys64\mingw64\bin\libgraphite2.dll
    C:\path\to\msys64\mingw64\bin\libpng16-16.dll
      C:\path\to\msys64\mingw64\bin\zlib1.dll

libgcc_s_seh-1.dll is an important one and IIRC, this dependency was the
one pushing Phillip into packaging the GCC source next to his binaries.

I checked the dependencies with cygcheck program shipped with Msys2 (and
snipped the standard Win libraries)

Best, Arash

Footnotes:
[1]  Files from https://git.sr.ht/~mplscorwin/emacs-w64

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

Thanks Arash.  This is very helpful.

On Sun, Jan 30, 2022 at 3:34 PM Arash Esbati <arash@gnu.org> wrote:
>
> I'm not tracking this development, so please bear with me.  Is the plan
> to unpack the file emacs-28.0.91-no-deps.zip into, say C:\emacs and
> unpack the .dll from emacs-28-deps.zip[1] into C:\emacs\bin?  If so, I'm

That should work.  Unpacking emacs-28.0.91.zip somewhere should be a
short-cut for that, exactly.

> afraid that the compiled features will not work since dependencies are
> missing in emacs-28-deps.zip, e.g.:
>
> 1)  libgnutls-30.dll has these dependencies:
> Found: C:\path\to\msys64\mingw64\bin\libgnutls-30.dll
> C:\path\to\msys64\mingw64\bin\libgnutls-30.dll
>   C:\path\to\msys64\mingw64\bin\libwinpthread-1.dll
>   C:\path\to\msys64\mingw64\bin\libgcc_s_seh-1.dll
>   C:\path\to\msys64\mingw64\bin\libgmp-10.dll
>   C:\path\to\msys64\mingw64\bin\libhogweed-6.dll
>     C:\path\to\msys64\mingw64\bin\libnettle-8.dll
>   C:\path\to\msys64\mingw64\bin\libidn2-0.dll
>     C:\path\to\msys64\mingw64\bin\libiconv-2.dll
>     C:\path\to\msys64\mingw64\bin\libintl-8.dll
>     C:\path\to\msys64\mingw64\bin\libunistring-2.dll
>   C:\path\to\msys64\mingw64\bin\libp11-kit-0.dll
>     C:\path\to\msys64\mingw64\bin\libffi-7.dll
>   C:\path\to\msys64\mingw64\bin\libtasn1-6.dll
>
> 2)  libharfbuzz-0.dll has these dependencies:
> Found: C:\path\to\msys64\mingw64\bin\libharfbuzz-0.dll
> C:\path\to\msys64\mingw64\bin\libharfbuzz-0.dll
>   C:\path\to\msys64\mingw64\bin\libgcc_s_seh-1.dll
>     C:\path\to\msys64\mingw64\bin\libwinpthread-1.dll
>   C:\path\to\msys64\mingw64\bin\libstdc++-6.dll
>   C:\path\to\msys64\mingw64\bin\libfreetype-6.dll
>     C:\path\to\msys64\mingw64\bin\libbz2-1.dll
>     C:\path\to\msys64\mingw64\bin\libbrotlidec.dll
>       C:\path\to\msys64\mingw64\bin\libbrotlicommon.dll
>     C:\path\to\msys64\mingw64\bin\libharfbuzz-0.dll
>       C:\path\to\msys64\mingw64\bin\libglib-2.0-0.dll
>         C:\path\to\msys64\mingw64\bin\libintl-8.dll
>           C:\path\to\msys64\mingw64\bin\libiconv-2.dll
>         C:\path\to\msys64\mingw64\bin\libpcre-1.dll
>       C:\path\to\msys64\mingw64\bin\libgraphite2.dll
>     C:\path\to\msys64\mingw64\bin\libpng16-16.dll
>       C:\path\to\msys64\mingw64\bin\zlib1.dll
>
> libgcc_s_seh-1.dll is an important one and IIRC, this dependency was the
> one pushing Phillip into packaging the GCC source next to his binaries.
>

IIUC, there is special-casing in the dependency packaging script that
prevents "recursing" exactly this chain:

$ grep -C1 gcc build-dep-zips.py
## Packages to fiddle with
## Source for gcc-libs is part of gcc
SKIP_SRC_PKGS=["mingw-w64-gcc-libs"]
SKIP_DEP_PKGS=["mingw-w64-glib2"]

I surmise people who want TLS (just as, now, people who want
native-comp) must install MSYS2 and place it's bin directory directory
on the windows path before starting Emacs.

If we want source zips to include gcc then we can probably do likewise
for gccjit.  To me, it sounds to if native-comp (and TLS) "just work",
i.e. so people don't need to hand-install MSYS2 to use the
"batteries-included"  versions of the binary distributions for
windows.

I'll do some experimentation.  I'd like to understand what this does
to the packaging times and file sizes. If this is something we are
already sure that we don't want to do, please let me know.

> I checked the dependencies with cygcheck program shipped with Msys2 (and
> snipped the standard Win libraries)
>
> Best, Arash
>
> Footnotes:
> [1]  Files from https://git.sr.ht/~mplscorwin/emacs-w64

Thanks for working on this; I really appreciate your diligence.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Arash Esbati]

Corwin Brust <corwin@bru.st> writes:

> I surmise people who want TLS (just as, now, people who want
> native-comp) must install MSYS2 and place it's bin directory directory
> on the windows path before starting Emacs.

I'm not sure about TLS, I had the impression that Emacs relies on the
.dll only.

My point was that currently the file emacs-28-deps.zip lacks
dependencies.  Take for example 3 files from the .zip libpng16-16.dll,
libturbojpeg.dll and libxml2-2.dll.  I've marked the missing
dependencies with '*' below:

Found: C:\path\to\msys64\mingw64\bin\libpng16-16.dll
C:\path\to\msys64\mingw64\bin\libpng16-16.dll
  C:\path\to\msys64\mingw64\bin\zlib1.dll *

Found: C:\path\to\msys64\mingw64\bin\libturbojpeg.dll
C:\path\to\msys64\mingw64\bin\libturbojpeg.dll
  C:\path\to\msys64\mingw64\bin\libgcc_s_seh-1.dll *
    C:\path\to\msys64\mingw64\bin\libwinpthread-1.dll *

Found: C:\path\to\msys64\mingw64\bin\libxml2-2.dll
C:\path\to\msys64\mingw64\bin\libxml2-2.dll
  C:\path\to\msys64\mingw64\bin\libiconv-2.dll *
  C:\path\to\msys64\mingw64\bin\liblzma-5.dll *
  C:\path\to\msys64\mingw64\bin\zlib1.dll *

Hence, people without an installed and available Msys2 (in $PATH) won't
be able to open a .png file in Emacs.

> Thanks for working on this; I really appreciate your diligence.

Thank your for working on this.

Best, Arash

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Hi Arash, hi Corwin

Arash Esbati <arash@gnu.org> writes:
> Corwin Brust <corwin@bru.st> writes:
>
>> I surmise people who want TLS (just as, now, people who want
>> native-comp) must install MSYS2 and place it's bin directory directory
>> on the windows path before starting Emacs.
> I'm not sure about TLS, I had the impression that Emacs relies on the
> .dll only.

I think you are right Corwin with respect to native compilation. Eli's
opinion is that we don't have to pack the dependencies for Windows.
Please see the thread:

  https://lists.gnu.org/archive/html/emacs-devel/2022-01/msg01020.html

For other "convenience dependencies" he advised me to look at
nt/INSTALL.W64.  Here's the list of packages for features which,
ideally, should be available in a GNU-Emacs distro out of the box on
Windows:

  mingw-w64-x86_64-xpm-nox \
  mingw-w64-x86_64-libtiff \
  mingw-w64-x86_64-giflib \
  mingw-w64-x86_64-libpng \
  mingw-w64-x86_64-libjpeg-turbo \
  mingw-w64-x86_64-librsvg \
  mingw-w64-x86_64-lcms2 \
  
  mingw-w64-x86_64-jansson \
  
  mingw-w64-x86_64-libxml2 \
  
  mingw-w64-x86_64-gnutls \
  
  mingw-w64-x86_64-zlib \
  
  mingw-w64-x86_64-harfbuzz

So I'm more in line with Arash: Image-, XML-, TLS-, json-,
(un-)archiving- and some text shaping support should be available.

> My point was that currently the file emacs-28-deps.zip lacks
> dependencies.

Yes, thank you Arash for pointing this out.  (Did a quick check, TLS is
not working, PNG viewing and EWW is not connecting to https sites.)

Currently we have 19 DLLs in the /bin folder (28.0.91), whereas 27.2 has
77 DLLs!  Corwin could you please check your build configuration, maybe
together with Phil?  (I'm sorry, I'm still of no help here.)

-- 
Thank you

     Dieter

Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

On Mon, Jan 31, 2022 at 9:58 AM H. Dieter Wilhelm
<dieter@duenenhof-wilhelm.de> wrote:
>
> Currently we have 19 DLLs in the /bin folder (28.0.91), whereas 27.2 has
> 77 DLLs!  Corwin could you please check your build configuration, maybe
> together with Phil?  (I'm sorry, I'm still of no help here.)

Excellent observation.

I hope Phil will chime-in with suggestions of what I may have
missed/broken.  I'll investigate also, obviously, and others'
kibitzing most welcome too.

Meanwhile, I suggest we should solve this problem before pushing
28.0.1 to the apha FTP site.  I don't want people to have different
expectations (needing to install MSYS) vs 27.1 and the pre-releases
Phillip created for that.

(Phil, is "Phil" fine? Preference for "Philip"?  I obsess on the oddest things.)

>
> --
> Thank you

^

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Corwin Brust <corwin@bru.st> writes:

> On Mon, Jan 31, 2022 at 9:58 AM H. Dieter Wilhelm
> <dieter@duenenhof-wilhelm.de> wrote:
>>
>> Currently we have 19 DLLs in the /bin folder (28.0.91), whereas 27.2 has
>> 77 DLLs!  Corwin could you please check your build configuration, maybe
>> together with Phil?  (I'm sorry, I'm still of no help here.)
>
> Excellent observation.
>
> I hope Phil will chime-in with suggestions of what I may have
> missed/broken.  I'll investigate also, obviously, and others'
> kibitzing most welcome too.

I think I've got, at least, a workaround!

I realised that ntldd.exe doesn't show any msys/mingw dependencies at
all, only Windows stuff! -> So I removed all Windows references from
PATH.  (Don't ask me why ntldd searches Windows paths first.)

Then, it seem's, that the msys/mingw people changed the tree layout of
their files!  When I'm changing "msys64" with "mingw64" in
build-dep-zips.py:

  def ntldd_munge(out):
      deps = out.splitlines()
      rtn = []
      for dep in deps:
          ## Output looks something like this

          ## KERNEL32.dll => C:\Windows\SYSTEM32\KERNEL32.dll (0x0000000002a30000)
          ## libwinpthread-1.dll => C:\msys64\mingw64\bin\libwinpthread-1.dll (0x0000000000090000)

          ## if it's the former, we want it, if its the later we don't
          splt = dep.split()
          ## if len(splt) > 2 and "msys64" in splt[2]:
          if len(splt) > 2 and "mingw64" in splt[2]:
              print("Adding dep", splt[0])
              rtn.append(splt[0].split(".")[0])

      return rtn

It copies much more DLLs to the deps folder.  :-)

(Oh dear, as if the whole building process wouldn't be complicated
enough..)

> Meanwhile, I suggest we should solve this problem before pushing
> 28.0.1 to the apha FTP site.  I don't want people to have different
> expectations (needing to install MSYS) vs 27.1 and the pre-releases
> Phillip created for that.
>
> (Phil, is "Phil" fine? Preference for "Philip"?  I obsess on the oddest things.)

I hope so, at least he signed his emails himself repeatedly with
"Phil"... ;-)

-- 
  Dieter

Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Óscar Fuentes]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> I realised that ntldd.exe doesn't show any msys/mingw dependencies at
> all, only Windows stuff! -> So I removed all Windows references from
> PATH.  (Don't ask me why ntldd searches Windows paths first.)

Are you saying that you have dlls with the same name under /mingw64 and
/Windows ?

> Then, it seem's, that the msys/mingw people changed the tree layout of
> their files!

No, we didn't.

> When I'm changing "msys64" with "mingw64" in
> build-dep-zips.py:
>
>   def ntldd_munge(out):
>       deps = out.splitlines()
>       rtn = []
>       for dep in deps:
>           ## Output looks something like this
>
>           ## KERNEL32.dll => C:\Windows\SYSTEM32\KERNEL32.dll (0x0000000002a30000)
>           ## libwinpthread-1.dll => C:\msys64\mingw64\bin\libwinpthread-1.dll (0x0000000000090000)
>
>           ## if it's the former, we want it, if its the later we don't

The comment above seems reversed.

>           splt = dep.split()
>           ## if len(splt) > 2 and "msys64" in splt[2]:
>           if len(splt) > 2 and "mingw64" in splt[2]:
>               print("Adding dep", splt[0])
>               rtn.append(splt[0].split(".")[0])
>
>       return rtn
>
> It copies much more DLLs to the deps folder.  :-)

Probably your MSYS2 install is not in a directory named "msys64" (the
"in" operator in Python is case-sensitive, isn't it?)

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Hello Óscar

Óscar Fuentes <ofv@wanadoo.es> writes:

> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
>
>> I realised that ntldd.exe doesn't show any msys/mingw dependencies at
>> all, only Windows stuff! -> So I removed all Windows references from
>> PATH.  (Don't ask me why ntldd searches Windows paths first.)
>
> Are you saying that you have dlls with the same name under /mingw64 and
> /Windows ?

I don't know, I haven't looked at it in this way, just realised that
there were many Window system DLLs . I'll send you a sample, if you want
to see it with your own eyes..

(I'm sorry I'm answering this with another computer.)

>> Then, it seem's, that the msys/mingw people changed the tree layout of
>> their files!
>
> No, we didn't.

Good! Then perhaps I, and probably Corwin as well, were installing
mingw/msys in another way as Phillip Lord did.  I will check
nt/README.W64 how the installation process is recommended.

>> When I'm changing "msys64" with "mingw64" in
>> build-dep-zips.py:
>>
>>   def ntldd_munge(out):
>>       deps = out.splitlines()
>>       rtn = []
>>       for dep in deps:
>>           ## Output looks something like this
>>
>>           ## KERNEL32.dll => C:\Windows\SYSTEM32\KERNEL32.dll (0x0000000002a30000)
>>           ## libwinpthread-1.dll => C:\msys64\mingw64\bin\libwinpthread-1.dll (0x0000000000090000)
>>
>>           ## if it's the former, we want it, if its the later we don't
>
> The comment above seems reversed.

Yes :-)

>>           splt = dep.split()
>>           ## if len(splt) > 2 and "msys64" in splt[2]:
>>           if len(splt) > 2 and "mingw64" in splt[2]:
>>               print("Adding dep", splt[0])
>>               rtn.append(splt[0].split(".")[0])
>>
>>       return rtn
>>
>> It copies much more DLLs to the deps folder.  :-)
>
> Probably your MSYS2 install is not in a directory named "msys64" (the

yes, it's just mingw64.

> "in" operator in Python is case-sensitive, isn't it?)

Ehem, I'm not so versed in Python but I'll check.

Thank you

      Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Arash Esbati]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> I realised that ntldd.exe doesn't show any msys/mingw dependencies at
> all, only Windows stuff! -> So I removed all Windows references from
> PATH.  (Don't ask me why ntldd searches Windows paths first.)
> [...]
> It copies much more DLLs to the deps folder.  :-)

I once wrote a small script to find out the dependencies myself.
Suppose these are the required DLLs for the features you want Emacs to
compile with:

libXpm-noX4.dll
libgif-7.dll
libgnutls-30.dll
libharfbuzz-0.dll
libjansson-4.dll
libjpeg-8.dll
liblcms2-2.dll
libpng16-16.dll
librsvg-2-2.dll
libsqlite3-0.dll
libtiff-5.dll
libwebp-7.dll
libxml2-2.dll

Then you should have these DLLs available:

libbrotlicommon.dll
libbrotlidec.dll
libbz2-1.dll
libcairo-2.dll
libcairo-gobject-2.dll
libdatrie-1.dll
libdeflate.dll
libexpat-1.dll
libffi-7.dll
libfontconfig-1.dll
libfreetype-6.dll
libfribidi-0.dll
libgcc_s_seh-1.dll
libgdk_pixbuf-2.0-0.dll
libgif-7.dll
libgio-2.0-0.dll
libglib-2.0-0.dll
libgmodule-2.0-0.dll
libgmp-10.dll
libgnutls-30.dll
libgobject-2.0-0.dll
libgraphite2.dll
libharfbuzz-0.dll
libhogweed-6.dll
libiconv-2.dll
libidn2-0.dll
libintl-8.dll
libjansson-4.dll
libjbig-0.dll
libjpeg-8.dll
liblcms2-2.dll
libLerc.dll
liblzma-5.dll
libnettle-8.dll
libp11-kit-0.dll
libpango-1.0-0.dll
libpangocairo-1.0-0.dll
libpangoft2-1.0-0.dll
libpangowin32-1.0-0.dll
libpcre-1.dll
libpixman-1-0.dll
libpng16-16.dll
librsvg-2-2.dll
libsqlite3-0.dll
libstdc++-6.dll
libtasn1-6.dll
libthai-0.dll
libtiff-5.dll
libunistring-2.dll
libwebp-7.dll
libwinpthread-1.dll
libxml2-2.dll
libXpm-noX4.dll
libzstd.dll
zlib1.dll

Maybe you want to double check this with your results.

HTH.  Best, Arash

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Arash Esbati <arash@gnu.org> writes:

> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
>
>> I realised that ntldd.exe doesn't show any msys/mingw dependencies at
>> all, only Windows stuff! -> So I removed all Windows references from
>> PATH.  (Don't ask me why ntldd searches Windows paths first.)
>> [...]
>> It copies much more DLLs to the deps folder.  :-)
>
> I once wrote a small script to find out the dependencies myself.
> Suppose these are the required DLLs for the features you want Emacs to
> compile with:
>
> libXpm-noX4.dll
> libgif-7.dll
> libgnutls-30.dll
> libharfbuzz-0.dll
> libjansson-4.dll
> libjpeg-8.dll
> liblcms2-2.dll
> libpng16-16.dll
> librsvg-2-2.dll
> libsqlite3-0.dll
> libtiff-5.dll
> libwebp-7.dll
> libxml2-2.dll
>
> Then you should have these DLLs available:
>
> libbrotlicommon.dll
> libbrotlidec.dll
> libbz2-1.dll
> libcairo-2.dll
> libcairo-gobject-2.dll
> libdatrie-1.dll
> libdeflate.dll
> libexpat-1.dll
> libffi-7.dll
> libfontconfig-1.dll
> libfreetype-6.dll
> libfribidi-0.dll
> libgcc_s_seh-1.dll
> libgdk_pixbuf-2.0-0.dll
> libgif-7.dll
> libgio-2.0-0.dll
> libglib-2.0-0.dll
> libgmodule-2.0-0.dll
> libgmp-10.dll
> libgnutls-30.dll
> libgobject-2.0-0.dll
> libgraphite2.dll
> libharfbuzz-0.dll
> libhogweed-6.dll
> libiconv-2.dll
> libidn2-0.dll
> libintl-8.dll
> libjansson-4.dll
> libjbig-0.dll
> libjpeg-8.dll
> liblcms2-2.dll
> libLerc.dll
> liblzma-5.dll
> libnettle-8.dll
> libp11-kit-0.dll
> libpango-1.0-0.dll
> libpangocairo-1.0-0.dll
> libpangoft2-1.0-0.dll
> libpangowin32-1.0-0.dll
> libpcre-1.dll
> libpixman-1-0.dll
> libpng16-16.dll
> librsvg-2-2.dll
> libsqlite3-0.dll
> libstdc++-6.dll
> libtasn1-6.dll
> libthai-0.dll
> libtiff-5.dll
> libunistring-2.dll
> libwebp-7.dll
> libwinpthread-1.dll
> libxml2-2.dll
> libXpm-noX4.dll
> libzstd.dll
> zlib1.dll
>
> Maybe you want to double check this with your results.

Thanks, I'll do

> HTH.  Best, Arash

Have you updated msys2 lately?

     Dieter

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Arash Esbati]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> Have you updated msys2 lately?

-> pacman -Sy
:: Synchronizing package databases...
 mingw32 is up to date
 mingw64 is up to date
 msys is up to date

Anything else I should do?

Best, Arash

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Arash Esbati <arash@gnu.org> writes:

> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
>
>> Have you updated msys2 lately?
>
> -> pacman -Sy
> :: Synchronizing package databases...
>  mingw32 is up to date
>  mingw64 is up to date
>  msys is up to date
>
> Anything else I should do?

Thank you for the information, good, my system was updated as well.


-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Hallo Arash :-)

Arash Esbati <arash@gnu.org> writes:

> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
>
>> I realised that ntldd.exe doesn't show any msys/mingw dependencies at
>> all, only Windows stuff! -> So I removed all Windows references from
>> PATH.  (Don't ask me why ntldd searches Windows paths first.)
>> [...]
>> It copies much more DLLs to the deps folder.  :-)
>
> I once wrote a small script to find out the dependencies myself.
> Suppose these are the required DLLs for the features you want Emacs to
> compile with:
>
> libXpm-noX4.dll
> libgif-7.dll
> libgnutls-30.dll
> libharfbuzz-0.dll
> libjansson-4.dll
> libjpeg-8.dll
> liblcms2-2.dll
> libpng16-16.dll
> librsvg-2-2.dll
> libsqlite3-0.dll
> libtiff-5.dll
> libwebp-7.dll
> libxml2-2.dll

I'm aiming for the DLL which are included in the list of packages in
nt/INSTALL.W64, sorry, I can't tell at the moment which ones are
included.

  mingw-w64-x86_64-xpm-nox \
  mingw-w64-x86_64-libtiff \
  mingw-w64-x86_64-giflib \
  mingw-w64-x86_64-libpng \
  mingw-w64-x86_64-libjpeg-turbo \
  mingw-w64-x86_64-librsvg \
  mingw-w64-x86_64-lcms2 \
  mingw-w64-x86_64-jansson \
  mingw-w64-x86_64-libxml2 \
  mingw-w64-x86_64-gnutls \
  mingw-w64-x86_64-zlib \
  mingw-w64-x86_64-harfbuzz

And I guess you are doing this for Emacs-29 and I'm going for 28.0.91,
so e.g.  sqlite3 is not in the portfolio yet. :-)

> Then you should have these DLLs available:

Arash                              Dieter

libbrotlicommon.dll                libbrotlicommon.dll	          		    
libbrotlidec.dll		   libbrotlidec.dll	    	  
libbz2-1.dll			   libbz2-1.dll		    	  
libcairo-2.dll			   libcairo-2.dll		  
libcairo-gobject-2.dll		   libcairo-gobject-2.dll	  
libdatrie-1.dll			   libdatrie-1.dll		    
libdeflate.dll			   libdeflate.dll		    
libexpat-1.dll			   libexpat-1.dll		    
libffi-7.dll			   libffi-7.dll		    	    
libfontconfig-1.dll		   libfontconfig-1.dll	    	    
libfreetype-6.dll		   libfreetype-6.dll	    	  
libfribidi-0.dll		   libfribidi-0.dll	    	  
libgcc_s_seh-1.dll		   libgcc_s_seh-1.dll	    	  
libgdk_pixbuf-2.0-0.dll		   libgdk_pixbuf-2.0-0.dll	  
libgif-7.dll			   libgif-7.dll		    	  
libgio-2.0-0.dll		   libgio-2.0-0.dll	    	    
libglib-2.0-0.dll		   libglib-2.0-0.dll	    	  
libgmodule-2.0-0.dll		   libgmodule-2.0-0.dll	    	  
libgmp-10.dll			   libgmp-10.dll		  
libgnutls-30.dll		   libgnutls-30.dll	    	  
libgobject-2.0-0.dll		   libgnutlsxx-28.dll	    	    
libgraphite2.dll		   libgobject-2.0-0.dll	    	  
libharfbuzz-0.dll		   libgraphite2.dll	    	  
libhogweed-6.dll		   libharfbuzz-0.dll	    	  
libiconv-2.dll			   libharfbuzz-gobject-0.dll   	  
libidn2-0.dll			   libharfbuzz-icu-0.dll	  
libintl-8.dll			   libharfbuzz-subset-0.dll    	  
libjansson-4.dll		   libhogweed-6.dll	    	    
libjbig-0.dll			   libiconv-2.dll		  
libjpeg-8.dll			   libicudt68.dll		  
liblcms2-2.dll			   libicuuc68.dll		    
libLerc.dll			   libidn2-0.dll		    
liblzma-5.dll			   libintl-8.dll		    
libnettle-8.dll			   libjansson-4.dll	    	    
libp11-kit-0.dll		   libjbig-0.dll		    
libpango-1.0-0.dll		   libjpeg-8.dll		  
libpangocairo-1.0-0.dll		   liblcms2-2.dll		    
libpangoft2-1.0-0.dll		   liblzma-5.dll		    
libpangowin32-1.0-0.dll		   libnettle-8.dll		    
libpcre-1.dll			   libp11-kit-0.dll	    	    
libpixman-1-0.dll		   libpango-1.0-0.dll	    	    
libpng16-16.dll			   libpangocairo-1.0-0.dll	  
librsvg-2-2.dll			   libpangoft2-1.0-0.dll	  
libsqlite3-0.dll		   libpangowin32-1.0-0.dll	    
libstdc++-6.dll			   libpcre-1.dll		    
libtasn1-6.dll			   libpixman-1-0.dll	    	    
libthai-0.dll			   libpng16-16.dll		    
libtiff-5.dll			   librsvg-2-2.dll		  
libunistring-2.dll		   libstdc++-6.dll		    
libwebp-7.dll			   libtasn1-6.dll		    
libwinpthread-1.dll		   libthai-0.dll		    
libxml2-2.dll			   libtiff-5.dll		    
libXpm-noX4.dll			   libtiffxx-5.dll		    
libzstd.dll			   libturbojpeg.dll	    	    
zlib1.dll			   libunistring-2.dll	    	    
				   libwebp-7.dll		  
				   libwinpthread-1.dll	    	  
				   libxml2-2.dll		    
				   libzstd.dll		    	  
				   zlib1.dll		    	    

The biggest difference is in the number of "harfbuzz" DLLs I think,
otherwise it look not too bad. :-)

Thanks
        Dieter

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 1850 bytes --]

Ahoy Dieter!

On Tue, Feb 1, 2022 at 6:45 AM H. Dieter Wilhelm
<dieter@duenenhof-wilhelm.de> wrote:
>
> I realised that ntldd.exe doesn't show any msys/mingw dependencies at
> all, only Windows stuff! -> So I removed all Windows references from
> PATH.  (Don't ask me why ntldd searches Windows paths first.)
>
> Then, it seem's, that the msys/mingw people changed the tree layout of
> their files!  When I'm changing "msys64" with "mingw64" in
> build-dep-zips.py:
[snip]
>           ## if len(splt) > 2 and "msys64" in splt[2]:
>           if len(splt) > 2 and "mingw64" in splt[2]:

With this, we are much closer.   New set is uploaded to the
with-native-compilation folder in the usual Not Sure If We Are Ready
For Prime Time location[1].

This version has 62 DLLs as were listed in the txt file attached to my
recent reply to to Phil in this same thread.  I still need to review
those against the 77 you reported from 27.2 but that's not going to be
tonight.

Meanwhile, I tested these (using the zip with deps version) on kiddos
gaming rig and verified that the tests from w32-feature pass,
package-refresh-package works, and various types of image load.  I
also checked with ProcessExplorer that I'm not loading DLLs except
those provided with the dist and those under some windows/system path.

As an interesting aside:  I see that it *is* loading the natively
compiled sources that were build and packaged with Emacs.  This is
despite getting a nil return from `native-comp-avialable-p', which
suggests that even people without a local libgccjit+gcc (e.g. a
working MSYS) can benefit from (the bundled) nativly compiled sources
we distribute.  [[stroking chin myscheviously]]

Please let me know if you think these are ready to push to alpha.

Attached are dffs for admin/nt/dist-build thus far.

[1] https://git.sr.ht/~mplscorwin/emacs-w64

[-- Attachment #2: emacs-28-nt-dist-build.patch --]
[-- Type: application/octet-stream, Size: 2865 bytes --]

diff --git a/admin/nt/dist-build/build-dep-zips.py b/admin/nt/dist-build/build-dep-zips.py
index c27045a001..9b8492ff0e 100755
--- a/admin/nt/dist-build/build-dep-zips.py
+++ b/admin/nt/dist-build/build-dep-zips.py
@@ -37,6 +37,7 @@
 mingw-w64-x86_64-librsvg
 mingw-w64-x86_64-libtiff
 mingw-w64-x86_64-libxml2
+mingw-w64-x86_64-gmp
 mingw-w64-x86_64-xpm-nox'''.split()
 
 DLL_REQ='''libgif
@@ -49,7 +50,9 @@
 librsvg
 libtiff
 libxml
-libXpm'''.split()
+libgmp
+libXpm
+libXpm-noX4'''.split()
 
 
 ## Options
@@ -101,7 +104,7 @@ def ntldd_munge(out):
 
         ## if it's the former, we want it, if its the later we don't
         splt = dep.split()
-        if len(splt) > 2 and "msys64" in splt[2]:
+        if len(splt) > 2 and "mingw64" in splt[2]:
             print("Adding dep", splt[0])
             rtn.append(splt[0].split(".")[0])
 
@@ -121,7 +124,7 @@ def ntldd_munge(out):
 
 ## Currently no packages seem to require this!
 ARCH_PKGS=[]
-SRC_REPO="https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources"
+SRC_REPO="https://repo.msys2.org/mingw/sources"
 
 
 def immediate_deps(pkg):
@@ -167,7 +170,7 @@ def download_source(tarball):
     if not os.path.exists("../emacs-src-cache/{}".format(tarball)):
         print("Downloading {}...".format(tarball))
         check_output_maybe(
-            "wget -a ../download.log -O ../emacs-src-cache/{} {}/{}/download"
+            "wget -a ../download.log -O ../emacs-src-cache/{} {}/{}"
             .format(tarball, SRC_REPO, tarball),
             shell=True
         )
diff --git a/admin/nt/dist-build/build-zips.sh b/admin/nt/dist-build/build-zips.sh
index 77d20a5a7b..fc3c40d691 100755
--- a/admin/nt/dist-build/build-zips.sh
+++ b/admin/nt/dist-build/build-zips.sh
@@ -47,13 +47,15 @@
     then
         echo [build] Configuring Emacs
         $REPO_DIR/$BRANCH/configure \
+	    --with-json \
             --without-dbus \
+            --with-native-compilation \
             --without-compress-install \
             $CACHE \
-            CFLAGS="$CFLAGS"
+            CFLAGS="$CFLAGS" 
     fi
 
-    make -j 4 $INSTALL_TARGET \
+    make -j 20 $INSTALL_TARGET \
          prefix=$HOME/emacs-build/install/emacs-$VERSION
     cd $HOME/emacs-build/install/emacs-$VERSION
     zip -r -9 emacs-$OF_VERSION-no-deps.zip *
@@ -96,7 +98,7 @@ BUILD=
 BUILD_64=1
 GIT_UP=0
 CONFIG=1
-CFLAGS="-O2 -static"
+CFLAGS="-O2"
 INSTALL_TARGET="install-strip"
 
 ## The location of the git repo
@@ -125,7 +127,7 @@ REPO_DIR=
         ;;
     s)
         SNAPSHOT="-snapshot"
-        CFLAGS="-O2 -static -g3"
+        CFLAGS="-O2 -g3"
         INSTALL_TARGET="install"
         ;;
     h)
@@ -191,7 +193,7 @@ OF_VERSION=
     OF_VERSION="$VERSION-`date +%Y-%m-%d`"
     ## Use snapshot dependencies
     SNAPSHOT=1
-    CFLAGS="-O2 -static -g3"
+    CFLAGS="-O2 -g3"
     INSTALL_TARGET="install"
 fi
 

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

[-- Attachment #1: Type: text/plain, Size: 3062 bytes --]

Ahoy Corwin

Corwin Brust <corwin@bru.st> writes:

> On Tue, Feb 1, 2022 at 6:45 AM H. Dieter Wilhelm
> <dieter@duenenhof-wilhelm.de> wrote:
>>
>> I realised that ntldd.exe doesn't show any msys/mingw dependencies at
>> all, only Windows stuff! -> So I removed all Windows references from
>> PATH.  (Don't ask me why ntldd searches Windows paths first.)
>>
>> Then, it seem's, that the msys/mingw people changed the tree layout of
>> their files!  When I'm changing "msys64" with "mingw64" in
>> build-dep-zips.py:

Óscar Fuentes said there were no changes in the layout lately, I assume
we just installed MSYS/MinGW differently as Phil did.  (I'm going to
learn what is recommended as the default path for MSYS/MinGW in
INSTALL.W64)

> [snip]
>>           ## if len(splt) > 2 and "msys64" in splt[2]:
>>           if len(splt) > 2 and "mingw64" in splt[2]:
>
> With this, we are much closer.   New set is uploaded to the
> with-native-compilation folder in the usual Not Sure If We Are Ready
> For Prime Time location[1].

Thanks, I had a look.  Before I forget - just in case - only stuff in
"with-native-compilation" is updated!  But I guess you'll upload only
"with-native-compilation" anyway. :-)

> This version has 62 DLLs as were listed in the txt file attached to my
> recent reply to to Phil in this same thread.  I still need to review
> those against the 77 you reported from 27.2 but that's not going to be
> tonight.

Please see in the attached file a comparison of Arash Esbati's (thanks
again), mine and your list of DLLs.  And it looks reasonable to me. 

I'm afraid we can't compare 27.2 builds from Phil's setup one on 1 with
our 28 builds.

> Meanwhile, I tested these (using the zip with deps version) on kiddos
> gaming rig and verified that the tests from w32-feature pass,
> package-refresh-package works, and various types of image load.  I
> also checked with ProcessExplorer that I'm not loading DLLs except
> those provided with the dist and those under some windows/system path.

Very good, thank you kiddos! :-)

> As an interesting aside:  I see that it *is* loading the natively
> compiled sources that were build and packaged with Emacs.  This is
> despite getting a nil return from `native-comp-avialable-p', which
> suggests that even people without a local libgccjit+gcc (e.g. a
> working MSYS) can benefit from (the bundled) nativly compiled sources
> we distribute.  [[stroking chin myscheviously]]

As far as I understood, this is exactly what Eli hoped for.

Such systems can't natively compile stuff but benefit from available
.eln files. :-)

> Please let me know if you think these are ready to push to alpha.

Please push.

Except others, with more experience, want to chime in..

> Attached are dffs for admin/nt/dist-build thus far.

They are looking good

(.. I don't comprehend the -static issue, yet.  But don't mind.)

> [1] https://git.sr.ht/~mplscorwin/emacs-w64

(... and then please go to the folder "with-native-compilation")


[-- Attachment #2: List of DLLs --]
[-- Type: text/plain, Size: 4181 bytes --]

Arash                              Dieter                          Corwin
                                                                   				   	
libbrotlicommon.dll                libbrotlicommon.dll	           libLerc.dll                   			    
libbrotlidec.dll		   libbrotlidec.dll	    	   libXpm-noX4.dll				
libbz2-1.dll			   libbz2-1.dll		    	   libbrotlicommon.dll			
libcairo-2.dll			   libcairo-2.dll		   libbrotlidec.dll			
libcairo-gobject-2.dll		   libcairo-gobject-2.dll	   libbz2-1.dll				
libdatrie-1.dll			   libdatrie-1.dll		   libcairo-2.dll				 
libdeflate.dll			   libdeflate.dll		   libcairo-gobject-2.dll			 
libexpat-1.dll			   libexpat-1.dll		   libdatrie-1.dll				 
libffi-7.dll			   libffi-7.dll		    	   libdeflate.dll				 
libfontconfig-1.dll		   libfontconfig-1.dll	    	   libexpat-1.dll				 
libfreetype-6.dll		   libfreetype-6.dll	    	   libffi-7.dll				
libfribidi-0.dll		   libfribidi-0.dll	    	   libfontconfig-1.dll			
libgcc_s_seh-1.dll		   libgcc_s_seh-1.dll	    	   libfreetype-6.dll			
libgdk_pixbuf-2.0-0.dll		   libgdk_pixbuf-2.0-0.dll	   libfribidi-0.dll			
libgif-7.dll			   libgif-7.dll		    	   libgcc_s_seh-1.dll			
libgio-2.0-0.dll		   libgio-2.0-0.dll	    	   libgdk_pixbuf-2.0-0.dll			 
libglib-2.0-0.dll		   libglib-2.0-0.dll	    	   libgif-7.dll				
libgmodule-2.0-0.dll		   libgmodule-2.0-0.dll	    	   libgio-2.0-0.dll			
libgmp-10.dll			   libgmp-10.dll		   libglib-2.0-0.dll			
libgnutls-30.dll		   libgnutls-30.dll	    	   libgmodule-2.0-0.dll			
libgobject-2.0-0.dll		   libgnutlsxx-28.dll	    	   libgmp-10.dll				 
libgraphite2.dll		   libgobject-2.0-0.dll	    	   libgmpxx-4.dll				
libharfbuzz-0.dll		   libgraphite2.dll	    	   libgnutls-30.dll			
libhogweed-6.dll		   libharfbuzz-0.dll	    	   libgnutls-openssl-27.dll		
libiconv-2.dll			   libharfbuzz-gobject-0.dll   	   libgnutlsxx-28.dll			
libidn2-0.dll			   libharfbuzz-icu-0.dll	   libgobject-2.0-0.dll			
libintl-8.dll			   libharfbuzz-subset-0.dll    	   libgraphite2.dll			
libjansson-4.dll		   libhogweed-6.dll	    	   libharfbuzz-0.dll			 
libjbig-0.dll			   libiconv-2.dll		   libharfbuzz-gobject-0.dll		
libjpeg-8.dll			   libicudt68.dll		   libharfbuzz-icu-0.dll			
liblcms2-2.dll			   libicuuc68.dll		   libharfbuzz-subset-0.dll		 
libLerc.dll			   libidn2-0.dll		   libhogweed-6.dll			 
liblzma-5.dll			   libintl-8.dll		   libiconv-2.dll				 
libnettle-8.dll			   libjansson-4.dll	    	   libidn2-0.dll				 
libp11-kit-0.dll		   libjbig-0.dll		   libintl-8.dll				 
libpango-1.0-0.dll		   libjpeg-8.dll		   libjansson-4.dll			
libpangocairo-1.0-0.dll		   liblcms2-2.dll		   libjbig-0.dll				 
libpangoft2-1.0-0.dll		   liblzma-5.dll		   libjpeg-8.dll				 
libpangowin32-1.0-0.dll		   libnettle-8.dll		   liblcms2-2.dll				 
libpcre-1.dll			   libp11-kit-0.dll	    	   liblzma-5.dll				 
libpixman-1-0.dll		   libpango-1.0-0.dll	    	   libnettle-8.dll				 
libpng16-16.dll			   libpangocairo-1.0-0.dll	   libp11-kit-0.dll			
librsvg-2-2.dll			   libpangoft2-1.0-0.dll	   libpango-1.0-0.dll			
libsqlite3-0.dll		   libpangowin32-1.0-0.dll	   libpangocairo-1.0-0.dll			 
libstdc++-6.dll			   libpcre-1.dll		   libpangoft2-1.0-0.dll			 
libtasn1-6.dll			   libpixman-1-0.dll	    	   libpangowin32-1.0-0.dll			 
libthai-0.dll			   libpng16-16.dll		   libpcre-1.dll				 
libtiff-5.dll			   librsvg-2-2.dll		   libpixman-1-0.dll			
libunistring-2.dll		   libstdc++-6.dll		   libpng16-16.dll				 
libwebp-7.dll			   libtasn1-6.dll		   librsvg-2-2.dll				 
libwinpthread-1.dll		   libthai-0.dll		   libstdc++-6.dll				 
libxml2-2.dll			   libtiff-5.dll		   libtasn1-6.dll				 
libXpm-noX4.dll			   libtiffxx-5.dll		   libthai-0.dll				 
libzstd.dll			   libturbojpeg.dll	    	   libtiff-5.dll				 
zlib1.dll			   libunistring-2.dll	    	   libtiffxx-5.dll				 
				   libwebp-7.dll		   libturbojpeg.dll			
				   libwinpthread-1.dll	    	   libunistring-2.dll			
				   libxml2-2.dll		   libwebp-7.dll				 
				   libzstd.dll		    	   libwinpthread-1.dll			
				   zlib1.dll		    	   libxml2-2.dll				 
				   				   libzstd.dll				
								   zlib1.dll                               































































[-- Attachment #3: Type: text/plain, Size: 21 bytes --]



--

        Dieter

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Wed, 2 Feb 2022 21:22:37 -0600
> Cc: Arash Esbati <arash@gnu.org>, Emacs developers <emacs-devel@gnu.org>, Eli Zaretskii <eliz@gnu.org>, 
> 	Phillip Lord <phillip.lord@russet.org.uk>
> 
> As an interesting aside:  I see that it *is* loading the natively
> compiled sources that were build and packaged with Emacs.  This is
> despite getting a nil return from `native-comp-avialable-p', which
> suggests that even people without a local libgccjit+gcc (e.g. a
> working MSYS) can benefit from (the bundled) nativly compiled sources
> we distribute.  [[stroking chin myscheviously]]

This is the expected behavior.  Only native-compilation of *.el files
for which no *.eln files are provided is disabled without libgccjit
and Binutils installation.  Loading of existing *.eln files doesn't
need libgccjit.

So I'm not sure why you are surprised.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

On Thu, Feb 3, 2022 at 1:47 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Corwin Brust <corwin@bru.st>
> > Date: Wed, 2 Feb 2022 21:22:37 -0600
> > Cc: Arash Esbati <arash@gnu.org>, Emacs developers <emacs-devel@gnu.org>, Eli Zaretskii <eliz@gnu.org>,
> >       Phillip Lord <phillip.lord@russet.org.uk>
> >
> > As an interesting aside:  I see that it *is* loading the natively
> > compiled sources that were build and packaged with Emacs.  This is
> > despite getting a nil return from `native-comp-avialable-p', which
> > suggests that even people without a local libgccjit+gcc (e.g. a
> > working MSYS) can benefit from (the bundled) nativly compiled sources
> > we distribute.  [[stroking chin myscheviously]]
>
> This is the expected behavior.  Only native-compilation of *.el files
> for which no *.eln files are provided is disabled without libgccjit
> and Binutils installation.  Loading of existing *.eln files doesn't
> need libgccjit.
>
> So I'm not sure why you are surprised.

Now that ask, I'm not sure either.

Should we consider adding NATIVE_FULL_AOT when building the windows
distribution?

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Thu, 3 Feb 2022 07:57:32 -0600
> Cc: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>, Arash Esbati <arash@gnu.org>, 
> 	Emacs developers <emacs-devel@gnu.org>, Phillip Lord <phillip.lord@russet.org.uk>
> 
> > > As an interesting aside:  I see that it *is* loading the natively
> > > compiled sources that were build and packaged with Emacs.  This is
> > > despite getting a nil return from `native-comp-avialable-p', which
> > > suggests that even people without a local libgccjit+gcc (e.g. a
> > > working MSYS) can benefit from (the bundled) nativly compiled sources
> > > we distribute.  [[stroking chin myscheviously]]
> >
> > This is the expected behavior.  Only native-compilation of *.el files
> > for which no *.eln files are provided is disabled without libgccjit
> > and Binutils installation.  Loading of existing *.eln files doesn't
> > need libgccjit.
> >
> > So I'm not sure why you are surprised.
> 
> Now that ask, I'm not sure either.
> 
> Should we consider adding NATIVE_FULL_AOT when building the windows
> distribution?

You could, but that still means users cannot modify the *.el files and
get them recompiled.

Whether NATIVE_FULL_AOT is worth is: it is yet to be determined.  I
tend to think we should provide a default build, and only use
NATIVE_FULL_AOT if other platforms do that (which will not happen
before Emacs 28 is released and we collect feedback from users and
downstream distributions).

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

On Thu, Feb 3, 2022 at 10:53 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Corwin Brust <corwin@bru.st>
> >
> > Should we consider adding NATIVE_FULL_AOT when building the windows
> > distribution?
>
> You could, but that still means users cannot modify the *.el files and
> get them recompiled.
>
> Whether NATIVE_FULL_AOT is worth is: it is yet to be determined.  I
> tend to think we should provide a default build, and only use
> NATIVE_FULL_AOT if other platforms do that (which will not happen
> before Emacs 28 is released and we collect feedback from users and
> downstream distributions).

What you say makes perfect sense.

FWIW, I have been using NATIVE_FULL_AOT with self-built emacs-28 as my
daily drive for around four months.  An instance of this is nearly
always running ERC.  I've been quite happy.

Let's go without NATIVE_FULL_AOT, for at least pre-release 1.

I will push what I have to the FTP site after work today.,
notwithstanding reports of (further) critical defects.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Corwin Brust <corwin@bru.st> writes:

> On Mon, Jan 31, 2022 at 9:58 AM H. Dieter Wilhelm
> <dieter@duenenhof-wilhelm.de> wrote:
>>
>> Currently we have 19 DLLs in the /bin folder (28.0.91), whereas 27.2 has
>> 77 DLLs!  Corwin could you please check your build configuration, maybe
>> together with Phil?  (I'm sorry, I'm still of no help here.)
>
> Excellent observation.
>
> I hope Phil will chime-in with suggestions of what I may have
> missed/broken.  I'll investigate also, obviously, and others'
> kibitzing most welcome too.


Strange. What are the differences?


>
> Meanwhile, I suggest we should solve this problem before pushing
> 28.0.1 to the apha FTP site.  I don't want people to have different
> expectations (needing to install MSYS) vs 27.1 and the pre-releases
> Phillip created for that.
>
> (Phil, is "Phil" fine? Preference for "Philip"?  I obsess on the oddest things.)


"Phil" is my usual. "Phillip" is my actually full name. "Philip" is an
alternative spelling but is not my name.

There's a story behind this, but this is probably not the place.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 3567 bytes --]

On Wed, Feb 2, 2022 at 4:07 PM Phillip Lord <phillip.lord@russet.org.uk> wrote:
>
> Corwin Brust <corwin@bru.st> writes:
>
> > On Mon, Jan 31, 2022 at 9:58 AM H. Dieter Wilhelm
> > <dieter@duenenhof-wilhelm.de> wrote:
> >>
> >> Currently we have 19 DLLs in the /bin folder (28.0.91), whereas 27.2 has
> >> 77 DLLs!  Corwin could you please check your build configuration, maybe
> >> together with Phil?  (I'm sorry, I'm still of no help here.)
> >
> > Excellent observation.
> >
> > I hope Phil will chime-in with suggestions of what I may have
> > missed/broken.  I'll investigate also, obviously, and others'
> > kibitzing most welcome too.
>
>
> Strange. What are the differences?

Firstly:  I'm not too sure yet.  I still need to download the Emacs
27.2 zip from the official source and do a direct "gap" comparison.

Overall, I'm leaning toward ditching automation of dependency
detection in favor of adding a separate file.   All efforts to
automating dependency recursion seem britle, ultimating causing more
work than they save us as we have to track down one mystery after
another. The proposed new file will
 - exhaustively lists each DLL we ship
 - be referenced in by documentation (such as README.W32)
 - be loaded by scripts, e.g. build-dep-zips.py
 - have additional "columns" to name the msys package, and
 - document why the dependency is required.

I have attached a txt file which is the complete list of what I would
start out including in that file.  Obviously, it's missing the msys
package names and has no commentary, so far.   I would probably call
it admin/nt/dist-build/deps.txt but I don't feel strongly about the
filename.

WDYT?

Meanwhile, in mesier news:

I wrote a perl script (also attached but don't bother) to find second
(and etc) order dependances using LDD that I believe should have been
(but have not been) getting pulled into the build.  I thought that the
msys installation folder path change Dieter caught would sort this,
but the numbers seem to don't add up (yet), probably because of the
undependability of the msys dependency checking that you mentioned.

I thought this would get me up to the same count (77) as Dieter reported, but:

  $ ls *.dll | wc -l
  19

  $ perl ../../../find-deps.pl | xargs -I '{}' cp '{}' .
  searched: 19, found: 392, of which 43 are missing in
/c/Users/corwi/emacs-build/install/emacs-28.0.91/bin

  $ ls *.dll | wc -l
  62

Note, these are the 62 DLLs listed in the attached textt file.

I also used this rather henus string of pipelined commands to
"cross-check" for missing dependencies and only came up with one,
which appears to be an optional dependency of harfbuzz:

  $  find ./ -name '*.dll' -print0 | xargs -0 -I '{}' cygcheck '{}'
2>&1 | grep 'could not' | cut -d ' ' -f 6 | sort | uniq
  libicuuc69.dll

Finally, I double-checked using an even more rediculous set of
pipeline commands. seemingly confirming everything the prior finds are
in the bin directory:

  $ find ./ -name '*.dll' -print0 | xargs -0 -I '{}' cygcheck '{}'
2>&1 | grep -v 'could not' | cut -d ' ' -f 6 | sort | uniq | xargs -0
-L1 echo | xargs -0 cygpath | perl -npe 's|c:|/c|g' | xargs -L1
basename | xargs ls >/dev/null

All this has me convinced some DLLs that shipped with Emacs 27.2
aren't findable as dependencies of anything we currently need for
Emacs 28, but not why.  And that leads me to the suggestion, above:  I
think we should hard-code the DLLs along with their source packages.
We can consider automatically generating/updating the deps file once
we get that far.

Warmest regards,
Corwin

[-- Attachment #2: dll-list.txt --]
[-- Type: text/plain, Size: 1062 bytes --]

libLerc.dll
libXpm-noX4.dll
libbrotlicommon.dll
libbrotlidec.dll
libbz2-1.dll
libcairo-2.dll
libcairo-gobject-2.dll
libdatrie-1.dll
libdeflate.dll
libexpat-1.dll
libffi-7.dll
libfontconfig-1.dll
libfreetype-6.dll
libfribidi-0.dll
libgcc_s_seh-1.dll
libgdk_pixbuf-2.0-0.dll
libgif-7.dll
libgio-2.0-0.dll
libglib-2.0-0.dll
libgmodule-2.0-0.dll
libgmp-10.dll
libgmpxx-4.dll
libgnutls-30.dll
libgnutls-openssl-27.dll
libgnutlsxx-28.dll
libgobject-2.0-0.dll
libgraphite2.dll
libharfbuzz-0.dll
libharfbuzz-gobject-0.dll
libharfbuzz-icu-0.dll
libharfbuzz-subset-0.dll
libhogweed-6.dll
libiconv-2.dll
libidn2-0.dll
libintl-8.dll
libjansson-4.dll
libjbig-0.dll
libjpeg-8.dll
liblcms2-2.dll
liblzma-5.dll
libnettle-8.dll
libp11-kit-0.dll
libpango-1.0-0.dll
libpangocairo-1.0-0.dll
libpangoft2-1.0-0.dll
libpangowin32-1.0-0.dll
libpcre-1.dll
libpixman-1-0.dll
libpng16-16.dll
librsvg-2-2.dll
libstdc++-6.dll
libtasn1-6.dll
libthai-0.dll
libtiff-5.dll
libtiffxx-5.dll
libturbojpeg.dll
libunistring-2.dll
libwebp-7.dll
libwinpthread-1.dll
libxml2-2.dll
libzstd.dll
zlib1.dll

[-- Attachment #3: find-deps.pl --]
[-- Type: application/octet-stream, Size: 2749 bytes --]

#!/usr/bin/env perl
#
# find-deps - find DLLs required by (but not in) the DLLs in CWD
#
# Copyright (C) 2022 Corwin Brust
#
# Quick and dirty check for missing dependancies of the DLLs included
# with a recently packaged build of Emacs. This should work given
# building Emacs works.  Run it from the install folder used for
# packaging, e.g.:
#
#   cd ~/emacs-build/install/emacs-28.0.91/bin
#   perl ../../../find-deps.pl > ~/emacs-build/dep-search.txt
# Or:
#  perl ../../../find-deps.pl | xargs -I '{}' cp '{}' .
#
# this rather obtuse alternative catches a couple more:
#   find ./ -name '*.dll' -print0 | \
#    xargs -0 -I '{}' cygcheck '{}' 2>&1 | \
#    grep 'could not' | cut -d ' ' -f 6 | \
#    sort | uniq | xargs -I '{}' cp '{}' .

## This file is not part of GNU Emacs.

## This program is free software: you can redistribute it and/or modify
## it under the terms of the GNU General Public License as published by
## the Free Software Foundation, either version 3 of the License, or
## (at your option) any later version.

## This program is distributed in the hope that it will be useful,
## but WITHOUT ANY WARRANTY; without even the implied warranty of
## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
## GNU General Public License for more details.

## You should have received a copy of the GNU General Public License
## along with this program.  If not, see <https://www.gnu.org/licenses/>.

use strict;
use warnings qw[all];

use File::Basename;

my %deps;

# use ldd to find deps
sub deps_of(_) {
  local $_ = `ldd $_`;
  s/^\s+//msg;
  map [ split / => / ]->[1] || $_,
    map { s/ \(.*?\)//; $_ }
    grep !/not found/,  # ignore optional
    grep !m|/Windows/|i,# ignore */Windows/*
    split /\n/
}

# call deps_of for each arg (unless we already have)
sub walk_deps(@) {
  my $found_count = 0;
  for (@_) {
    my $key = fileparse($_);
    next if exists $deps{$key} or !/.dll$/;
    my @found = deps_of $_;
    if(@found) {
      $deps{$key} = \@found;
      $found_count += 1;
    }
  }
  return $found_count;
}

# get a list of DLLs to start with
my @libs = glob('*.dll');

# search for deps of DLLs we included
walk_deps @libs;

# search for deps of our deps
my $added_count = 0;
do {
  $added_count = walk_deps map @$_, values %deps
} while($added_count > 0);

# de-dup
my %uniq;
for(map @$_, values %deps) {
  my $dll = fileparse($_);
  next if -e $dll; # skip those we included
  $uniq{$_} += 1;
}

# report
$\ = "\n";
print for sort keys %uniq;
warn
  "searched: ",  scalar(@libs),
  ", found: ",   scalar( map @$_, values %deps),
  ", of which ", scalar(keys %uniq),
  " are missing in ", `pwd`;

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Corwin Brust <corwin@bru.st> writes:

> On Wed, Feb 2, 2022 at 4:07 PM Phillip Lord <phillip.lord@russet.org.uk> wrote:
>>
>> Corwin Brust <corwin@bru.st> writes:
>>
>> > On Mon, Jan 31, 2022 at 9:58 AM H. Dieter Wilhelm
>> > <dieter@duenenhof-wilhelm.de> wrote:
>> >>
>> >> Currently we have 19 DLLs in the /bin folder (28.0.91), whereas 27.2 has
>> >> 77 DLLs!  Corwin could you please check your build configuration, maybe
>> >> together with Phil?  (I'm sorry, I'm still of no help here.)
>> >
>> > Excellent observation.
>> >
>> > I hope Phil will chime-in with suggestions of what I may have
>> > missed/broken.  I'll investigate also, obviously, and others'
>> > kibitzing most welcome too.
>>
>>
>> Strange. What are the differences?
>
> Firstly:  I'm not too sure yet.  I still need to download the Emacs
> 27.2 zip from the official source and do a direct "gap" comparison.
>
> Overall, I'm leaning toward ditching automation of dependency
> detection in favor of adding a separate file.   All efforts to
> automating dependency recursion seem britle, ultimating causing more
> work than they save us as we have to track down one mystery after
> another. The proposed new file will
>  - exhaustively lists each DLL we ship
>  - be referenced in by documentation (such as README.W32)
>  - be loaded by scripts, e.g. build-dep-zips.py
>  - have additional "columns" to name the msys package, and
>  - document why the dependency is required.
>
> I have attached a txt file which is the complete list of what I would
> start out including in that file.  Obviously, it's missing the msys
> package names and has no commentary, so far.   I would probably call
> it admin/nt/dist-build/deps.txt but I don't feel strongly about the
> filename.
>
> WDYT?


I think it sounds sensible, but there is a big unknown. The list of DLLs
that Emacs depends on is not very big and it does not change very much
(I think three or four new ones in the time that I did the build). So,
the question is, how much change will there be in the list of indirect
dependencies; that is partly in the hands of the developers of the
libraries we depend on and partly in the hands of MSYS2. And it also
depends on whether you want to update or check for just minor releases
or just major ones.

My guess would be that this will be managable and that you are unlikely
to find a lot of noise in the process, but it is just a guess.

Personally, I'd just bundle the list as a string in a python script. Why
parse something that you don't have to?

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

On Wed, Feb 2, 2022 at 4:07 PM Phillip Lord <phillip.lord@russet.org.uk> wrote:
>
> Corwin Brust <corwin@bru.st> writes:
>
> > (Phil, is "Phil" fine? Preference for "Philip"?  I obsess on the oddest things.)
>
>
> "Phil" is my usual. "Phillip" is my actually full name. "Philip" is an
> alternative spelling but is not my name.

Thanks, I really do appreciate your indulging my obsessiveness.

> There's a story behind this, but this is probably not the place.

Oh boy.  Count me as interested (and blushing).

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Sun, 30 Jan 2022 16:16:52 -0600
> Cc: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>, Eli Zaretskii <eliz@gnu.org>, 
> 	Emacs developers <emacs-devel@gnu.org>, Phillip Lord <phillip.lord@russet.org.uk>
> 
> I surmise people who want TLS (just as, now, people who want
> native-comp) must install MSYS2 and place it's bin directory directory
> on the windows path before starting Emacs.

The GnuTLS library is very important to have, as without it many
features, like package.el and EWW, will be unable to work, or will be
very limited.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Corwin Brust <corwin@bru.st> writes:

> IIUC, there is special-casing in the dependency packaging script that
> prevents "recursing" exactly this chain:
>
> $ grep -C1 gcc build-dep-zips.py
> ## Packages to fiddle with
> ## Source for gcc-libs is part of gcc
> SKIP_SRC_PKGS=["mingw-w64-gcc-libs"]
> SKIP_DEP_PKGS=["mingw-w64-glib2"]

If I remember correctly, this is because the msys dependency tree is
not totally reliable.

At the time, I was trying hard to decrease the binary
size. Unfortuately, the msys tree is an "at most" dependency rather than
an "at least"; so, it's very easy to find that, again, you've pulled in
half of msys2 which is not the intention.

Hence this skipping.


> I surmise people who want TLS (just as, now, people who want
> native-comp) must install MSYS2 and place it's bin directory directory
> on the windows path before starting Emacs.


TLS should work if at all possible. The tests that I have for a
functional windows Emacs (in etc/w32-feature.el) are better than nothing
but don't test whether emacs is working just whether it thinks that it
should be.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

On Wed, Feb 2, 2022 at 4:04 PM Phillip Lord <phillip.lord@russet.org.uk> wrote:
>
> If I remember correctly, this is because the msys dependency tree is
> not totally reliable.

I noticed :P

Locally I've been investigating by comparing results from several
different sources including ProcessExplorer, ldd, ntldd and cygcheck.
(To get recursive checking with ldd I wrote a rather ugly Perl
script.)  Then I'm looking at the documentation from MSYS website
(one-by-one by hand) to figure out which ones are optional.

>
> TLS should work if at all possible. The tests that I have for a
> functional windows Emacs (in etc/w32-feature.el) are better than nothing
> but don't test whether emacs is working just whether it thinks that it
> should be.
>

ACK, and thanks for the nudge on using etc/w32-feature.el to test the
installed versions.

FTR, as I hack on the scripts I've been testing locally running both
scripts then by opening Emacs from the
~/emacs-build/install/emacs-28.0.91/bin folder and then "doing suff".
For example opening a PNG.  Then I check (with ProcessExplorer) that
Emacs is loading (e.g.) libpng16-16.dll from the Emacs bin folder (and
not from the msys bin or some other place).

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Corwin Brust <corwin@bru.st> writes:

> On Wed, Feb 2, 2022 at 4:04 PM Phillip Lord <phillip.lord@russet.org.uk> wrote:
>>
>> If I remember correctly, this is because the msys dependency tree is
>> not totally reliable.
>
> I noticed :P
>
> Locally I've been investigating by comparing results from several
> different sources including ProcessExplorer, ldd, ntldd and cygcheck.
> (To get recursive checking with ldd I wrote a rather ugly Perl
> script.)  Then I'm looking at the documentation from MSYS website
> (one-by-one by hand) to figure out which ones are optional.

I was clearly keen to get something reasonable automatic as the list may
change over time. Although, as I always used the same dependencies for
an entire main release of Emacs, having something manual would be possible.


>> TLS should work if at all possible. The tests that I have for a
>> functional windows Emacs (in etc/w32-feature.el) are better than nothing
>> but don't test whether emacs is working just whether it thinks that it
>> should be.
>>
>
> ACK, and thanks for the nudge on using etc/w32-feature.el to test the
> installed versions.

I used to set up the .emacs on my build machine to automatically run
w32-feature.el on start up which was a great time saver!

> FTR, as I hack on the scripts I've been testing locally running both
> scripts then by opening Emacs from the
> ~/emacs-build/install/emacs-28.0.91/bin folder and then "doing suff".
> For example opening a PNG.  Then I check (with ProcessExplorer) that
> Emacs is loading (e.g.) libpng16-16.dll from the Emacs bin folder (and
> not from the msys bin or some other place).

Indeed; it would be good to get a bit more of this kind of thing into
w32-feature.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: Arash Esbati <arash@gnu.org>,  "H. Dieter Wilhelm"
>  <dieter@duenenhof-wilhelm.de>,  Eli Zaretskii <eliz@gnu.org>,  Emacs
>  developers <emacs-devel@gnu.org>
> Date: Wed, 02 Feb 2022 22:04:42 +0000
> 
> If I remember correctly, this is because the msys dependency tree is
> not totally reliable.
> 
> At the time, I was trying hard to decrease the binary
> size. Unfortuately, the msys tree is an "at most" dependency rather than
> an "at least"; so, it's very easy to find that, again, you've pulled in
> half of msys2 which is not the intention.
> 
> Hence this skipping.

I can suggest a way to find all the actual dependencies of a set of
*.exe programs and DLLs by using only the standard tools, and
is thus very reliable.  The command to use is this:

  objdump -p *.exe *.dll | fgrep "DLL Name:" | gawk " {print $3, $4, $5}" | sort -u

Run this in the 'bin/' directory where you have executable programs
and DLLs, and it will show a sorted list of all the DLLs needed by
those programs and DLLs.  You will need to recognize and ignore any
system DLLs, such as msvcrt.dll, KERNEL32.dll, USER32.dll, WS2_32.dll
etc. in the list; for the rest, make sure every one of them is in the
same 'bin/' directory.  Each time you add some DLL to the 'bin/'
directory, re-run the above command again, because adding DLLs can
potentially add dependencies.  Repeat this process until all the DLLs
shown by the command are in the same directory: then you are done.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Cc: Arash Esbati <arash@gnu.org>,  "H. Dieter Wilhelm"
>>  <dieter@duenenhof-wilhelm.de>,  Eli Zaretskii <eliz@gnu.org>,  Emacs
>>  developers <emacs-devel@gnu.org>
>> Date: Wed, 02 Feb 2022 22:04:42 +0000
>> 
>> If I remember correctly, this is because the msys dependency tree is
>> not totally reliable.
>> 
>> At the time, I was trying hard to decrease the binary
>> size. Unfortuately, the msys tree is an "at most" dependency rather than
>> an "at least"; so, it's very easy to find that, again, you've pulled in
>> half of msys2 which is not the intention.
>> 
>> Hence this skipping.
>
> I can suggest a way to find all the actual dependencies of a set of
> *.exe programs and DLLs by using only the standard tools, and
> is thus very reliable.  The command to use is this:
>
>   objdump -p *.exe *.dll | fgrep "DLL Name:" | gawk " {print $3, $4, $5}" | sort -u
>
> Run this in the 'bin/' directory where you have executable programs
> and DLLs, and it will show a sorted list of all the DLLs needed by
> those programs and DLLs.  You will need to recognize and ignore any
> system DLLs, such as msvcrt.dll, KERNEL32.dll, USER32.dll, WS2_32.dll
> etc. in the list; for the rest, make sure every one of them is in the
> same 'bin/' directory.  Each time you add some DLL to the 'bin/'
> directory, re-run the above command again, because adding DLLs can
> potentially add dependencies.  Repeat this process until all the DLLs
> shown by the command are in the same directory: then you are done.

When trying above iterative approach, I just find libgmp and after
adding libgmp-10.dll to the folder there are no further dependencies to
see!?

  uidg1626@VSL7V7PW MINGW64 ~/emacs-build/build/emacs-28.0.91/bin
  $ objdump.exe -p *.exe *.dll |grep "DLL Name"|sort -u
          DLL Name: ADVAPI32.dll
          DLL Name: COMCTL32.dll
          DLL Name: GDI32.dll
          DLL Name: KERNEL32.dll
          DLL Name: MPR.dll
          DLL Name: SHELL32.dll
          DLL Name: USER32.dll
          DLL Name: USP10.dll
          DLL Name: WINMM.dll
          DLL Name: WINSPOOL.DRV
          DLL Name: WSOCK32.dll
          DLL Name: comdlg32.dll
          DLL Name: libgmp-10.dll
          DLL Name: msvcrt.dll
          DLL Name: ole32.dll

Thank you

        Dieter

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: Phillip Lord <phillip.lord@russet.org.uk>,  arash@gnu.org,
>   corwin@bru.st,  emacs-devel@gnu.org
> Date: Mon, 21 Feb 2022 22:59:07 +0100
> 
> > Run this in the 'bin/' directory where you have executable programs
> > and DLLs, and it will show a sorted list of all the DLLs needed by
> > those programs and DLLs.  You will need to recognize and ignore any
> > system DLLs, such as msvcrt.dll, KERNEL32.dll, USER32.dll, WS2_32.dll
> > etc. in the list; for the rest, make sure every one of them is in the
> > same 'bin/' directory.  Each time you add some DLL to the 'bin/'
> > directory, re-run the above command again, because adding DLLs can
> > potentially add dependencies.  Repeat this process until all the DLLs
> > shown by the command are in the same directory: then you are done.
> 
> When trying above iterative approach, I just find libgmp and after
> adding libgmp-10.dll to the folder there are no further dependencies to
> see!?
> 
>   uidg1626@VSL7V7PW MINGW64 ~/emacs-build/build/emacs-28.0.91/bin
>   $ objdump.exe -p *.exe *.dll |grep "DLL Name"|sort -u
>           DLL Name: ADVAPI32.dll
>           DLL Name: COMCTL32.dll
>           DLL Name: GDI32.dll
>           DLL Name: KERNEL32.dll
>           DLL Name: MPR.dll
>           DLL Name: SHELL32.dll
>           DLL Name: USER32.dll
>           DLL Name: USP10.dll
>           DLL Name: WINMM.dll
>           DLL Name: WINSPOOL.DRV
>           DLL Name: WSOCK32.dll
>           DLL Name: comdlg32.dll
>           DLL Name: libgmp-10.dll
>           DLL Name: msvcrt.dll
>           DLL Name: ole32.dll

The dependency on the optional DLLs is implicit, because Emacs loads
them dynamically at run time.  You need to place the DLLs mentioned in
dynamic-library-alist defined in w32-win.el in the bin/ directory, and
_then_ repeat the process.  That's because we are talking about the
dependencies of those dynamically-loaded DLLs.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: Phillip Lord <phillip.lord@russet.org.uk>,  arash@gnu.org,
>>   corwin@bru.st,  emacs-devel@gnu.org
>> Date: Mon, 21 Feb 2022 22:59:07 +0100
>> 
>> > Run this in the 'bin/' directory where you have executable programs
>> > and DLLs, and it will show a sorted list of all the DLLs needed by
>> > those programs and DLLs.  You will need to recognize and ignore any
>> > system DLLs, such as msvcrt.dll, KERNEL32.dll, USER32.dll, WS2_32.dll
>> > etc. in the list; for the rest, make sure every one of them is in the
>> > same 'bin/' directory.  Each time you add some DLL to the 'bin/'
>> > directory, re-run the above command again, because adding DLLs can
>> > potentially add dependencies.  Repeat this process until all the DLLs
>> > shown by the command are in the same directory: then you are done.
>> 
>> When trying above iterative approach, I just find libgmp and after
>> adding libgmp-10.dll to the folder there are no further dependencies to
>> see!?
>> 
>>   uidg1626@VSL7V7PW MINGW64 ~/emacs-build/build/emacs-28.0.91/bin
>>   $ objdump.exe -p *.exe *.dll |grep "DLL Name"|sort -u
>>           DLL Name: ADVAPI32.dll
>>           DLL Name: COMCTL32.dll
>>           DLL Name: GDI32.dll
>>           DLL Name: KERNEL32.dll
>>           DLL Name: MPR.dll
>>           DLL Name: SHELL32.dll
>>           DLL Name: USER32.dll
>>           DLL Name: USP10.dll
>>           DLL Name: WINMM.dll
>>           DLL Name: WINSPOOL.DRV
>>           DLL Name: WSOCK32.dll
>>           DLL Name: comdlg32.dll
>>           DLL Name: libgmp-10.dll
>>           DLL Name: msvcrt.dll
>>           DLL Name: ole32.dll
>
> The dependency on the optional DLLs is implicit, because Emacs loads
> them dynamically at run time.  You need to place the DLLs mentioned in
> dynamic-library-alist defined in w32-win.el in the bin/ directory, and
> _then_ repeat the process.  That's because we are talking about the
> dependencies of those dynamically-loaded DLLs.

I see, thank you for this concrete recipe.

I'm a bit baffled, why is in dynamic-library-alist no harfbuzz but
gccjit, gdiplus and shlwapi are there which seem not to be included in
the list of mingw packages mentioned in README.W64?

Emacs-28.0.91
  dynamic-library-alist                                nt/README.W64
(
 (xpm "libxpm.dll" "xpm4.dll" "libXpm-nox4.dll")       mingw-w64-x86_64-xpm-nox \	     
 (tiff "libtiff-5.dll" "libtiff3.dll" "libtiff.dll")   mingw-w64-x86_64-libtiff \	     
 (gif "libgif-7.dll")				       mingw-w64-x86_64-giflib \	     
 (png "libpng16.dll" "libpng16-16.dll")		       mingw-w64-x86_64-libpng \	     
 (jpeg "libjpeg-8.dll")				       mingw-w64-x86_64-libjpeg-turbo \   
 (svg "librsvg-2-2.dll")			       mingw-w64-x86_64-librsvg \	     
 (lcms2 "liblcms2-2.dll")			       mingw-w64-x86_64-lcms2 \	     
 (json "libjansson-4.dll")			       mingw-w64-x86_64-jansson \	     
 (libxml2 "libxml2-2.dll" "libxml2.dll")	       mingw-w64-x86_64-libxml2 \	     
 (gnutls "libgnutls-30.dll")			       mingw-w64-x86_64-gnutls \	     
 (zlib "zlib1.dll" "libz-1.dll")		       mingw-w64-x86_64-zlib \	     
						       mingw-w64-x86_64-harfbuzz          
 (gdk-pixbuf "libgdk_pixbuf-2.0-0.dll")
 (glib "libglib-2.0-0.dll")
 (gio "libgio-2.0-0.dll")
 (gobject "libgobject-2.0-0.dll")
  ;; below DLLs are not part of Emacs-28.0.91 for Windows!
 (gccjit "libgccjit-0.dll")
 (gdiplus "gdiplus.dll")
 (shlwapi "shlwapi.dll")
 )

Are gdiplus and shlwapi somehow missing in the current Emacs Windows
distribution?  (Existing ert tests don't indicate that.)


               Dieter

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Arash Esbati]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> Are gdiplus and shlwapi somehow missing in the current Emacs Windows
> distribution?  (Existing ert tests don't indicate that.)

I think they are provided by MS-Windows itself, so you don't need to
package them with Emacs.

Best, Arash

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Hallo Arash

Nice to read from you.

Arash Esbati <arash@gnu.org> writes:

> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
>
>> Are gdiplus and shlwapi somehow missing in the current Emacs Windows
>> distribution?  (Existing ert tests don't indicate that.)
>
> I think they are provided by MS-Windows itself, so you don't need to
> package them with Emacs.

Right, found the DLLs in a Windows system folder. :-)

Thank you

      Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: arash@gnu.org,  corwin@bru.st,  emacs-devel@gnu.org,
>   phillip.lord@russet.org.uk
> Date: Wed, 23 Feb 2022 07:44:50 +0100
> 
> > The dependency on the optional DLLs is implicit, because Emacs loads
> > them dynamically at run time.  You need to place the DLLs mentioned in
> > dynamic-library-alist defined in w32-win.el in the bin/ directory, and
> > _then_ repeat the process.  That's because we are talking about the
> > dependencies of those dynamically-loaded DLLs.
> 
> I see, thank you for this concrete recipe.
> 
> I'm a bit baffled, why is in dynamic-library-alist no harfbuzz

Sorry, forgot about harfbuzz.  We load that one by name.  (I can
explain why, but I don't think it's relevant to this discussion.)  The
name is fixed: libharfbuzz-0.dll.

> but gccjit, gdiplus and shlwapi are there

Ignore gdiplus and shlwapi, they are system libraries that you don't
need (and cannot, for obvious legal reasons) provide in the binary
package.  The reasons they are there are historical, and again
irrelevant.

Re: Acquiring dependencies for Windows builds, was: Native compilation on Windows

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: arash@gnu.org,  corwin@bru.st,  emacs-devel@gnu.org,
>>   phillip.lord@russet.org.uk
>> Date: Wed, 23 Feb 2022 07:44:50 +0100
>> 
>> > The dependency on the optional DLLs is implicit, because Emacs loads
>> > them dynamically at run time.  You need to place the DLLs mentioned in
>> > dynamic-library-alist defined in w32-win.el in the bin/ directory, and
>> > _then_ repeat the process.  That's because we are talking about the
>> > dependencies of those dynamically-loaded DLLs.
>> 
>> I see, thank you for this concrete recipe.
>> 
>> I'm a bit baffled, why is in dynamic-library-alist no harfbuzz
>
> Sorry, forgot about harfbuzz.  We load that one by name.  (I can
> explain why, but I don't think it's relevant to this discussion.)  The
> name is fixed: libharfbuzz-0.dll.

Don't worry, I only hoped that dynamic-library-alist might be the only
"source of truth".  OK, so we also need take care of libharfbuzz-0.dll.

Furthermore, it seems that your recipe and can save about 10 DLLs in
comparison to the build-dep-zips.py approach (So far I tested this only
with etc/w32-feature.el).

It's drawback is that such an iterative approach is painful to do by
hand.  I'd like to do this in a script and call below MinGW64 command
from within Emacs

  objdump.exe -p *.exe *.dll| fgrep DLL

My idea is to do something like the following:

  (let ((Objdump-buffer (get-buffer-create "*objdump*"))
        (Bin-folder "~/emacs-build/build/emacs-28.0.91/bin/")
        (Objdump "d:/appl/MSYS2/mingw64/bin/objdump.exe"))
    (setq default-directory Bin-folder)
    (with-current-buffer Objdump-buffer
      ;;(goto-char (point-min))
      (erase-buffer)
      (shell-command (concat Objdump " -p *.exe *.dll") Objdump-buffer)
      (keep-lines "	DLL Name:")
      ;; And then filtering steps
      ;; Iteratively copying files into the Bin-folder..
      
Which brings output into the Objdump-buffer but I'm not sure if above
approach might miss the - possibly - necessary MinGW64 environment
information, don't know if it makes a difference when objdump.exe is
called from the MinGW64 shell or with shell-command .  How would you
bring the output of objdump into Emacs?

>> but gccjit, gdiplus and shlwapi are there
>
> Ignore gdiplus and shlwapi, they are system libraries that you don't
> need (and cannot, for obvious legal reasons) provide in the binary
> package.  The reasons they are there are historical, and again
> irrelevant.

It seems to me that all MSYS DLLs on which Emacs depends have a name
component "lib", whereas the Windows libraries have no such substring.
(So I can easily filter them out.)

Thank you very much

          Dieter

Re: Acquiring dependencies for Windows builds, was: Native compilation on Windows

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: arash@gnu.org,  corwin@bru.st,  phillip.lord@russet.org.uk,
>   emacs-devel@gnu.org
> Date: Sun, 27 Feb 2022 11:49:29 +0100
> 
> It's drawback is that such an iterative approach is painful to do by
> hand.  I'd like to do this in a script and call below MinGW64 command
> from within Emacs
> 
>   objdump.exe -p *.exe *.dll| fgrep DLL
> 
> My idea is to do something like the following:
> 
>   (let ((Objdump-buffer (get-buffer-create "*objdump*"))
>         (Bin-folder "~/emacs-build/build/emacs-28.0.91/bin/")
>         (Objdump "d:/appl/MSYS2/mingw64/bin/objdump.exe"))
>     (setq default-directory Bin-folder)
>     (with-current-buffer Objdump-buffer
>       ;;(goto-char (point-min))
>       (erase-buffer)
>       (shell-command (concat Objdump " -p *.exe *.dll") Objdump-buffer)
>       (keep-lines "	DLL Name:")
>       ;; And then filtering steps
>       ;; Iteratively copying files into the Bin-folder..

First, no need to hard-code the location of objdump.exe:
executable-find should be able to find it.

Second, you change the default-directory in the wrong buffer, I think.

More importantly, I don't see where you have the list of all the DLLs
that Emacs loads.  You need to start from that list.

> Which brings output into the Objdump-buffer but I'm not sure if above
> approach might miss the - possibly - necessary MinGW64 environment
> information, don't know if it makes a difference when objdump.exe is
> called from the MinGW64 shell or with shell-command .

I don't think I follow your fears here.  What exactly did you think
could go wrong?  What do you mean by "necessary MinGW64 environment
information"?

> How would you bring the output of objdump into Emacs?

shell-command accepts an optional argument to that effect.

> It seems to me that all MSYS DLLs on which Emacs depends have a name
> component "lib", whereas the Windows libraries have no such substring.
> (So I can easily filter them out.)

There are exceptions: zlib1.dll.  For now, it's the only exception
AFAIK, but watch out for others.

Re: Acquiring dependencies for Windows builds, was: Native compilation on Windows

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

> First, no need to hard-code the location of objdump.exe:
> executable-find should be able to find it.

Nice, thank you. :-)

> Second, you change the default-directory in the wrong buffer, I think.
> More importantly, I don't see where you have the list of all the DLLs
> that Emacs loads.  You need to start from that list.

Indeed

>> Which brings output into the Objdump-buffer but I'm not sure if above
>> approach might miss the - possibly - necessary MinGW64 environment
>> information, don't know if it makes a difference when objdump.exe is
>> called from the MinGW64 shell or with shell-command .
>
> I don't think I follow your fears here.  What exactly did you think
> could go wrong?  What do you mean by "necessary MinGW64 environment
> information"?

I thought, maybe, a call to objdump.exe needs further environment
information than Emacs is providing..

>> It seems to me that all MSYS DLLs on which Emacs depends have a name
>> component "lib", whereas the Windows libraries have no such substring.
>> (So I can easily filter them out.)
>
> There are exceptions: zlib1.dll.  For now, it's the only exception
> AFAIK, but watch out for others.

Luckily zlib1.dll contains also "lib". :-)

In the following I'm using copy-file. But I'm wondering why the argument
OK-IF-ALREADY-EXISTS can't be used to skip a copy process if the file
exists?  In the following some files will be multiple times overwritten..

Here's a further attempt (and it seems to work for me):

  (defun Copy-dll ( Dll)
    (let ((Build-folder "~/emacs-build/build/emacs-28.0.91/bin/")
          (Exists (file-exists-p Dll)))
      (if Exists
          (copy-file Dll Build-folder 'overwrite)
        (message (concat "[Info] Can't find " Dll)))))

  (let ((Objdump-buffer (get-buffer-create "*objdump*"))
        (Build-folder "~/emacs-build/build/emacs-28.0.91/bin/")
        (Dll-folder "d:/appl/MSYS2/mingw64/bin/")
        (Objdump (executable-find "objdump.exe")) ;mingw64 is in PATH
        ;; starting dependency list from dynamic-library-alist
        (Dependencies (add-to-list
                       'dynamic-library-alist
                       '(harfbuzz "libharfbuzz-0.dll"))) ;add harfbuzz entry
        (Build-deps)		 ; current dlls in the build folder
        (Dll nil)) ; temp variable

    ;; prepare (flatten) dependency list
    (dolist (D Dependencies)
      (add-to-list 'Dll (cdr D)))
    (setq Dependencies (flatten-tree Dll))
  
    ;; iterative proceedure: Add to Build-deps the dependencies
    (while (< (length Build-deps) (length Dependencies))
      (message "Number of dependencies %s" (length Dependencies ))
      (dolist (D Dependencies)
        (setq D (concat Dll-folder D))
        ;; Copy DLL from Dll-folder into build-folder
        (Copy-dll D))

      ;; Check the dependencies of *.exe and *.dll in the build folder
      (with-current-buffer Objdump-buffer
        (setq default-directory Build-folder)
        (erase-buffer)
        (shell-command (concat Objdump " -p *.exe *.dll") Objdump-buffer)
        ;; filtering steps
        (keep-lines "	DLL Name:")
        (keep-lines "lib")			;remove Windows DLLs
        (delete-duplicate-lines (point-min) (point-max))
        (goto-char (point-min))
        ;; remove cruft
        (while (re-search-forward "^[\t]DLL Name: " nil t)
          (replace-match ""))
        ;; Update dependencies in build folder
        (setq Dependencies
              (split-string	;default splitter is \n among others
               (buffer-substring-no-properties (point-min) (point-max)))))

      ;; Update Build-deps in Build-folder
      (setq Build-deps (directory-files Build-folder nil "dll$"))))


      Dieter

Re: Acquiring dependencies for Windows builds, was: Native compilation on Windows

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: arash@gnu.org,  corwin@bru.st,  emacs-devel@gnu.org,
>   phillip.lord@russet.org.uk
> Date: Mon, 28 Feb 2022 22:51:57 +0100
> 
> >> It seems to me that all MSYS DLLs on which Emacs depends have a name
> >> component "lib", whereas the Windows libraries have no such substring.
> >> (So I can easily filter them out.)
> >
> > There are exceptions: zlib1.dll.  For now, it's the only exception
> > AFAIK, but watch out for others.
> 
> Luckily zlib1.dll contains also "lib". :-)

You are relying on luck too much: look in your Windows/system32
directory, and you will see there quite a few system DLLs that match
"lib".  My suggestion is to have an explicit list of system DLLs that
should be filtered out.

> In the following I'm using copy-file. But I'm wondering why the argument
> OK-IF-ALREADY-EXISTS can't be used to skip a copy process if the file
> exists?

I'm not sure I understand, but you should stop copying when you find
the first existing DLL for each feature in dynamic-library-alist.  We
arrange the DLLs there in the descending order of
versions/importance/quality, so the leftmost DLL that is found is
always the best, and should not be overwritten by any existing DLLs
for the same feature whose names are to the right of that.  For
example:

     '(tiff "libtiff-5.dll" "libtiff3.dll" "libtiff.dll")

If you find libtiff-5.dll, you should not overwrite it with
libtiff3.dll, even if the latter exists.

Also, I don't see that you ignore DLLs that don't match the version
with which Emacs was compiled.  E.g., this part of
dynamic-library-alist:

       (if (>= libgif-version 50100)
	   ;; Yes, giflib 5.0 uses 6 as the major version of the API,
	   ;; and giflib 5.1 uses 7, thus "libgif-7.dll" and
	   ;; "libgif-6.dll" below (giflib 4.x used 5 as the major API
	   ;; version).  giflib5.dll is from the lua-files project,
	   ;; and gif.dll is from luapower.
	   '(gif "libgif-7.dll")
	 (if (>= libgif-version 50000)
	     '(gif "libgif-6.dll" "giflib5.dll" "gif.dll")
	 '(gif "libgif-5.dll" "giflib4.dll" "libungif4.dll" "libungif.dll")))

means that Emacs compiled against giflib 5.1 and later should not try
to use libgif-6.dll or libgif-5.dll, it should only use libgif-7.dll.
We do this when a newer version is binary incompatible with an older
one, so you cannot just choose whichever DLL you have, you must use
the one that is binary-compatible with Emacs.

Thanks.

Re: Acquiring dependencies for Windows builds

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: arash@gnu.org,  corwin@bru.st,  emacs-devel@gnu.org,
>>   phillip.lord@russet.org.uk
>> Date: Mon, 28 Feb 2022 22:51:57 +0100
>> 
>> >> It seems to me that all MSYS DLLs on which Emacs depends have a name
>> >> component "lib", whereas the Windows libraries have no such substring.
>> >> (So I can easily filter them out.)
>> >
>> > There are exceptions: zlib1.dll.  For now, it's the only exception
>> > AFAIK, but watch out for others.
>> 
>> Luckily zlib1.dll contains also "lib". :-)
>
> You are relying on luck too much: look in your Windows/system32
> directory, and you will see there quite a few system DLLs that match
> "lib".  My suggestion is to have an explicit list of system DLLs that
> should be filtered out.

I understand, thanks.  I'll grab all DLL filenames of the
Windows/system32 folder and filter them out.  Or how would you build or
hard-code such a filter list?

>> In the following I'm using copy-file. But I'm wondering why the argument
>> OK-IF-ALREADY-EXISTS can't be used to skip a copy process if the file
>> exists?
>
> I'm not sure I understand,

Would it make sense to you to have the (additonal) capability for
copy-file to do nothing if the target file already exists?

At the moment I can 1) trigger an error, 2) overwrite an existing file
and 3) ask for confirmation either to overwrite or to have an error.  So
3) is actually only an - annoying - way to have 1) or 2), there's no way
to leave an existing file alone without an error.

And here's another thing I stumbled over: An excerpt of copy-file's
documentation string:

  OK-IF-ALREADY-EXISTS is an integer, request confirmation from the user
  about overwriting; this is what happens in interactive use with M-x.

The interactive part in the documentation is unnecessary, I think, the
request happens for me in batch mode as well.

> but you should stop copying when you find
> the first existing DLL for each feature in dynamic-library-alist.  We
> arrange the DLLs there in the descending order of
> versions/importance/quality, so the leftmost DLL that is found is
> always the best, and should not be overwritten by any existing DLLs
> for the same feature whose names are to the right of that.  For
> example:
>
>      '(tiff "libtiff-5.dll" "libtiff3.dll" "libtiff.dll")

Actually my code was copying all of above tiff DLLs to the bin
folder.

> If you find libtiff-5.dll, you should not overwrite it with
> libtiff3.dll, even if the latter exists.

I see, I'll just copy the available, leftmost DLLs.

> Also, I don't see that you ignore DLLs that don't match the version
> with which Emacs was compiled.  E.g., this part of
> dynamic-library-alist:
>
>        (if (>= libgif-version 50100)
> 	   ;; Yes, giflib 5.0 uses 6 as the major version of the API,
> 	   ;; and giflib 5.1 uses 7, thus "libgif-7.dll" and
> 	   ;; "libgif-6.dll" below (giflib 4.x used 5 as the major API
> 	   ;; version).  giflib5.dll is from the lua-files project,
> 	   ;; and gif.dll is from luapower.
> 	   '(gif "libgif-7.dll")
> 	 (if (>= libgif-version 50000)
> 	     '(gif "libgif-6.dll" "giflib5.dll" "gif.dll")
> 	 '(gif "libgif-5.dll" "giflib4.dll" "libungif4.dll" "libungif.dll")))

Please tell me where I can find above definition of
dynamic-library-alist?  Mine (Emacs-27.2) just says:

  dynamic-library-alist is a variable defined in ‘C source code’.

  ((harfbuzz "libharfbuzz-0.dll")
   (gdiplus "gdiplus.dll")
   (shlwapi "shlwapi.dll")
   (xpm "libxpm.dll" "xpm4.dll" "libXpm-nox4.dll")
   (png "libpng16.dll" "libpng16-16.dll")
   (tiff "libtiff-5.dll" "libtiff3.dll" "libtiff.dll")
   (jpeg "libjpeg-8.dll")
   (gif "libgif-7.dll")
   (svg "librsvg-2-2.dll")
   (webp "libwebp-7.dll" "libwebp.dll")
   (sqlite3 "libsqlite3-0.dll")
   (gdk-pixbuf "libgdk_pixbuf-2.0-0.dll")
   (glib "libglib-2.0-0.dll")
   (gio "libgio-2.0-0.dll")
   (gobject "libgobject-2.0-0.dll")
   (gnutls "libgnutls-30.dll")
   (libxml2 "libxml2-2.dll" "libxml2.dll")
   (zlib "zlib1.dll" "libz-1.dll")
   (lcms2 "liblcms2-2.dll")
   (json "libjansson-4.dll")
   (gccjit "libgccjit-0.dll"))

> means that Emacs compiled against giflib 5.1 and later should not try
> to use libgif-6.dll or libgif-5.dll, it should only use libgif-7.dll.
> We do this when a newer version is binary incompatible with an older
> one, so you cannot just choose whichever DLL you have, you must use
> the one that is binary-compatible with Emacs.

I'm struggling to understand that, I thought ./configure would sort out
what is necessary and available on my system?

Thank you for your interest

      Dieter

Re: Acquiring dependencies for Windows builds

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: arash@gnu.org,  corwin@bru.st,  emacs-devel@gnu.org,
>   phillip.lord@russet.org.uk
> Date: Tue, 08 Mar 2022 16:52:52 +0100
> 
> >> Luckily zlib1.dll contains also "lib". :-)
> >
> > You are relying on luck too much: look in your Windows/system32
> > directory, and you will see there quite a few system DLLs that match
> > "lib".  My suggestion is to have an explicit list of system DLLs that
> > should be filtered out.
> 
> I understand, thanks.  I'll grab all DLL filenames of the
> Windows/system32 folder and filter them out.  Or how would you build or
> hard-code such a filter list?

I'd maintain a hand-crafted list.  There aren't so many system DLLs
you will see in the dependencies.

> >> In the following I'm using copy-file. But I'm wondering why the argument
> >> OK-IF-ALREADY-EXISTS can't be used to skip a copy process if the file
> >> exists?
> >
> > I'm not sure I understand,
> 
> Would it make sense to you to have the (additonal) capability for
> copy-file to do nothing if the target file already exists?
> 
> At the moment I can 1) trigger an error, 2) overwrite an existing file
> and 3) ask for confirmation either to overwrite or to have an error.  So
> 3) is actually only an - annoying - way to have 1) or 2), there's no way
> to leave an existing file alone without an error.

I think this is answered by the below:

> > If you find libtiff-5.dll, you should not overwrite it with
> > libtiff3.dll, even if the latter exists.
> 
> I see, I'll just copy the available, leftmost DLLs.

Yes.

> > Also, I don't see that you ignore DLLs that don't match the version
> > with which Emacs was compiled.  E.g., this part of
> > dynamic-library-alist:
> >
> >        (if (>= libgif-version 50100)
> > 	   ;; Yes, giflib 5.0 uses 6 as the major version of the API,
> > 	   ;; and giflib 5.1 uses 7, thus "libgif-7.dll" and
> > 	   ;; "libgif-6.dll" below (giflib 4.x used 5 as the major API
> > 	   ;; version).  giflib5.dll is from the lua-files project,
> > 	   ;; and gif.dll is from luapower.
> > 	   '(gif "libgif-7.dll")
> > 	 (if (>= libgif-version 50000)
> > 	     '(gif "libgif-6.dll" "giflib5.dll" "gif.dll")
> > 	 '(gif "libgif-5.dll" "giflib4.dll" "libungif4.dll" "libungif.dll")))
> 
> Please tell me where I can find above definition of
> dynamic-library-alist?  Mine (Emacs-27.2) just says:
> 
>   dynamic-library-alist is a variable defined in ‘C source code’.
> 
>   ((harfbuzz "libharfbuzz-0.dll")
>    (gdiplus "gdiplus.dll")
>    (shlwapi "shlwapi.dll")
>    (xpm "libxpm.dll" "xpm4.dll" "libXpm-nox4.dll")
>    (png "libpng16.dll" "libpng16-16.dll")
>    (tiff "libtiff-5.dll" "libtiff3.dll" "libtiff.dll")
>    (jpeg "libjpeg-8.dll")
>    (gif "libgif-7.dll")
>    (svg "librsvg-2-2.dll")
>    (webp "libwebp-7.dll" "libwebp.dll")
>    (sqlite3 "libsqlite3-0.dll")
>    (gdk-pixbuf "libgdk_pixbuf-2.0-0.dll")
>    (glib "libglib-2.0-0.dll")
>    (gio "libgio-2.0-0.dll")
>    (gobject "libgobject-2.0-0.dll")
>    (gnutls "libgnutls-30.dll")
>    (libxml2 "libxml2-2.dll" "libxml2.dll")
>    (zlib "zlib1.dll" "libz-1.dll")
>    (lcms2 "liblcms2-2.dll")
>    (json "libjansson-4.dll")
>    (gccjit "libgccjit-0.dll"))

I guess you took this from a running Emacs?  The original is in
w32-win.el, and that's where I took the fragment I've shown.

If you intend to collect the dependencies by running the same Emacs
binary as the one you will be packaging, then using the list from the
running Emacs will solve you the versions problem, because Emacs
filters it to have the DLLs it needs.

> > means that Emacs compiled against giflib 5.1 and later should not try
> > to use libgif-6.dll or libgif-5.dll, it should only use libgif-7.dll.
> > We do this when a newer version is binary incompatible with an older
> > one, so you cannot just choose whichever DLL you have, you must use
> > the one that is binary-compatible with Emacs.
> 
> I'm struggling to understand that, I thought ./configure would sort out
> what is necessary and available on my system?

Not configure, but Emacs itself when it starts up.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: Arash Esbati <arash@gnu.org>
> Cc: Corwin Brust <corwin@bru.st>,  Eli Zaretskii <eliz@gnu.org>,  Emacs
>  developers <emacs-devel@gnu.org>,  Phillip Lord
>  <phillip.lord@russet.org.uk>
> Date: Sun, 30 Jan 2022 22:33:40 +0100
> 
> I'm not tracking this development, so please bear with me.  Is the plan
> to unpack the file emacs-28.0.91-no-deps.zip into, say C:\emacs and
> unpack the .dll from emacs-28-deps.zip[1] into C:\emacs\bin?  If so, I'm
> afraid that the compiled features will not work since dependencies are
> missing in emacs-28-deps.zip, e.g.:

Yes, all the dependency DLLs, recursively, should be available to
Emacs, otherwise it will fail to load some of the DLLs, and the result
will behave as if the relevant DLLs (such as libgnutls-30.dll) were
absent.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[chad]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

On Sat, Jan 29, 2022 at 5:24 PM H. Dieter Wilhelm <
dieter@duenenhof-wilhelm.de> wrote:

> Thanks, do you know what jannson is needed for?
>

It lets emacs do a bunch of JSON processing in C rather than elisp, which
is very helpful for a bunch of cases, especially some LSP modules.

Just in case: Language Server Protocol; a semi-standard way for outboard
toolsets like specialized compiler suites to communicate with editors
(primarily VSCode, but somewhat generalized and so supported in other
editors as well) about things like code completion and language-aware
editing. The protocol ends up being pretty chatty, often sending a lot of
json back and forth, and the pure-elisp method of converting it had a
pretty high overhead.

HtH,
~Chad

[-- Attachment #2: Type: text/html, Size: 1165 bytes --]

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

chad <yandros@gmail.com> writes:

> On Sat, Jan 29, 2022 at 5:24 PM H. Dieter Wilhelm <dieter@duenenhof-wilhelm.de> wrote:
>
>  Thanks, do you know what jannson is needed for?
>
> It lets emacs do a bunch of JSON processing in C rather than elisp,
> which is very helpful for a bunch of cases, especially some LSP
> modules.
>
> Just in case: Language Server Protocol; a semi-standard way for
> outboard toolsets like specialized compiler suites to communicate with
> editors (primarily VSCode, but somewhat generalized and so supported
> in other editors as well) about things like code completion and
> language-aware editing. The protocol ends up being pretty chatty,
> often sending a lot of json back and forth, and the pure-elisp method
> of converting it had a pretty high overhead.
>

Thanks for the explanation Chad, even though I never got in contact with
LSP, your remarks gave me the idea to look in another way to the
"standard" dependencies of Emacs.  Namely, that the "standard" is what
developers are are activating - or trying to activate - in the configure
script.

--
        Dieter

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Hello Chad

could you please give a minimal setup to test the jansson library for
the language server protocol?  Is it enough to create a json file, setq
a default server address and search for function names? :-)

Thanks a lot

       Dieter

chad <yandros@gmail.com> writes:

> On Sat, Jan 29, 2022 at 5:24 PM H. Dieter Wilhelm <dieter@duenenhof-wilhelm.de> wrote:
>
>  Thanks, do you know what jannson is needed for?
>
> It lets emacs do a bunch of JSON processing in C rather than elisp, which is very helpful for a bunch of cases, especially some LSP modules.
>
> Just in case: Language Server Protocol; a semi-standard way for outboard toolsets like specialized compiler suites to communicate with editors
> (primarily VSCode, but somewhat generalized and so supported in other editors as well) about things like code completion and language-aware
> editing. The protocol ends up being pretty chatty, often sending a lot of json back and forth, and the pure-elisp method of converting it had a
> pretty high overhead.
>
> HtH,
> ~Chad
>

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: Corwin Brust <corwin@bru.st>,  Eli Zaretskii <eliz@gnu.org>,  Phillip
>  Lord <phillip.lord@russet.org.uk>,  Emacs developers <emacs-devel@gnu.org>
> Date: Tue, 01 Feb 2022 18:17:50 +0100
> 
> Hello Chad
> 
> could you please give a minimal setup to test the jansson library for
> the language server protocol?  Is it enough to create a json file, setq
> a default server address and search for function names? :-)

I'm not Chad, but what's wrong with test/src/json-tests.el?

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: Corwin Brust <corwin@bru.st>,  Eli Zaretskii <eliz@gnu.org>,  Phillip
>>  Lord <phillip.lord@russet.org.uk>,  Emacs developers <emacs-devel@gnu.org>
>> Date: Tue, 01 Feb 2022 18:17:50 +0100
>> 
>> Hello Chad
>> 
>> could you please give a minimal setup to test the jansson library for
>> the language server protocol?  Is it enough to create a json file, setq
>> a default server address and search for function names? :-)
>
> I'm not Chad

I'm sorry, I removed the wrong quotations from the following post

  From: chad <yandros@gmail.com>
  Subject: Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed
  To: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
  Cc: Corwin Brust <corwin@bru.st>, Eli Zaretskii <eliz@gnu.org>,  Phillip Lord <phillip.lord@russet.org.uk>, Emacs developers <emacs-devel@gnu.org>

> , but what's wrong with test/src/json-tests.el?

Hurraaah, there are already tests for it!

Thanks a lot Eli.

How nice, we can test the dists with image-tests.el, json-tests.el,
lcms-tests.el and xml-tests.el.  And I had the impression that tests are
only working in batch mode!

For the archiving (zlib) calling Info should work, for gnutls calling
eww and for harfbuzz, I think, checking the persian language environment
could do, or can we use for this some character or font tests?

      Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: yandros@gmail.com,  corwin@bru.st,  phillip.lord@russet.org.uk,
>   emacs-devel@gnu.org
> Date: Tue, 01 Feb 2022 19:59:37 +0100
> 
> For the archiving (zlib) calling Info should work, for gnutls calling
> eww and for harfbuzz, I think, checking the persian language environment
> could do, or can we use for this some character or font tests?

HarfBuzz's functionality has nothing to do with Persian specifically,
it is a general-purpose text shaping engine.  Just "C-h h" followed by
"C-u C-x =" on some exotic characters should do to test it's working.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Robert Pluim]

>>>>> On Tue, 01 Feb 2022 21:23:32 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
    >> Cc: yandros@gmail.com,  corwin@bru.st,  phillip.lord@russet.org.uk,
    >> emacs-devel@gnu.org
    >> Date: Tue, 01 Feb 2022 19:59:37 +0100
    >> 
    >> For the archiving (zlib) calling Info should work, for gnutls calling
    >> eww and for harfbuzz, I think, checking the persian language environment
    >> could do, or can we use for this some character or font tests?

    Eli> HarfBuzz's functionality has nothing to do with Persian specifically,
    Eli> it is a general-purpose text shaping engine.  Just "C-h h" followed by
    Eli> "C-u C-x =" on some exotic characters should do to test it's working.

etc/HELLO is a good source of those.

Robert
-- 

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Tue, 01 Feb 2022 21:23:32 +0200, Eli Zaretskii <eliz@gnu.org> said:
>
>     >> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>     >> Cc: yandros@gmail.com,  corwin@bru.st,  phillip.lord@russet.org.uk,
>     >> emacs-devel@gnu.org
>     >> Date: Tue, 01 Feb 2022 19:59:37 +0100
>     >> 
>     >> For the archiving (zlib) calling Info should work, for gnutls calling
>     >> eww and for harfbuzz, I think, checking the persian language environment
>     >> could do, or can we use for this some character or font tests?
>
>     Eli> HarfBuzz's functionality has nothing to do with Persian specifically,
>     Eli> it is a general-purpose text shaping engine.  Just "C-h h" followed by
>     Eli> "C-u C-x =" on some exotic characters should do to test it's working.
>
> etc/HELLO is a good source of those.

Indeed, C-h h, thanks :-)

-- 
        Dieter

Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Cc: yandros@gmail.com,  corwin@bru.st,  emacs-devel@gnu.org,
>   phillip.lord@russet.org.uk,  Eli Zaretskii <eliz@gnu.org>
> Date: Wed, 02 Feb 2022 11:36:05 +0100
> 
>     Eli> HarfBuzz's functionality has nothing to do with Persian specifically,
>     Eli> it is a general-purpose text shaping engine.  Just "C-h h" followed by
>     Eli> "C-u C-x =" on some exotic characters should do to test it's working.
> 
> etc/HELLO is a good source of those.

Yes, that's what "C-h h" is all about ;-)

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Robert Pluim]

>>>>> On Wed, 02 Feb 2022 14:57:25 +0200, Eli Zaretskii <eliz@gnu.org> said:

    >> From: Robert Pluim <rpluim@gmail.com>
    >> Cc: yandros@gmail.com,  corwin@bru.st,  emacs-devel@gnu.org,
    >> phillip.lord@russet.org.uk,  Eli Zaretskii <eliz@gnu.org>
    >> Date: Wed, 02 Feb 2022 11:36:05 +0100
    >> 
    Eli> HarfBuzz's functionality has nothing to do with Persian specifically,
    Eli> it is a general-purpose text shaping engine.  Just "C-h h" followed by
    Eli> "C-u C-x =" on some exotic characters should do to test it's working.
    >> 
    >> etc/HELLO is a good source of those.

    Eli> Yes, that's what "C-h h" is all about ;-)

D'oh. Iʼm sure Iʼve got some reading comprehension lying around here
somewhere, I should really use it sometime. :-)

Robert
-- 

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Wed, 02 Feb 2022 14:57:25 +0200, Eli Zaretskii <eliz@gnu.org> said:
>
>     >> From: Robert Pluim <rpluim@gmail.com>
>     >> Cc: yandros@gmail.com,  corwin@bru.st,  emacs-devel@gnu.org,
>     >> phillip.lord@russet.org.uk,  Eli Zaretskii <eliz@gnu.org>
>     >> Date: Wed, 02 Feb 2022 11:36:05 +0100
>     >> 
>     Eli> HarfBuzz's functionality has nothing to do with Persian specifically,
>     Eli> it is a general-purpose text shaping engine.  Just "C-h h" followed by
>     Eli> "C-u C-x =" on some exotic characters should do to test it's working.
>     >> 
>     >> etc/HELLO is a good source of those.
>
>     Eli> Yes, that's what "C-h h" is all about ;-)
>
> D'oh. Iʼm sure Iʼve got some reading comprehension lying around here
> somewhere, I should really use it sometime. :-)

Have you seen the new language "emoji"? :D (Too sad, Emacs 28 hasn't
learned it..)

     Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Robert Pluim]

>>>>> On Wed, 02 Feb 2022 16:23:22 +0100, "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> said:

    Dieter> Have you seen the new language "emoji"? :D (Too sad, Emacs 28 hasn't
    Dieter> learned it..)

What's the codepoint? Iʼm not aware of a language emoji.

Robert
-- 

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: Eli Zaretskii <eliz@gnu.org>,  yandros@gmail.com,  corwin@bru.st,
>   phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Wed, 02 Feb 2022 16:23:22 +0100
> 
> Have you seen the new language "emoji"? :D (Too sad, Emacs 28 hasn't
> learned it..)

Of course it did!  From Emacs 28's fontset.el:

  (set-fontset-font "fontset-default" 'emoji
                                      ^^^^^^
                    '("Noto Color Emoji" . "iso10646-1") nil 'prepend)

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  yandros@gmail.com,  corwin@bru.st,
>>   phillip.lord@russet.org.uk,  emacs-devel@gnu.org
>> Date: Wed, 02 Feb 2022 16:23:22 +0100
>> 
>> Have you seen the new language "emoji"? :D (Too sad, Emacs 28 hasn't
>> learned it..)
>
> Of course it did!  From Emacs 28's fontset.el:
>
>   (set-fontset-font "fontset-default" 'emoji
>                                       ^^^^^^
>                     '("Noto Color Emoji" . "iso10646-1") nil 'prepend)

Yes and no.

A really brilliant joke is that you can see in etc/HELLO - with Emacs-28
- for the language "emoji" a "WAVING HAND SIGN" icon.

Then I tried a follow-up joke of my own: A simple character emoji ":D"
and claimed that Emacs-28 hasn't learned "emoji".  In Emacs-29 I would
have put in (with C-x 8 e e) a real, graphical emoji.

(Undaunted, I'm trying another one) Can we settle for: Emacs-28 can
"read emoji" and Emacs-29 is able to "speak emoji"?

-- 
    Dieter (emoji in cheek ;-)) 

Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Robert Pluim]

>>>>> On Wed, 02 Feb 2022 18:58:08 +0100, "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> said:
    Dieter> Yes and no.

    Dieter> A really brilliant joke is that you can see in etc/HELLO - with Emacs-28
    Dieter> - for the language "emoji" a "WAVING HAND SIGN" icon.

    Dieter> Then I tried a follow-up joke of my own: A simple character emoji ":D"
    Dieter> and claimed that Emacs-28 hasn't learned "emoji".  In Emacs-29 I would
    Dieter> have put in (with C-x 8 e e) a real, graphical emoji.

In emacs 28 if you know the codepoint(s) you can enter them with
'C-x 8 RET <codepoint>' and they'll display properly. The emacs-29 support for
entering them that youʼre talking about will likely be available as a
package for emacs-28 at some point.

    Dieter> (Undaunted, I'm trying another one) Can we settle for: Emacs-28 can
    Dieter> "read emoji" and Emacs-29 is able to "speak emoji"?

Theyʼre all unicode codepoints and emacs versions previous to 28 could
read them fine. The only thing that happened in emacs-28 is that their
display changed :-)

Robert
-- 

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[chad]

[-- Attachment #1: Type: text/plain, Size: 797 bytes --]

On Tue, Feb 1, 2022 at 12:53 PM Eli Zaretskii <eliz@gnu.org> wrote:

> > could you please give a minimal setup to test the jansson library for
> > the language server protocol?  Is it enough to create a json file, setq
> > a default server address and search for function names? :-)
>
> I'm not Chad, but what's wrong with test/src/json-tests.el?
>

I hope nothing, because that's what I was going to suggest. :-)

I also note that there is some #ifdef WINDOWSNT code in src/json.c that
checks for the jansson library near the top of DEFUN ("json-serialize"...
that suggests it will signal a useful error if it can't find the library,
and json-serialize is heavily exercised in test/src/json-tests.el, so as a
specific check, "cd tests; make json-tests" should do the job.

Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 1281 bytes --]

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

chad <yandros@gmail.com> writes:

> On Tue, Feb 1, 2022 at 12:53 PM Eli Zaretskii <eliz@gnu.org> wrote:
>
>  > could you please give a minimal setup to test the jansson library for
>  > the language server protocol?  Is it enough to create a json file, setq
>  > a default server address and search for function names? :-)
>
>  I'm not Chad, but what's wrong with test/src/json-tests.el?
>
> I hope nothing, because that's what I was going to suggest. :-)
>
> I also note that there is some #ifdef WINDOWSNT code in src/json.c
> that checks for the jansson library near the top of DEFUN
> ("json-serialize"... that suggests it will signal a useful error if it
> can't find the library, and json-serialize is heavily exercised in
> test/src/json-tests.el, so as a specific check, "cd tests; make
> json-tests" should do the job.

Oh, there's also a makefile and a README!  Damn, before your post I
bombed already the mailing list with, potentially, redundant questions.

> Hope that helps,

Yes, thank you :-)

     Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>> Cc: Corwin Brust <corwin@bru.st>,  Eli Zaretskii <eliz@gnu.org>,  Phillip
>>  Lord <phillip.lord@russet.org.uk>,  Emacs developers <emacs-devel@gnu.org>
>> Date: Tue, 01 Feb 2022 18:17:50 +0100
>> 
>> Hello Chad
>> 
>> could you please give a minimal setup to test the jansson library for
>> the language server protocol?  Is it enough to create a json file, setq
>> a default server address and search for function names? :-)
>
> I'm not Chad, but what's wrong with test/src/json-tests.el?


The only thing to be ware of here is if you are running
test/src/json-tests.el that you are running them using just the files in
the Emacs package that you are testing. If you run them using the
makefile from inside an msys2 shell, then are likely to be getting the
dependencies from the msys2 shell.

That's why I added w32-feature.el to etc/ so that it ends up in the
install version. Clearly it makes no sense to replicate everything from
json-tests.el in w32-feature.el. But there needs to be something to make
sure that Emacs is using only it's packaged dependencies.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
>>> Cc: Corwin Brust <corwin@bru.st>,  Eli Zaretskii <eliz@gnu.org>,  Phillip
>>>  Lord <phillip.lord@russet.org.uk>,  Emacs developers <emacs-devel@gnu.org>
>>> Date: Tue, 01 Feb 2022 18:17:50 +0100
>>> 
>>> Hello Chad
>>> 
>>> could you please give a minimal setup to test the jansson library for
>>> the language server protocol?  Is it enough to create a json file, setq
>>> a default server address and search for function names? :-)
>>
>> I'm not Chad, but what's wrong with test/src/json-tests.el?
>
>
> The only thing to be ware of here is if you are running
> test/src/json-tests.el that you are running them using just the files in
> the Emacs package that you are testing. If you run them using the
> makefile from inside an msys2 shell, then are likely to be getting the
> dependencies from the msys2 shell.
>
> That's why I added w32-feature.el to etc/ so that it ends up in the
> install version. Clearly it makes no sense to replicate everything from
> json-tests.el in w32-feature.el. But there needs to be something to make
> sure that Emacs is using only it's packaged dependencies.

Thanks for w32-feature.el just ran your tests successfully with the
latest build of Corwin.  Now I'll have a look on json-tests.el.

Please tell me why you didn't put w32-feature.el in the test/src/
folder?

--
        Dieter

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

On Sat, Feb 5, 2022 at 3:46 PM H. Dieter Wilhelm
<dieter@duenenhof-wilhelm.de> wrote:
>
> Thanks for w32-feature.el just ran your tests successfully with the
> latest build of Corwin.  Now I'll have a look on json-tests.el.

Great news!

>
> Please tell me why you didn't put w32-feature.el in the test/src/
> folder?
>

I'm pretty sure the tests aren't shipped with Emacs, while the etc
folder is.  The header comments say this is expected to be available
for end-users to see if features may be missing under Windows.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Stefan Monnier]

> I'm pretty sure the tests aren't shipped with Emacs, while the etc
> folder is.  The header comments say this is expected to be available
> for end-users to see if features may be missing under Windows.

FWIW, I'll just (re)mention that I think `w32-feature.el` should be made
to work on non-w32 systems and then moved to `lisp`.


        Stefan

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I'm pretty sure the tests aren't shipped with Emacs, while the etc
>> folder is.  The header comments say this is expected to be available
>> for end-users to see if features may be missing under Windows.
>
> FWIW, I'll just (re)mention that I think `w32-feature.el` should be made
> to work on non-w32 systems and then moved to `lisp`.

From a cursory glance I can't see code which might be restricted to
Windows or "w32" systems, except the naming.  Just ran the tests of
w32-features.el on:

GNU Emacs 28.0.91 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.22.30,
cairo version 1.15.10) of 2022-01-14

And the result is as expected for my (default) configuration:

  Selector: t
  Passed:  13
  Failed:  3 (3 unexpected)
  Skipped: 0
  Total:   16/16

  Started at:   2022-02-06 18:05:36+0100
  Finished.
  Finished at:  2022-02-06 18:05:36+0100

  ...F.F...F......

  F feature-harfbuzz
      (ert-test-failed
       ((should
         (eq 'harfbuzz
              (car
               (frame-parameter nil ...))))
        :form
        (eq harfbuzz ftcrhb)
        :value nil))

  F feature-json
      (ert-test-failed
       ((should
         (fboundp 'json-serialize))
        :form
        (fboundp json-serialize)
        :value nil))

  F feature-optimization
      (ert-test-failed
       ((should
         (string-match-p "CFLAGS=-O2" system-configuration-options))
        :form
        (string-match-p "CFLAGS=-O2" "")
        :value nil))

Please tell me what else do you expect from w32-feature.el, maybe
extensive build information for any platform? :-)

   Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Stefan Monnier]

> From a cursory glance I can't see code which might be restricted to
> Windows or "w32" systems, except the naming.  Just ran the tests of
> w32-features.el on:
[...]
> Please tell me what else do you expect from w32-feature.el, maybe
> extensive build information for any platform? :-)

Something like that, yes.  But mostly change the output such that
missing features (or different compilation options) aren't all displayed
as scary test errors but with a more informational tone, I guess.


        Stefan

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> From a cursory glance I can't see code which might be restricted to
>> Windows or "w32" systems, except the naming.  Just ran the tests of
>> w32-features.el on:
> [...]
>> Please tell me what else do you expect from w32-feature.el, maybe
>> extensive build information for any platform? :-)
>
> Something like that, yes.  But mostly change the output such that
> missing features (or different compilation options) aren't all displayed
> as scary test errors but with a more informational tone, I guess.

From the old thread and your remarks it seems to me that this "feature"
library should fulfil two aims: Testing features when building Emacs
distros (especially under Windows) and providing information for users
without the complete source tree.

I took up the idea from Alan Third and completed some missing "standard"
features (from nt/INSTALL.W64):

  (defun insert-feature (description test)
    (indent-to 2)
    (insert (if test "✔" "✖"))
    (indent-to 5)
    (insert description)
    (insert "\n"))

  ;; feature list from nt/INSTALL.W64 
  (defun list-features ()
    (interactive)
    (switch-to-buffer
     (get-buffer-create "*Features*"))
    (read-only-mode -1)
    (erase-buffer)
    (insert "Standard Emacs Features:\n")

    ;; harfbuzz
    (insert-feature "Harfbuzz - text shaping library "
                    (eq 'harfbuzz 
                        (car (frame-parameter nil 'font-backend))))
    ;; zlib
    (insert-feature "Zlib - compression library" (zlib-available-p))
    ;; libxml
    (insert-feature "xml2 - parser library" (libxml-available-p))
    ;; lcms lcms2-available-p
    (insert-feature "lcms2 - color translator library" (image-type-available-p 'pbm))
    (insert-feature "JSON - data exchange library"
                    (progn (require 'json) (fboundp 'json-serialize)))
    (insert-feature "GNUTLS - secure communication library" (gnutls-available-p))
    (insert-feature "pbm - image library" (image-type-available-p 'pbm))
    (insert-feature "xpm - image library" (image-type-available-p 'xpm))
    (insert-feature "bmp - image library" (image-type-available-p 'bmp))
    (insert-feature "gif - image library" (image-type-available-p 'gif))
    (insert-feature "png - image library" (image-type-available-p 'png))
    (insert-feature "xpm - image library" (image-type-available-p 'xpm))
    (insert-feature "jpeg - image library" (image-type-available-p 'jpeg))
    (insert-feature "tiff - image library" (image-type-available-p 'tiff))
    (insert-feature "svg - image library" (image-type-available-p 'svg))
    (insert-feature "native images" (image-type-available-p 'native-image))
    ;; what are native images?
    ;; jansson = json?
    ;; thread support?
    ;; modules
    ;; gmp <- gnutls
    (read-only-mode 1))


What do you think?

By the way, please tell me what are native images?

-- 
   Dieter
   
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: Corwin Brust <corwin@bru.st>,  Eli Zaretskii <eliz@gnu.org>,  Emacs
>  developers <emacs-devel@gnu.org>,  chad <yandros@gmail.com>,  Phillip Lord
>  <phillip.lord@russet.org.uk>
> Date: Wed, 09 Feb 2022 19:23:17 +0100
> 
> By the way, please tell me what are native images?

See w32-use-native-image-API.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Stefan Monnier]

> I took up the idea from Alan Third and completed some missing "standard"
> features (from nt/INSTALL.W64):
[...]
> What do you think?

I like it, thanks.


        Stefan

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

This is looking good!  I have a few suggestions but not all will
likely interest you.

On Wed, Feb 9, 2022 at 12:23 PM H. Dieter Wilhelm
<dieter@duenenhof-wilhelm.de> wrote:
>
> I took up the idea from Alan Third and completed some missing "standard"
> features (from nt/INSTALL.W64):
>
>   (defun insert-feature (description test)

I wonder if it would be worth separating the name from the rest of the
description?  This could make the output even prettier.

in a similar vein, I think tabulated list mode could be a good fit for
this.  For the sake of argument, let's suppose the tests are moved to
an var, something like:

(defcustom list-features-list
  '((name:"Zlib"
    description:"compression library"
    test: zlib-available-p)
    ;; ... and so on
  )
  "List of optional Emacs features.

This is used by `list-features', which see.")

At this point `list-features' only has to walk the last transforming
the function to results to become suitable for use as a
"tabulated-list-entries function".

As a bonus, the "testable features list" could enable users to
customize which features should usually be tested/listed, for example
to add non-standard features (as long as they have a test in mind for
them), and to remove things they don't typically care out about per in
their own uses

Even aside putting the tests into a var, I think the sortable
header-line approach taken by tabulated-list-mode could make
`list-features' nicely consistent with, e.g. `list-processes.

>
>     ;; jansson = json?

I think we only get jason-parse-string if jannson is available, so:
  (functionp 'json-parse-string)

>     ;; thread support?

Not sure on this one, but I suspect we might be able to use:
  (functionp 'make-thread)

>     ;; modules
>     ;; gmp <- gnutls

Pretty sure this is just
  (gnutls-available-p)

>
> What do you think?
>

*I* think `list-features' is going to be very nice to have.

Please & thank you!

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Robert Pluim]

>>>>> On Wed, 09 Feb 2022 19:23:17 +0100, "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> said:

    Dieter>     ;; harfbuzz
    Dieter>     (insert-feature "Harfbuzz - text shaping library "
    Dieter>                     (eq 'harfbuzz 
    Dieter>                         (car (frame-parameter nil
    Dieter>     'font-backend))))

If only. On GNU/Linux 'font-backend will return something like
'(ftcrhb x) or '(fthb x) or '(xfthb x). Although adding a
`harfbuzz-available-p' function could be done.

    Dieter>     ;; zlib
    Dieter>     (insert-feature "Zlib - compression library" (zlib-available-p))
    Dieter>     ;; libxml
    Dieter>     (insert-feature "xml2 - parser library" (libxml-available-p))
    Dieter>     ;; lcms lcms2-available-p
    Dieter>     (insert-feature "lcms2 - color translator library" (image-type-available-p 'pbm))
    Dieter>     (insert-feature "JSON - data exchange library"
    Dieter>                     (progn (require 'json) (fboundp
    Dieter>     'json-serialize)))

`json-available-p'

    Dieter>     (insert-feature "GNUTLS - secure communication library" (gnutls-available-p))
    Dieter>     (insert-feature "pbm - image library" (image-type-available-p 'pbm))
    Dieter>     (insert-feature "xpm - image library" (image-type-available-p 'xpm))
    Dieter>     (insert-feature "bmp - image library" (image-type-available-p 'bmp))
    Dieter>     (insert-feature "gif - image library" (image-type-available-p 'gif))
    Dieter>     (insert-feature "png - image library" (image-type-available-p 'png))
    Dieter>     (insert-feature "xpm - image library" (image-type-available-p 'xpm))
    Dieter>     (insert-feature "jpeg - image library" (image-type-available-p 'jpeg))
    Dieter>     (insert-feature "tiff - image library" (image-type-available-p 'tiff))
    Dieter>     (insert-feature "svg - image library" (image-type-available-p 'svg))
    Dieter>     (insert-feature "native images" (image-type-available-p 'native-image))
    Dieter>     ;; what are native images?

Itʼs using the platform native image APIs rather than libjpeg etc. I
think this is macOS/Windows specific?

    Dieter>     ;; jansson = json?

Yes.

    Dieter>     ;; thread support?

(featurep 'thread)

    Dieter>     ;; modules

(not (null module-file-suffix))

    Dieter>     ;; gmp <- gnutls
    Dieter>     (read-only-mode 1))


Robert
-- 

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Corwin Brust <corwin@bru.st> writes:

> On Sat, Jan 29, 2022 at 3:32 PM H. Dieter Wilhelm
> <dieter@duenenhof-wilhelm.de> wrote:
>>
>>
>>    libXpm-noX4.dll
>>
>> provides a nicer logo and icons.  (I didn't test much functionality
>> yet.)
>
> Thanks! I'll see if that could be a gap somewhere in the build
> process.  The windows release builds seem to have a pretty ugly logo
> on the splash.
>
>>
>> In /nt/README.W32 is a list of further dependencies provided in MSYS2
>> packages:
>>
>>     mingw-w64-x86_64-giflib
>>     mingw-w64-x86_64-gnutls
>>     mingw-w64-x86_64-libjpeg-turbo
>>     mingw-w64-x86_64-libpng
>>     mingw-w64-x86_64-librsvg
>>     mingw-w64-x86_64-libtiff
>>     mingw-w64-x86_64-libxml2
>>     mingw-w64-x86_64-xpm-nox
>>     mingw-w64-x86_64-lcms2
>>
>
> Here is the list from build-deps-zips.  I'm fairly sure at least
> harfbuzz and jansson should be added, but I strongly suspect the list
> in the script is a good one (at least, I like the emacs release
> packages Phillip has been creating).
>
> mingw-w64-x86_64-giflib
> mingw-w64-x86_64-gnutls
> mingw-w64-x86_64-harfbuzz
> mingw-w64-x86_64-jansson
> mingw-w64-x86_64-lcms2
> mingw-w64-x86_64-libjpeg-turbo
> mingw-w64-x86_64-libpng
> mingw-w64-x86_64-librsvg
> mingw-w64-x86_64-libtiff
> mingw-w64-x86_64-libxml2
> mingw-w64-x86_64-xpm-nox
>
> Removing the intersection of these gives me:
>
> mingw-w64-x86_64-harfbuzz
> mingw-w64-x86_64-jansson
>
> However, when I look a little further down the script at the DLLs that
> we'll bundle, I note that xpm-nox is missing from the latter list, so
> that probably confirms the change needed per comments above.
>
> I will test that I confirm back.
>
>> are above dependencies considered as the absolut minimal set for a state
>> of the art Emacs under Windows?
>
> Do you want to patch the documentation while I further mangle Phillips scripts?
>
> PS, not intending to side track the "simpler path" conversation you
> are having with Eli, here, but I do think we should carry what we
> learn back to the scripts that appear to do the job now.   In all
> events, I suspect jannsson and harbuzz should be documented in
> README.W32 in *some* fashion.


Just to confirm this will just be an omission on be behalf. IIRC,
jannson and harfbuzz are the last two additions to the
dependencies. lcms2, I think, but I remembered to patch the
documentation there.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Corwin Brust]

On Wed, Feb 2, 2022 at 3:58 PM Phillip Lord <phillip.lord@russet.org.uk> wrote:
>
> Just to confirm this will just be an omission on be behalf. IIRC,
> jannson and harfbuzz are the last two additions to the
> dependencies. lcms2, I think, but I remembered to patch the
> documentation there.

Thanks so much for confirming it was an omission (not anything
intentional or more complex :)

I'll add jannson and harbuzz to the dep-zip script and (eventually).

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Corwin Brust <corwin@bru.st> writes:
>
>> On Sat, Jan 29, 2022 at 3:32 PM H. Dieter Wilhelm
>> <dieter@duenenhof-wilhelm.de> wrote:
>>>    libXpm-noX4.dll
>>>
>>> provides a nicer logo and icons.  (I didn't test much functionality
>>> yet.)
>>
>> Thanks! I'll see if that could be a gap somewhere in the build
>> process.  The windows release builds seem to have a pretty ugly logo
>> on the splash.
>>>
>>> In /nt/README.W32 is a list of further dependencies provided in MSYS2
>>> packages:
>>>
>>>     mingw-w64-x86_64-giflib
>>>     mingw-w64-x86_64-gnutls
>>>     mingw-w64-x86_64-libjpeg-turbo
>>>     mingw-w64-x86_64-libpng
>>>     mingw-w64-x86_64-librsvg
>>>     mingw-w64-x86_64-libtiff
>>>     mingw-w64-x86_64-libxml2
>>>     mingw-w64-x86_64-xpm-nox
>>>     mingw-w64-x86_64-lcms2
>>
>> Here is the list from build-deps-zips.  I'm fairly sure at least
>> harfbuzz and jansson should be added, but I strongly suspect the list
>> in the script is a good one (at least, I like the emacs release
>> packages Phillip has been creating).

I like and appreciate them too. (I was just figuring out what's going
on.. ;-))

>> mingw-w64-x86_64-giflib
>> mingw-w64-x86_64-gnutls
>> mingw-w64-x86_64-harfbuzz
>> mingw-w64-x86_64-jansson
>> mingw-w64-x86_64-lcms2
>> mingw-w64-x86_64-libjpeg-turbo
>> mingw-w64-x86_64-libpng
>> mingw-w64-x86_64-librsvg
>> mingw-w64-x86_64-libtiff
>> mingw-w64-x86_64-libxml2
>> mingw-w64-x86_64-xpm-nox
>>
>> Removing the intersection of these gives me:
>>
>> mingw-w64-x86_64-harfbuzz
>> mingw-w64-x86_64-jansson
>>
>> However, when I look a little further down the script at the DLLs that
>> we'll bundle, I note that xpm-nox is missing from the latter list, so
>> that probably confirms the change needed per comments above.
>>
>> I will test that I confirm back.
>>
>>> are above dependencies considered as the absolut minimal set for a state
>>> of the art Emacs under Windows?
>>
>> Do you want to patch the documentation while I further mangle Phillips scripts?
>>
>> PS, not intending to side track the "simpler path" conversation you
>> are having with Eli, here, but I do think we should carry what we
>> learn back to the scripts that appear to do the job now.   In all
>> events, I suspect jannsson and harbuzz should be documented in
>> README.W32 in *some* fashion.
>
> Just to confirm this will just be an omission on be behalf. IIRC,
> jannson and harfbuzz are the last two additions to the
> dependencies. lcms2, I think, but I remembered to patch the
> documentation there.

Yes, I compared your list of dependencies with the "nice-to-have" list
suggested in nt/INSTALL.W64 and these are all there - in your
script. :-)

Thank you

        Dieter

We had an issue with a missing libgmp-10.dll on computers without MSYS2,
though. So Corwin added it as an additional dependency.  I read that
this library should not be strictily necessary for Emacs to run - there
should be some inbuild fallback method available.

I suspect on OUR MSYS systems the gmp library is present and then
configure is setting up our builds to expect this presence on all
machines..

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Stefan Monnier]

> We had an issue with a missing libgmp-10.dll on computers without MSYS2,
> though. So Corwin added it as an additional dependency.  I read that
> this library should not be strictily necessary for Emacs to run - there
> should be some inbuild fallback method available.

IIUC it's not necessary for Emacs, but it's necessary for libgnutls.


        Stefan

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> We had an issue with a missing libgmp-10.dll on computers without MSYS2,
>> though. So Corwin added it as an additional dependency.  I read that
>> this library should not be strictily necessary for Emacs to run - there
>> should be some inbuild fallback method available.
>
> IIUC it's not necessary for Emacs, but it's necessary for libgnutls.

Understood, thanks.  Corwin took care about this.

--
        Dieter

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de>
> Cc: corwin@bru.st,  phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Sat, 29 Jan 2022 22:32:46 +0100
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > 1. First, I build the package (in this case, Emacs) as usual:
> >   $ ./configure --prefix=...
> >   $ make
> > 2. Then I install it in a special place:
> >   $ make install --prefix=/some/directory
> > 3. Then to produce a zip file with all the installed files, I go to
> >    /some/directory and invoke 'zip':
> >   $ zip -9r ./bin ./include ./lib ./libexec ./share ./var
> 
> Why is the folder ./var necessary?
> 
> (var/ wasn't installed on my system (28.0.91).)

It was used at some point in the past.  If it doesn't exist in the
installed tree, it isn't needed.

> > (be sure to mention all the top-level directories).  Now, the tricky
> > part is between steps 2 and 3.  To make the zip file self-contained,
> > you need to populate /some/directory tree with the dependencies.  If
> > those are DLLs, simply copy all of them into /some/directory/bin/.  If
> > there are other dependencies, they should go
> 
> It seems that only
> 
>    libgmp-10.dll
> 
> is a prerequisite to get Emacs (28.0.91) running on a Windows system
> without MSYS and
> 
>    libXpm-noX4.dll
> 
> provides a nicer logo and icons.  (I didn't test much functionality
> yet.)
> 
> In /nt/README.W32 is a list of further dependencies provided in MSYS2
> packages:
> 
>     mingw-w64-x86_64-giflib
>     mingw-w64-x86_64-gnutls
>     mingw-w64-x86_64-libjpeg-turbo
>     mingw-w64-x86_64-libpng
>     mingw-w64-x86_64-librsvg
>     mingw-w64-x86_64-libtiff
>     mingw-w64-x86_64-libxml2
>     mingw-w64-x86_64-xpm-nox
>     mingw-w64-x86_64-lcms2
> 
> are above dependencies considered as the absolut minimal set for a state
> of the art Emacs under Windows? 

They are not the "absolute minimum", but they are nice to have, yes.
Assuming you built Emacs with all those features, of course.  But the
up-to-date list is in nt/INSTALL.W64, and it includes a few more
packages (notably, HarfBuzz).

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> The console returns an error:
>>> 
>>>    make: *** No rule to make target 'install-strip'. Stop
>>
>> The top-level Makefile.in has that target, so I wonder how did that
>> happen for you.
>
> OK, I need to cleanup the whole build tree..
>
>>> archive (for example libgmp-10.dll is missing).  I hope Corwin can bring
>>> some light into the build system.
>>
>> I hate to maybe take you back some steps, but why exactly do you need
>> those build-zips.sh etc. scripts?  If the problem is to produce a zip
>
> Because I thought that this is the OFFICIALL way of doing Windows
> builds, since this script is under admin/nt/dist-build!  But don't
> worry, I want to learn a robust and simple way of distributing
> Emacs. :-)


I used to produce multiple binaries: with deps, without multipled by
32bit/64bit. And, they had to be done by hand and in the right order
(once the deps version has been produced the install tree is no longer
clean), and also one at a time. As this process took many hours, scripts
were necessary, so I didn't have to stand over it.

Some of that has dropped away now, but in the mean time, I added the
NSIS installer to build-zips. That functionality exists nowhere else in
Emacs. Also the "build a snapshot with a dated name" functionality. So
that would have to go somewhere, if not in build-zips.sh.


>
> I see myself as a backup of Corwin: First I'd like to build a zip file
> with Emacs and its necessary dependencies.
>
> Later configured --with-native-compilation.
>
> And later building an installer out of it.
>
>> archive with the products of a build (and nothing else), then I can
>> propose a much simpler solution, which I use for all the ezwinports
>> binaries.  Interested?
>
> Yes please.  A step by step approach to building Emacs is probably more
> successful in the end.


I did it that way for a long time.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Hello Phil

Thank you for your messages, I'll pause and study them tomorrow.

Phillip Lord <phillip.lord@russet.org.uk> writes:
>> Yes please.  A step by step approach to building Emacs is probably more
>> successful in the end.

> I did it that way for a long time.

What I meant to say is that, at the moment, our builds are failing,
without me knowing what's going on.  Without much build experience it
might be better if I slowly learn to build things step by step - like
you did - and then coming back and using your scripts in a dignified
way.

    Dieter


-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>> And when we are at it, please tell me your recipe how to build a
>>> snapshot of the master branch.  I think my build tree is damaged, I
>>> don't manage to build a snapshot of emacs-29 with a date included..
>>>
>>> Shouldn't it be just:
>>>
>>> $ build-zips.sh -g
>>> $ build-zips.sh -s
>>
>> Yes, I think that should do it.
>>
>> Then, the I build the source like so.
>>
>> ./make-dist --snapshot --no-check --no-changelog
>> rename .tar _`date +%Y-%m-%d`.tar emacs*.tar.gz
>> mv emacs-*.tar.gz ~/emacs-upload
>>
>> I don't think I ever automated this bit.
>
> Uups, I thought this - or similar - would be done by below part of
> build-zips.sh?
>
>
>     make -j 4 $INSTALL_TARGET \
>          prefix=$HOME/emacs-build/install/emacs-$VERSION
>     cd $HOME/emacs-build/install/emacs-$VERSION
>     zip -r -9 emacs-$OF_VERSION-no-deps.zip *
>     mv emacs-$OF_VERSION-no-deps.zip $HOME/emacs-upload
>
>     if [ -z $SNAPSHOT ];
>     then
>         DEPS_FILE=$HOME/emacs-build/deps/emacs-$MAJOR_VERSION-deps.zip
>     else
>         ## Pick the most recent snapshot whatever that is
>         DEPS_FILE=`ls $HOME/emacs-build/deps/emacs-$MAJOR_VERSION-*-deps.zip | tail -n 1`
>     fi
>
>     echo [build] Using $DEPS_FILE
>     unzip -d bin $DEPS_FILE
>
>     zip -r -9 emacs-$OF_VERSION.zip *
>     mv emacs-$OF_VERSION.zip ~/emacs-upload

I was struggling to remember what was going on there, and then it hit
me!

This bit of code adds the dependencies. I have to ship the source code
for the dependencies also, but that is produced with build-deps-zips.py.
That doesn't change for an Emacs release cycle (I normally make a deps
bundle at the time of the first pre-test, then do not change it for a
major version).

Source for Emacs is different. I do *not* make source for an Emacs
release build, because I put the Windows binaries on ftp.gnu.org which
is also where the main Emacs release goes which is already a source
release; and as I don't build the binaries till after the source release
happens, this means I fulfil GPL that way.

For the snapshot, I have to release the source code along with the
binaries because no one builds snapshot tarballs (I mean what would be
the point).

This bit does that:

./make-dist --snapshot --no-check --no-changelog

The overly complicated command line was to stop configure being re-run
which then forces a full build when I build snapshots
incrementally. When I used to do them every month or so this was a big
advantage.

>
>> The model of build-zips.sh is probably a bit complicated and could be
>> simplified. The addition of snapshots and the ability to build from a
>> branch makes it much more so and the documentation isn't always well
>> done. I have been trying to remember why I called the variable
>> "OF_VERSION" so, or what "OF" stands for, and have failed!
>
> Not only above is confusing, but also why is
>
>     INSTALL_TARGET="install-strip"
>
> in the script?  (Back to pretest building) I've got an error for
>
> $ build-zips.sh -V28
>
> in the emacs-28 worktree, maybe, that is a wrong usage of the scrip?
> Anyway, "install-strip" seems not to be a valid target for make!?


Eli suggested it! I think install-strip follows the GNU standards. It
makes the release smaller and the binaries faster, IIRC.


>> Apologies for this.
>
> No reason to apologise, it was working for you.  Rather, thank you for
> your work. :-)


Yeah, but now I struggle to remember what some of it was for. I always
teach my students to comment for this reason and, yet, here I am. Do as
I say, I guess.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
>> Sorry, I meant the script admin/nt/dist-build/build-zips.sh
>
> I could build Gnu-Emacs-28.0.91 when I used the archived sources
>
>   $ ./configure --with-native-compilation
>
> The difference to the build-zips.sh seems to me some more options from
> the script (and that is build from the git tree):
>
>         echo [build] Configuring Emacs
>         $REPO_DIR/$BRANCH/configure \
>             --without-dbus \
>             --without-compress-install \
>             --with-native-compilation \
>             $CACHE \
>             CFLAGS="$CFLAGS"
>
> Could it be that these additional options are spoiling the build?
> (CACHE=-C , I think.)


--without-compress-install switches off the gzipping of various files,
such as the info files. It was necessary, if I remember correctly, when
I added it because Emacs uses external calls to gzip which wasn't
necessarily bundled.

I do not remember why I added --without-dbus.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Óscar Fuentes]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> --without-compress-install switches off the gzipping of various files,
> such as the info files. It was necessary, if I remember correctly, when
> I added it because Emacs uses external calls to gzip which wasn't
> necessarily bundled.

Isn't the built-in zip support used for uncompressing things (.el.gz and
.info.gz) when gzip is missing? IIRC that was recently added.

> I do not remember why I added --without-dbus.

Probably because the dbus libraries might be present on the system and
Emacs would pick up them, which is not a good thing.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> --without-compress-install switches off the gzipping of various files,
>> such as the info files. It was necessary, if I remember correctly, when
>> I added it because Emacs uses external calls to gzip which wasn't
>> necessarily bundled.
>
> Isn't the built-in zip support used for uncompressing things (.el.gz and
> .info.gz) when gzip is missing? IIRC that was recently added.

OK, I'll note this onto the todo list.

Thanks for the hint Óscar

       Dieter
-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Óscar Fuentes]

"H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:

>> Isn't the built-in zip support used for uncompressing things (.el.gz and
>> .info.gz) when gzip is missing? IIRC that was recently added.
>
> OK, I'll note this onto the todo list.

That was a genuine question. I'm not sure about the status of that
feature.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Óscar Fuentes <ofv@wanadoo.es> writes:

> "H. Dieter Wilhelm" <dieter@duenenhof-wilhelm.de> writes:
>
>>> Isn't the built-in zip support used for uncompressing things (.el.gz and
>>> .info.gz) when gzip is missing? IIRC that was recently added.
>>
>> OK, I'll note this onto the todo list.
>
> That was a genuine question. I'm not sure about the status of that
> feature.

No problem, I meant to check if this feature is working or not. :-)

   Dieter

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> --without-compress-install switches off the gzipping of various files,
>> such as the info files. It was necessary, if I remember correctly, when
>> I added it because Emacs uses external calls to gzip which wasn't
>> necessarily bundled.
>
> Isn't the built-in zip support used for uncompressing things (.el.gz and
> .info.gz) when gzip is missing? IIRC that was recently added.


I haven't tried! If it does do that now then this switch needs
removing. It didn't when I added it many years back.


>> I do not remember why I added --without-dbus.
>
> Probably because the dbus libraries might be present on the system and
> Emacs would pick up them, which is not a good thing.

Glad I go something right!

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Robert Pluim]

>>>>> On Mon, 17 Jan 2022 20:21:53 +0000, Phillip Lord <phillip.lord@russet.org.uk> said:

    Phillip> Óscar Fuentes <ofv@wanadoo.es> writes:
    >> Phillip Lord <phillip.lord@russet.org.uk> writes:
    >> 
    >>> --without-compress-install switches off the gzipping of various files,
    >>> such as the info files. It was necessary, if I remember correctly, when
    >>> I added it because Emacs uses external calls to gzip which wasn't
    >>> necessarily bundled.
    >> 
    >> Isn't the built-in zip support used for uncompressing things (.el.gz and
    >> .info.gz) when gzip is missing? IIRC that was recently added.


    Phillip> I haven't tried! If it does do that now then this switch needs
    Phillip> removing. It didn't when I added it many years back.

I donʼt know if the builtin zip support works here, but itʼs moot:
zlib support is optional, so you canʼt rely on it always being
available (unless youʼre going to ensure that here).

Robert
-- 

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: Robert Pluim <rpluim@gmail.com>
> Gmane-Reply-To-List: yes
> Date: Tue, 18 Jan 2022 10:03:55 +0100
> Cc: Óscar Fuentes <ofv@wanadoo.es>, emacs-devel@gnu.org
> 
> >>>>> On Mon, 17 Jan 2022 20:21:53 +0000, Phillip Lord <phillip.lord@russet.org.uk> said:
> 
>     Phillip> Óscar Fuentes <ofv@wanadoo.es> writes:
>     >> Phillip Lord <phillip.lord@russet.org.uk> writes:
>     >> 
>     >>> --without-compress-install switches off the gzipping of various files,
>     >>> such as the info files. It was necessary, if I remember correctly, when
>     >>> I added it because Emacs uses external calls to gzip which wasn't
>     >>> necessarily bundled.
>     >> 
>     >> Isn't the built-in zip support used for uncompressing things (.el.gz and
>     >> .info.gz) when gzip is missing? IIRC that was recently added.
> 
> 
>     Phillip> I haven't tried! If it does do that now then this switch needs
>     Phillip> removing. It didn't when I added it many years back.
> 
> I donʼt know if the builtin zip support works here, but itʼs moot:
> zlib support is optional, so you canʼt rely on it always being
> available (unless youʼre going to ensure that here).

For Emacs with native compilation enabled, the zlib support takes
another role: if the *.el files are installed compressed, then Emacs
_must_ have zlib support, or else it will be unable to use the *.eln
files, because it must verify the *.el source files correspond to the
*.eln files.  So if you build an Emacs with native compilation, but
without zlib, do NOT compress the *.el file at "make install" time.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Óscar Fuentes]

Robert Pluim <rpluim@gmail.com> writes:

>     >> Isn't the built-in zip support used for uncompressing things (.el.gz and
>     >> .info.gz) when gzip is missing? IIRC that was recently added.
>
>
>     Phillip> I haven't tried! If it does do that now then this switch needs
>     Phillip> removing. It didn't when I added it many years back.
>
> I donʼt know if the builtin zip support works here, but itʼs moot:
> zlib support is optional, so you canʼt rely on it always being
> available (unless youʼre going to ensure that here).

We are talking about a binary distribution. Its creator can ensure that
the required dependencies are included on the binary package.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Robert Pluim <rpluim@gmail.com> writes:

>>>>>> On Mon, 17 Jan 2022 20:21:53 +0000, Phillip Lord <phillip.lord@russet.org.uk> said:
>
>     Phillip> Óscar Fuentes <ofv@wanadoo.es> writes:
>     >> Phillip Lord <phillip.lord@russet.org.uk> writes:
>     >> 
>     >>> --without-compress-install switches off the gzipping of various files,
>     >>> such as the info files. It was necessary, if I remember correctly, when
>     >>> I added it because Emacs uses external calls to gzip which wasn't
>     >>> necessarily bundled.
>     >> 
>     >> Isn't the built-in zip support used for uncompressing things (.el.gz and
>     >> .info.gz) when gzip is missing? IIRC that was recently added.
>
>
>     Phillip> I haven't tried! If it does do that now then this switch needs
>     Phillip> removing. It didn't when I added it many years back.
>
> I donʼt know if the builtin zip support works here, but itʼs moot:
> zlib support is optional, so you canʼt rely on it always being
> available (unless youʼre going to ensure that here).

From the point of the windows package, it is under the control of the
packager. If builtin zlib support works on the compressed installed
files, when it can be guaranteed to be packaged.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[phillip.lord]

On 2022-01-15 19:06, H. Dieter Wilhelm wrote:
> Corwin Brust <corwin@bru.st> writes:
>>  > phillip.lord@russet.org.uk writes:
>>  The performance that I got on Windows builds when I did do it was 
>> highly
>>  variable (between half an hour and three), but I think this was 
>> mostly
>>  the shared CPU infrastructure I was using. I never worked out how to 
>> get
>>  native comp working in a way that I could package it for windows, so 
>> the
>>  builds were all without it.
>> 
>> This is my present situation also.  I can build with native comp but
>> not (yet) found a way to package that version.
> 
> I don't understand yet the packaging requirements, is it not possible 
> to
> copy additionally the native-lisp/ folder to the package?
> 
> So far I could compile (locally) the current pretest 28.0.91 version
> --with-native-compilation.
> 
> Then I tried to build (with build.zips.sh) a snapshot of the
> masterbranch --with-native-compilation on the same system.  And now
> ./configure is complaining:
> 
>   configure: error: Elisp native compiler was requested, but libgccjit
> was not found.
> 
> Does Emacs-29 has different requirements?


The problem is that native comp needs libgccjit at compile time but also 
at
runtime. To make an executable install package like we have at the 
moment
essentially means packaging libgccjit. Which means gcc and half of the 
msys2
toolchain.

We were there a long time back when Emacs used to be packaged with lots 
of other
stuff, including a full python installation. It didn't make sense then 
and it's
not a great thing to go back to.

So, I think, Emacs releases for windows will be without native-comp. 
Anyone who
wants it will have to install it over an msys2 installation, or with 
chocolaty
or something equivalent.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

phillip.lord@russet.org.uk writes:
> The problem is that native comp needs libgccjit at compile time but
> also at runtime. To make an executable install package like we have at
> the moment essentially means packaging libgccjit. Which means gcc and
> half of the msys2 toolchain.

As far as I understood Eli it should be possible to build Emacs on
Windows --with-native-compilation and distribute it even on systems
without libgccjit.  Only Emacs internal lisp files will then be natively
compiled and if, for other lisp packages, there are no .eln files
available Emacs will just load the respective .elc files.

Do you think I'm mistaken?

    Dieter

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> Date: Sat, 15 Jan 2022 22:14:56 +0000
> From: phillip.lord@russet.org.uk
> Cc: Corwin Brust <corwin@bru.st>, Emacs developers <emacs-devel@gnu.org>
> 
> The problem is that native comp needs libgccjit at compile time but
> also at runtime. To make an executable install package like we have
> at the moment essentially means packaging libgccjit. Which means gcc
> and half of the msys2 toolchain.

We made a point of implementing native-compilation on MS-Windows in a
way that doesn't require such extreme measures.  Emacs built with
native-compilation support should be able to run on a Windows system
without GCC and Binutils installed, it just won't be able to
natively-compile any new or modified Lisp files.  It will use the
*.elc files instead.

Or at least that is the theory.

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Andrea Corallo]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 15 Jan 2022 22:14:56 +0000
>> From: phillip.lord@russet.org.uk
>> Cc: Corwin Brust <corwin@bru.st>, Emacs developers <emacs-devel@gnu.org>
>> 
>> The problem is that native comp needs libgccjit at compile time but
>> also at runtime. To make an executable install package like we have
>> at the moment essentially means packaging libgccjit. Which means gcc
>> and half of the msys2 toolchain.
>
> We made a point of implementing native-compilation on MS-Windows in a
> way that doesn't require such extreme measures.  Emacs built with
> native-compilation support should be able to run on a Windows system
> without GCC and Binutils installed, it just won't be able to
> natively-compile any new or modified Lisp files.  It will use the
> *.elc files instead.
>
> Or at least that is the theory.

Agree, before we rely on this behavior would be nice if someone could
test it tho.

Best Regards

  Andrea

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> Date: Sat, 15 Jan 2022 22:14:56 +0000
>> From: phillip.lord@russet.org.uk
>> Cc: Corwin Brust <corwin@bru.st>, Emacs developers <emacs-devel@gnu.org>
>> 
>> The problem is that native comp needs libgccjit at compile time but
>> also at runtime. To make an executable install package like we have
>> at the moment essentially means packaging libgccjit. Which means gcc
>> and half of the msys2 toolchain.
>
> We made a point of implementing native-compilation on MS-Windows in a
> way that doesn't require such extreme measures.  Emacs built with
> native-compilation support should be able to run on a Windows system
> without GCC and Binutils installed, it just won't be able to
> natively-compile any new or modified Lisp files.  It will use the
> *.elc files instead.
>
> Or at least that is the theory.


Ah, okay. I remember asking about this some time back when I got the
impression this would not work. It's a little bit late in the release
process to be testing the theory now. The last snapshot is a year old,
and that is not native comp.

Phil

Re: Native compilation on Windows, was Re: Bootstrap Compilation Speed

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: dieter@duenenhof-wilhelm.de,  corwin@bru.st,  emacs-devel@gnu.org
> Date: Mon, 17 Jan 2022 09:56:44 +0000
> 
> I remember asking about this some time back when I got the
> impression this would not work.

AFAIR, it did work in my (very limited) testing.

> It's a little bit late in the release process to be testing the
> theory now. The last snapshot is a year old, and that is not native
> comp.

I don't see how we could do better, given the (un)availability of
snapshots and pretest binaries for Windows.  Certainly, delaying Emacs
28.1 release until this is tested well enough would be a worse
alternative.

Re: Bootstrap Compilation Speed

[H. Dieter Wilhelm]

Hello Corwin,

some progress on my side.. :-)

Corwin Brust <corwin@bru.st> writes:
> On Sat, Jan 15, 2022, 05:32 <phillip.lord@russet.org.uk> wrote:
>  On 2022-01-15 10:29, H. Dieter Wilhelm wrote:
>  > Are you doing these builds for Windows?

>  No, I am afraid not.
>
>  The performance that I got on Windows builds when I did do it was highly 
>  variable (between half an hour and three), but I think this was mostly 
>  the shared CPU infrastructure I was using. I never worked out how to get 
>  native comp working in a way that I could package it for windows, so the 
>  builds were all without it.
> ...
> Corwin:

> This is my present situation also.  I can build with native comp but
> not (yet) found a way to package that version.

When I'm removing all the configure options and include
--with-native-compilation in build-zips.sh then - at last - the build
process is creating .eln files...

Wouldn't it be sufficient to copy the native-lisp/ folder into the
distro tree?

     Dieter

-- 
Best wishes
H. Dieter Wilhelm
Zwingenberg, Germany