[feature/dll-only-windows] A new windows build, comments wanted

[feature/dll-only-windows] A new windows build, comments wanted

[Phillip Lord]

In preparation for making a native-comp build of Emacs on Windows, I
have revisited the windows packaging process.

The current version figures out dependencies using msys2 and as a result
has an awful lot of dependencies. Even with some efforts to reduce this
size (e.g. python isn't included any more) it's still pretty large.

So I have tried a new technique which is now on
feature/dll-only-windows. Essentially, I just name all of the DLLs that
Emacs uses directly and then figure out any dependencies of these using
`ntldd`. I've tried `ntldd` directly on emacs.exe which does not
work. The practical upshot of this is that the dependencies file is just
a pile of DLLs. The installer version of Emacs (which is the smallest
because it is most compressed) is now only 45Mb (vs 63) in size, with
zip being 130Mb (vs 160Mb).

The source download still using the same technique as before; i.e. the
source bundle will produce a lot more than just the DLLs included.


Some questions for now:

1) Does this seem reasonable? I can some upload some binaries to alpha
if anyone is prepared to try them.

2) harfbuzz is currently not working on i686. Before I fix this, I think
it is worth asking whether I still need to produce a i686 binary. It
doubles the build time, obviously.

3) Currently the "no-deps" version actually includes libXpm. Emacs
starts without it, but looks ugly. I would like to no longer special
case libXpm and just make the "no-deps" download really include no
deps. I think this is reasonable, because this download is now, really
special purpose and "with-deps" is the default.


Some questions for when I add a native-comp build

4) Currently, native-comp is an "--with-nativecomp" option even on the
native-comp branch. It is likely to be merged this way to master?
Currently, I build the Windows distribution of Emacs with all the
default options. I can make an alternative release "--with-nativecomp"
but needs some effort; it also does not address the question of how I
should build Emacs for Windows when a full release of Emacs-28
happens. If, the full release of Emacs-28 will be native-comp, I'd
rather start building snapshots with it as soon as it is merged.

5) Why is it "--with-nativecomp", shouldn't it be "--with-native-comp".

Thoughts welcome.

Phil

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Date: Sat, 09 Jan 2021 19:57:00 +0000
> 
> So I have tried a new technique which is now on
> feature/dll-only-windows. Essentially, I just name all of the DLLs that
> Emacs uses directly and then figure out any dependencies of these using
> `ntldd`. I've tried `ntldd` directly on emacs.exe which does not
> work.

Of course, it won't: we load all the DLLs dynamically at run time, we
don't link against their import libraries.  The only exceptions I know
of are (1) GMP, and (2) libgccjit (in the native-comp build).

> The practical upshot of this is that the dependencies file is just
> a pile of DLLs.

Did you verify that the DLLs include all of _their_ dependencies?  If
yes, how did you do that?

> 2) harfbuzz is currently not working on i686.

Is that a bug in the MinGW64 HarfBuzz port, or is that a bug in Emacs?
I'm using a 32-bit Emacs build with HarfBuzz all the time, but it's
HarfBuzz I built myself (it's available from the ezwinports site).

> Before I fix this, I think it is worth asking whether I still need
> to produce a i686 binary.

I cannot tell you what to do, but it would be nice to have binaries
that can be run on older Windows versions.  So if the build supports
XP and older Windows, keeping the 32-bit build would be an advantage.

> 3) Currently the "no-deps" version actually includes libXpm. Emacs
> starts without it, but looks ugly. I would like to no longer special
> case libXpm and just make the "no-deps" download really include no
> deps. I think this is reasonable, because this download is now, really
> special purpose and "with-deps" is the default.

If we believe no one will want the no-deps download, why have it at
all?  If we think someone will want it, I don't think they should be
punished by having BW icons on the tool bar.

> 4) Currently, native-comp is an "--with-nativecomp" option even on the
> native-comp branch. It is likely to be merged this way to master?

Yes, I think so.

> Currently, I build the Windows distribution of Emacs with all the
> default options.

Which non-default options of practical importance does that leave out?

> I can make an alternative release "--with-nativecomp" but needs some
> effort; it also does not address the question of how I should build
> Emacs for Windows when a full release of Emacs-28 happens. If, the
> full release of Emacs-28 will be native-comp, I'd rather start
> building snapshots with it as soon as it is merged.

I think you should build --with-nativecomp.  People can always
uninstall libgccjit or rename it if they don't want to use native
compilation.

> 5) Why is it "--with-nativecomp", shouldn't it be "--with-native-comp".

Probably, but we didn't yet get to splitting such thin hair in that
branch ;-)

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Alan Third]

On Sat, Jan 09, 2021 at 07:57:00PM +0000, Phillip Lord wrote:
> 
> 3) Currently the "no-deps" version actually includes libXpm. Emacs
> starts without it, but looks ugly. I would like to no longer special
> case libXpm and just make the "no-deps" download really include no
> deps. I think this is reasonable, because this download is now, really
> special purpose and "with-deps" is the default.

Do you even really need libXpm? We manage without it on NS and Cairo
builds by using the built-in functions at src/image.c:4671. Perhaps
that could be extended to Windows as well?
-- 
Alan Third

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Date: Sat, 09 Jan 2021 19:57:00 +0000
>> 
>> So I have tried a new technique which is now on
>> feature/dll-only-windows. Essentially, I just name all of the DLLs that
>> Emacs uses directly and then figure out any dependencies of these using
>> `ntldd`. I've tried `ntldd` directly on emacs.exe which does not
>> work.
>
> Of course, it won't: we load all the DLLs dynamically at run time, we
> don't link against their import libraries.  The only exceptions I know
> of are (1) GMP, and (2) libgccjit (in the native-comp build).

Indeed.

>> The practical upshot of this is that the dependencies file is just
>> a pile of DLLs.
>
> Did you verify that the DLLs include all of _their_ dependencies?  If
> yes, how did you do that?

It's totally dependent on ntldd being correct, of course. It comes with
a --recursive option which I used rather than reimplementing it
myself. As ever, the only tests I run on the windows package for Emacs
are to unpack, start and then run "w32-feature.el" from etc. These all
pass.



>> 2) harfbuzz is currently not working on i686.
>
> Is that a bug in the MinGW64 HarfBuzz port, or is that a bug in Emacs?
> I'm using a 32-bit Emacs build with HarfBuzz all the time, but it's
> HarfBuzz I built myself (it's available from the ezwinports site).

I don't know, I haven't debugged it yet:-)

>> Before I fix this, I think it is worth asking whether I still need
>> to produce a i686 binary.
>
> I cannot tell you what to do, but it would be nice to have binaries
> that can be run on older Windows versions.  So if the build supports
> XP and older Windows, keeping the 32-bit build would be an advantage.

I know that you use a very old windows, but I doubt you use my packages!
I have no other data of how many people this would impact, but my guess
would be not many, so I am asking for opinions.

One solution could be to stop making the 32-bit build and see if anyone
complains. It could be restored, either before or after the Emacs-28
release.


>> 3) Currently the "no-deps" version actually includes libXpm. Emacs
>> starts without it, but looks ugly. I would like to no longer special
>> case libXpm and just make the "no-deps" download really include no
>> deps. I think this is reasonable, because this download is now, really
>> special purpose and "with-deps" is the default.
>
> If we believe no one will want the no-deps download, why have it at
> all?  If we think someone will want it, I don't think they should be
> punished by having BW icons on the tool bar.

Indeed. My guess is that currently if people use -no-deps they do so
because they want the smaller download. This change, of course, we
reduce the difference and regardless the installer version is smaller
still (by a half).

Given all that, the only real audience for the -no-deps version would be
those who have a mingw64 installation already. They won't get BW icons
(assuming that libXpm is installed). Of course, that audience is likely
to be technically skilled and they could just delete the DLLs in the
"with-deps" download.

On balance, therefore, I would say we don't need the -no-deps version
and it would clean up the download site and reduce confusion.


>> 4) Currently, native-comp is an "--with-nativecomp" option even on the
>> native-comp branch. It is likely to be merged this way to master?
>
> Yes, I think so.

Are you worried about stability or newness? I guess it will become the
default at some point. It seems to have few disadvantages, other than
hammering the CPU at bit at initial start up.


>
>> Currently, I build the Windows distribution of Emacs with all the
>> default options.
>
> Which non-default options of practical importance does that leave out?

Currently, none. If --with-nativecomp is not default, then, I think it
would become the example.

>> I can make an alternative release "--with-nativecomp" but needs some
>> effort; it also does not address the question of how I should build
>> Emacs for Windows when a full release of Emacs-28 happens. If, the
>> full release of Emacs-28 will be native-comp, I'd rather start
>> building snapshots with it as soon as it is merged.
>
> I think you should build --with-nativecomp.  People can always
> uninstall libgccjit or rename it if they don't want to use native
> compilation.

Okay. I guess "make NATIVE_FULL_AOT=1" would be the thing as well,
depending on how long it takes.


>> 5) Why is it "--with-nativecomp", shouldn't it be "--with-native-comp".
>
> Probably, but we didn't yet get to splitting such thin hair in that
> branch ;-)

Yeah, can't help it. Splitting hairs is my job.

Phil

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Phillip Lord]

Alan Third <alan@idiocy.org> writes:

> On Sat, Jan 09, 2021 at 07:57:00PM +0000, Phillip Lord wrote:
>> 
>> 3) Currently the "no-deps" version actually includes libXpm. Emacs
>> starts without it, but looks ugly. I would like to no longer special
>> case libXpm and just make the "no-deps" download really include no
>> deps. I think this is reasonable, because this download is now, really
>> special purpose and "with-deps" is the default.
>
> Do you even really need libXpm? We manage without it on NS and Cairo
> builds by using the built-in functions at src/image.c:4671. Perhaps
> that could be extended to Windows as well?

I don't think we do need it in the version with deps. AFAIU, it's just
there as the smallest dependency that can give colour icons.

Phil

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> Before I fix this, I think it is worth asking whether I still need
>> to produce a i686 binary.
>
> I cannot tell you what to do, but it would be nice to have binaries
> that can be run on older Windows versions.  So if the build supports
> XP and older Windows, keeping the 32-bit build would be an advantage.

MSYS2 MinGW-w64 binaries do not support XP, they officially decided this
some years ago and now some runtime dlls use APIs not present on XP.

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Andrea Corallo via Emacs development discussions.]

Eli Zaretskii <eliz@gnu.org> writes:

[...]

> I think you should build --with-nativecomp.  People can always
> uninstall libgccjit or rename it if they don't want to use native
> compilation.

ATM if using --with-nativecomp libgccjit is not present (or non
functional) we stop at configure time with an error.  I guess that's one
of the hairs we'll have to split then :)

  Andrea

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Alan Third]

On Sat, Jan 09, 2021 at 09:33:56PM +0000, Phillip Lord wrote:
> Alan Third <alan@idiocy.org> writes:
> 
> > Do you even really need libXpm? We manage without it on NS and Cairo
> > builds by using the built-in functions at src/image.c:4671. Perhaps
> > that could be extended to Windows as well?
> 
> I don't think we do need it in the version with deps. AFAIU, it's just
> there as the smallest dependency that can give colour icons.

Yes, my point is we have colour icons with no deps at all by using
functions already in image.c.

-- 
Alan Third

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Eli Zaretskii]

> Date: Sat, 9 Jan 2021 20:47:41 +0000
> From: Alan Third <alan@idiocy.org>
> Cc: emacs-devel@gnu.org
> 
> On Sat, Jan 09, 2021 at 07:57:00PM +0000, Phillip Lord wrote:
> > 
> > 3) Currently the "no-deps" version actually includes libXpm. Emacs
> > starts without it, but looks ugly. I would like to no longer special
> > case libXpm and just make the "no-deps" download really include no
> > deps. I think this is reasonable, because this download is now, really
> > special purpose and "with-deps" is the default.
> 
> Do you even really need libXpm? We manage without it on NS and Cairo
> builds by using the built-in functions at src/image.c:4671. Perhaps
> that could be extended to Windows as well?

Emacs supports native image display on Windows, but that is still
experimental.  In time, we could indeed try to get rid of the libXpm
dependency.

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Eli Zaretskii]

> From: Andrea Corallo <akrl@sdf.org>
> Cc: Phillip Lord <phillip.lord@russet.org.uk>, emacs-devel@gnu.org
> Date: Sat, 09 Jan 2021 21:51:16 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> [...]
> 
> > I think you should build --with-nativecomp.  People can always
> > uninstall libgccjit or rename it if they don't want to use native
> > compilation.
> 
> ATM if using --with-nativecomp libgccjit is not present (or non
> functional) we stop at configure time with an error.

The situation relevant here is that libgccjit is present at build
time, but not at ruin time (on another system).

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Arash Esbati]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>> Did you verify that the DLLs include all of _their_ dependencies?  If
>> yes, how did you do that?
>
> It's totally dependent on ntldd being correct, of course. It comes with
> a --recursive option which I used rather than reimplementing it
> myself.

cygcheck.exe has this capability as well.  Try

    cygcheck libjansson-4.dll

in a mingw64 console.  Maybe you can use this to double check the
dependencies reported by ntldd.

Best, Arash

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Phillip Lord]

Andrea Corallo <akrl@sdf.org> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> [...]
>
>> I think you should build --with-nativecomp.  People can always
>> uninstall libgccjit or rename it if they don't want to use native
>> compilation.
>
> ATM if using --with-nativecomp libgccjit is not present (or non
> functional) we stop at configure time with an error.  I guess that's one
> of the hairs we'll have to split then :)


I think there are two hairs here. Having an emacs compiled with
nativecomp behave cleanly if libgccjit is not available at runtime. And
having a runtime mechanism for switching nativecomp of totally even if
it is compiled in.

Phil

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Phillip Lord]

My test for GMP is also currently failing. It is this:

(ert-deftest feature-gmp ()
  (should
   (string-match-p "GMP" system-configuration-features)))

This test also fails on Emacs-28 master on my Ubuntu machine, as well as
on my Windows package, although all of them report that libgmp support
is available at configure time. And the test succeeds on Emacs-27 on the
same machines. So it looks like the test is wrong, rather libgmp support
broken.

Any thoughts on why GMP is no longer in system-configuration-features
for Emacs-28. Is there a better test that I can use? Can we add
`gmp-available-p' or equivalent?

Phil

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Phillip Lord]

Arash Esbati <arash@gnu.org> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>>> Did you verify that the DLLs include all of _their_ dependencies?  If
>>> yes, how did you do that?
>>
>> It's totally dependent on ntldd being correct, of course. It comes with
>> a --recursive option which I used rather than reimplementing it
>> myself.
>
> cygcheck.exe has this capability as well.  Try
>
>     cygcheck libjansson-4.dll
>
> in a mingw64 console.  Maybe you can use this to double check the
> dependencies reported by ntldd.


Thanks, I will have a poke.

Phil

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Phillip Lord]

I have pushed some binaries up to alpha in case anyone wants to give
them a trial.

Directory:

https://alpha.gnu.org/gnu/emacs/pretest/windows/emacs-28/

Two download links: as zip or installer version.


https://alpha.gnu.org/gnu/emacs/pretest/windows/emacs-28/emacs-28.0.50-snapshot-feature_dll-only-windows-2021-01-09-x86_64.zip
https://alpha.gnu.org/gnu/emacs/pretest/windows/emacs-28/emacs-28.0.50-snapshot-feature_dll-only-windows-2021-01-09-x86_64-installer.exe

Phil

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: emacs-devel@gnu.org
> Date: Sat, 09 Jan 2021 21:31:45 +0000
> 
> > Did you verify that the DLLs include all of _their_ dependencies?  If
> > yes, how did you do that?
> 
> It's totally dependent on ntldd being correct, of course. It comes with
> a --recursive option which I used rather than reimplementing it
> myself.

You can use objdump:

  objdump -p *.exe *.dll | fgrep "DLL Name:" | gawk " {print $3, $4, $5}" | sort -u

Keep adding DLLs and running the above command until all the DLLs it
shows are in your staging directory.

> >> 4) Currently, native-comp is an "--with-nativecomp" option even on the
> >> native-comp branch. It is likely to be merged this way to master?
> >
> > Yes, I think so.
> 
> Are you worried about stability or newness? I guess it will become the
> default at some point. It seems to have few disadvantages, other than
> hammering the CPU at bit at initial start up.

Usually, such radical features start as optional, but maybe we will
decide otherwise.

Why does it matter?

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sat, 09 Jan 2021 22:36:36 +0100
> 
> > I cannot tell you what to do, but it would be nice to have binaries
> > that can be run on older Windows versions.  So if the build supports
> > XP and older Windows, keeping the 32-bit build would be an advantage.
> 
> MSYS2 MinGW-w64 binaries do not support XP, they officially decided this
> some years ago and now some runtime dlls use APIs not present on XP.

That's what I knew.  So in that case, I see no reason for us to
produce official 32-bit binaries, as the chances for them being used
are likely nil.

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Date: Sun, 10 Jan 2021 15:14:03 +0000
> 
> Any thoughts on why GMP is no longer in system-configuration-features
> for Emacs-28.

It's a bug, please report it.

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> >> 4) Currently, native-comp is an "--with-nativecomp" option even on the
>> >> native-comp branch. It is likely to be merged this way to master?
>> >
>> > Yes, I think so.
>> 
>> Are you worried about stability or newness? I guess it will become the
>> default at some point. It seems to have few disadvantages, other than
>> hammering the CPU at bit at initial start up.
>
> Usually, such radical features start as optional, but maybe we will
> decide otherwise.
>
> Why does it matter?

I'd like to keep the Windows package using the default options as far as
possible, mostly because I am lazy and because it means that the windows
package updates when ever the defaults do.

Currently, we have

--without-dbus --without-compress-install

The latter is so the documentation works without gzip. The latter is
because we don't normally have dbus available. I am not sure why either
of these needed (i.e. why they are not the default on windows).

Adding --with-nativecomp isn't massively problematic, but would need
removing when and if it becomes default. Not a big problem, the less
difference between the windows package and the defaults, the happier I
would be.

Phil

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Óscar Fuentes <ofv@wanadoo.es>
>> Date: Sat, 09 Jan 2021 22:36:36 +0100
>> 
>> > I cannot tell you what to do, but it would be nice to have binaries
>> > that can be run on older Windows versions.  So if the build supports
>> > XP and older Windows, keeping the 32-bit build would be an advantage.
>> 
>> MSYS2 MinGW-w64 binaries do not support XP, they officially decided this
>> some years ago and now some runtime dlls use APIs not present on XP.
>
> That's what I knew.  So in that case, I see no reason for us to
> produce official 32-bit binaries, as the chances for them being used
> are likely nil.


Then I shall disable the code that produces it for Emacs-28. I will
leave the code and the "-x86_64-" names in place, I think, just in case
we get any complaints from Windows 7/10 32-bit users. Then, that can go
for Emacs-29, so binaries will become simply "emacs-29.1.zip" or
"emacs-29.1-installer.exe".

Assuming that this smaller DLL only distribution tests once I start
producing snapshots, I will also remove the "-no-deps" zip from
Emacs-28. At this point, I no longer need the independent copies of the
dependencies, so they can go as well. That will reduce the total number
of files for releases from 8 binary + 1 source (for the deps), to 2
binary + 1 source.

Phil

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Date: Sun, 10 Jan 2021 18:23:06 +0000
> Cc: emacs-devel@gnu.org
> 
> I'd like to keep the Windows package using the default options as far as
> possible, mostly because I am lazy and because it means that the windows
> package updates when ever the defaults do.

It's your call.  I just envision many users to expect native-comp
support and asking why isn't it there.

> Currently, we have
> 
> --without-dbus --without-compress-install
> 
> The latter is so the documentation works without gzip. The latter is
> because we don't normally have dbus available. I am not sure why either
> of these needed (i.e. why they are not the default on windows).

The gzip thing is again your call (having gzip in the package is no
big deal, IMO).  Dbus is not really useful on MS-Windows, so IMO it
makes no sense building with it.

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Andrea Corallo via Emacs development discussions.]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> [...]
>>
>>> I think you should build --with-nativecomp.  People can always
>>> uninstall libgccjit or rename it if they don't want to use native
>>> compilation.
>>
>> ATM if using --with-nativecomp libgccjit is not present (or non
>> functional) we stop at configure time with an error.  I guess that's one
>> of the hairs we'll have to split then :)
>
>
> I think there are two hairs here. Having an emacs compiled with
> nativecomp behave cleanly if libgccjit is not available at runtime.

Right, at this stage this should be easy to implement (on Windows).

> And having a runtime mechanism for switching nativecomp of totally
> even if it is compiled in.

We should define "switching nativecomp of" and the triggering mechanism,
this might be already implmentented.

  Andrea

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> I'd like to keep the Windows package using the default options as far as
>> possible, mostly because I am lazy and because it means that the windows
>> package updates when ever the defaults do.
>
> It's your call.  I just envision many users to expect native-comp
> support and asking why isn't it there.

I agree. In fact, some popular packages already recommend using
native-comp.

>> Currently, we have
>> 
>> --without-dbus --without-compress-install
>> 
>> The latter is so the documentation works without gzip. The latter is
>> because we don't normally have dbus available. I am not sure why either
>> of these needed (i.e. why they are not the default on windows).
>
> The gzip thing is again your call (having gzip in the package is no
> big deal, IMO).  Dbus is not really useful on MS-Windows, so IMO it
> makes no sense building with it.

I think that Phillip's problem with that is that gzip does not exist as
a MinGW package in the MSYS2 ecosystem.

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Sun, 10 Jan 2021 20:20:22 +0100
> 
> I think that Phillip's problem with that is that gzip does not exist as
> a MinGW package in the MSYS2 ecosystem.

I'm surprised, but then it's very easy to build it.

`gzip` dependency (was: [feature/dll-only-windows] A new windows build, comments wanted)

[Stefan Monnier]

> The gzip thing is again your call (having gzip in the package is no
> big deal, IMO).

BTW, we don't really need a `gzip` executable as long as we have the
`zlib-decompress-region` function, right?


        Stefan

Re: `gzip` dependency (was: [feature/dll-only-windows] A new windows build, comments wanted)

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: Phillip Lord <phillip.lord@russet.org.uk>,  emacs-devel@gnu.org
> Date: Sun, 10 Jan 2021 15:52:07 -0500
> 
> > The gzip thing is again your call (having gzip in the package is no
> > big deal, IMO).
> 
> BTW, we don't really need a `gzip` executable as long as we have the
> `zlib-decompress-region` function, right?

Not necessarily, because Info files can be read outside Emacs.

Did we completely replaced jka-compr in Emacs?

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Phillip Lord]

Andrea Corallo <akrl@sdf.org> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>>
>> I think there are two hairs here. Having an emacs compiled with
>> nativecomp behave cleanly if libgccjit is not available at runtime.
>
> Right, at this stage this should be easy to implement (on Windows).
>
>> And having a runtime mechanism for switching nativecomp of totally
>> even if it is compiled in.
>
> We should define "switching nativecomp of" and the triggering mechanism,
> this might be already implmentented.


From a user perspective, I think that there are two reasons we might
want to switch off native-comp.

The first is because it's broken or providing buggy behaviour. In the
extreme, and if I understand the build process fully, I guess this is
impossible to switch of in a binary distribution, because some
native-comp happens before dumping. Other than that, we can "switch off"
native-comp by just deleting the *eln files, right, or otherwise
preventing their loading. In practice, I think this is a minor
motivation; if we discover bugs in the native-comp, they would be fixed
by making another release.

The second reason is that the initial compilation takes quite a lot of
CPU. I wouldn't like that to happen while my laptop where in battery
power, for instance. At the moment, it's possible to drop the number of
jobs that native-comp uses. I don't think that there is a way to drop
this to zero at the moment. Probably, we need that.

Phil

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Date: Sun, 10 Jan 2021 18:23:06 +0000
>> Cc: emacs-devel@gnu.org
>> 
>> I'd like to keep the Windows package using the default options as far as
>> possible, mostly because I am lazy and because it means that the windows
>> package updates when ever the defaults do.
>
> It's your call.  I just envision many users to expect native-comp
> support and asking why isn't it there.

No, you misunderstand me. I would plan to make Emacs-28
--with-nativecomp. But if I am doing this for the windows build, it is
reasonable to assume that other packagers will be doing for their
builds. If all that is true, it would make sense to me for it to be the
default.



>
>> Currently, we have
>> 
>> --without-dbus --without-compress-install
>> 
>> The latter is so the documentation works without gzip. The latter is
>> because we don't normally have dbus available. I am not sure why either
>> of these needed (i.e. why they are not the default on windows).
>
> The gzip thing is again your call (having gzip in the package is no
> big deal, IMO).  Dbus is not really useful on MS-Windows, so IMO it
> makes no sense building with it.

We've been down this road before. I could add gzip and I am happy to
consider this. But what other tools do we add. What about aspell
(or equivalent), so we can spell check? Git so we can version? If I add
them, then we get more "out-of-the-box" behaviour. But where do we stop.

Actually, for both of those, I'd rather see Emacs depend on their
respective libraries. Magit is widely reported to be usuable on Windows;
libgit integration might work around this. Anyway, getting off-topic.

Phil

Re: `gzip` dependency

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Stefan Monnier <monnier@iro.umontreal.ca>
>> Cc: Phillip Lord <phillip.lord@russet.org.uk>,  emacs-devel@gnu.org
>> Date: Sun, 10 Jan 2021 15:52:07 -0500
>> 
>> > The gzip thing is again your call (having gzip in the package is no
>> > big deal, IMO).
>> 
>> BTW, we don't really need a `gzip` executable as long as we have the
>> `zlib-decompress-region` function, right?
>
> Not necessarily, because Info files can be read outside Emacs.


This is not an issue for the Windows package of Emacs, as far as I can tell.


Phil

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Andrea Corallo via Emacs development discussions.]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>
>>>
>>> I think there are two hairs here. Having an emacs compiled with
>>> nativecomp behave cleanly if libgccjit is not available at runtime.
>>
>> Right, at this stage this should be easy to implement (on Windows).
>>
>>> And having a runtime mechanism for switching nativecomp of totally
>>> even if it is compiled in.
>>
>> We should define "switching nativecomp of" and the triggering mechanism,
>> this might be already implmentented.
>
>
>>From a user perspective, I think that there are two reasons we might
> want to switch off native-comp.
>
> The first is because it's broken or providing buggy behaviour. In the
> extreme, and if I understand the build process fully, I guess this is
> impossible to switch of in a binary distribution, because some
> native-comp happens before dumping. Other than that, we can "switch off"
> native-comp by just deleting the *eln files, right, or otherwise
> preventing their loading. In practice, I think this is a minor
> motivation; if we discover bugs in the native-comp, they would be fixed
> by making another release.

Hi Phillip,

ATM we can prevent Emacs loading .eln file in place of .elc
automatically setting `load-no-native' to t.

> The second reason is that the initial compilation takes quite a lot of
> CPU. I wouldn't like that to happen while my laptop where in battery
> power, for instance. At the moment, it's possible to drop the number of
> jobs that native-comp uses. I don't think that there is a way to drop
> this to zero at the moment. Probably, we need that.

Setting `comp-deferred-compilation' to nil should stop any automatic
attempt of compiling asycronousy.

Maybe these two settings are already covering most of these needs?

Regards

  Andrea

Re: `gzip` dependency

[Stefan Monnier]

>> > The gzip thing is again your call (having gzip in the package is no
>> > big deal, IMO).
>> BTW, we don't really need a `gzip` executable as long as we have the
>> `zlib-decompress-region` function, right?
> Not necessarily, because Info files can be read outside Emacs.

I don't think we should worry about that: those who want to read Info
files with something else than Emacs may need to have `gzip` installed,
and I think it's perfectly acceptable.

My question was instead about whether or not Emacs does indeed make use
of `zlib-decompress-region` when it tries to read a gzipped Info file,
and whether the Windows build does always provide
`zlib-decompress-region` or whether that function also requires an
external dependency that may or may not be available (in which case
it's no better than using the `gzip` executable in this respect).


        Stefan

Re: `gzip` dependency

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>>> > The gzip thing is again your call (having gzip in the package is no
>>> > big deal, IMO).
>>> BTW, we don't really need a `gzip` executable as long as we have the
>>> `zlib-decompress-region` function, right?
>> Not necessarily, because Info files can be read outside Emacs.
>
> I don't think we should worry about that: those who want to read Info
> files with something else than Emacs may need to have `gzip` installed,
> and I think it's perfectly acceptable.
>
> My question was instead about whether or not Emacs does indeed make use
> of `zlib-decompress-region` when it tries to read a gzipped Info file,
> and whether the Windows build does always provide
> `zlib-decompress-region` or whether that function also requires an
> external dependency that may or may not be available (in which case
> it's no better than using the `gzip` executable in this respect).



zlib is not given as an explicit dependency of Emacs on Windows at the
moment, although perhaps it should be. However, it is included as a
transitive dependency.

If I gzip all the info files then do M-x info and look into a node, I
get an error with message "Unpression program `gzip' not found". If I
open a gzip node with find-file-literally then do eval
(zlib-decompress-region (point-min) (point-max)), I get the contents of
the info file.

Conclusions:

1) Emacs uses gzip to decompress but doesn't need to.

2) zlib should probably be added an explicit dependency regardless,
since it's a feature of Emacs that can be disabled.

Phil

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: emacs-devel@gnu.org
> Date: Mon, 11 Jan 2021 09:59:29 +0000
> 
> > It's your call.  I just envision many users to expect native-comp
> > support and asking why isn't it there.
> 
> No, you misunderstand me. I would plan to make Emacs-28
> --with-nativecomp. But if I am doing this for the windows build, it is
> reasonable to assume that other packagers will be doing for their
> builds.

Not necessarily: on platforms other than MS-Windows, if Emacs was
linked against some optional library, that library must be installed
for Emacs to run without crashing.

So on MS-Windows you could build Emacs with all the possible
extensions, and let users decide which ones they want to install and
use.

> > The gzip thing is again your call (having gzip in the package is no
> > big deal, IMO).  Dbus is not really useful on MS-Windows, so IMO it
> > makes no sense building with it.
> 
> We've been down this road before.

Did we reach any conclusions?

> I could add gzip and I am happy to consider this. But what other
> tools do we add. What about aspell (or equivalent), so we can spell
> check? Git so we can version? If I add them, then we get more
> "out-of-the-box" behaviour. But where do we stop.

We stop where you decide to stop.  (And the utility of aspell is much
smaller than that of gzip, btw.  And the complexity of aspell
installation is OTOH significantly greater.)  There's no need for any
consistency here.  "Consistency is the hobgoblin", and all that.

Re: `gzip` dependency

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: Stefan Monnier <monnier@iro.umontreal.ca>,  emacs-devel@gnu.org
> Date: Mon, 11 Jan 2021 10:00:32 +0000
> 
> >> BTW, we don't really need a `gzip` executable as long as we have the
> >> `zlib-decompress-region` function, right?
> >
> > Not necessarily, because Info files can be read outside Emacs.
> 
> This is not an issue for the Windows package of Emacs, as far as I can tell.

Emacs installs Info files that can be used outside of Emacs.  So I
think it could be an issue for Emacs.

Re: `gzip` dependency

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: phillip.lord@russet.org.uk,  emacs-devel@gnu.org
> Date: Mon, 11 Jan 2021 09:59:19 -0500
> 
> My question was instead about whether or not Emacs does indeed make use
> of `zlib-decompress-region` when it tries to read a gzipped Info file,

No, I don't think we do.

> and whether the Windows build does always provide
> `zlib-decompress-region`

Only if Emacs was built with zlib.

> or whether that function also requires an external dependency that
> may or may not be available

One needs both an Emacs compiled with zlib, and the zlib DLL
available.

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Phillip Lord]

Andrea Corallo <akrl@sdf.org> writes:

>> The first is because it's broken or providing buggy behaviour. In the
>> extreme, and if I understand the build process fully, I guess this is
>> impossible to switch of in a binary distribution, because some
>> native-comp happens before dumping. Other than that, we can "switch off"
>> native-comp by just deleting the *eln files, right, or otherwise
>> preventing their loading. In practice, I think this is a minor
>> motivation; if we discover bugs in the native-comp, they would be fixed
>> by making another release.
>
> Hi Phillip,
>
> ATM we can prevent Emacs loading .eln file in place of .elc
> automatically setting `load-no-native' to t.
>
>> The second reason is that the initial compilation takes quite a lot of
>> CPU. I wouldn't like that to happen while my laptop where in battery
>> power, for instance. At the moment, it's possible to drop the number of
>> jobs that native-comp uses. I don't think that there is a way to drop
>> this to zero at the moment. Probably, we need that.
>
> Setting `comp-deferred-compilation' to nil should stop any automatic
> attempt of compiling asycronousy.
>
> Maybe these two settings are already covering most of these needs?


Yes, I think so, but it's probably an issue of thinking about the user
interface. Having this a bit more consistent (i.e. a single name space,
rather than two (native-comp and comp) and none (i.e. load-no-native))
and perhaps a single option that does both ("comp-disable").

Just thoughts.

Phil

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Andrea Corallo via Emacs development discussions.]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Andrea Corallo <akrl@sdf.org> writes:
>
>>> The first is because it's broken or providing buggy behaviour. In the
>>> extreme, and if I understand the build process fully, I guess this is
>>> impossible to switch of in a binary distribution, because some
>>> native-comp happens before dumping. Other than that, we can "switch off"
>>> native-comp by just deleting the *eln files, right, or otherwise
>>> preventing their loading. In practice, I think this is a minor
>>> motivation; if we discover bugs in the native-comp, they would be fixed
>>> by making another release.
>>
>> Hi Phillip,
>>
>> ATM we can prevent Emacs loading .eln file in place of .elc
>> automatically setting `load-no-native' to t.
>>
>>> The second reason is that the initial compilation takes quite a lot of
>>> CPU. I wouldn't like that to happen while my laptop where in battery
>>> power, for instance. At the moment, it's possible to drop the number of
>>> jobs that native-comp uses. I don't think that there is a way to drop
>>> this to zero at the moment. Probably, we need that.
>>
>> Setting `comp-deferred-compilation' to nil should stop any automatic
>> attempt of compiling asycronousy.
>>
>> Maybe these two settings are already covering most of these needs?
>
>
> Yes, I think so, but it's probably an issue of thinking about the user
> interface. Having this a bit more consistent (i.e. a single name space,
> rather than two (native-comp and comp) and none (i.e. load-no-native))
> and perhaps a single option that does both ("comp-disable").
>
> Just thoughts.

Welcome thoughts from my side.

I did my best to fit current name conventions but indeed this might be
not the best arrangement, especially looking at it as single UI
interface towards the "native compiler" seen as single entity.

Looking forward to discuss and tackle this in the code review.

  Andrea

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Cc: emacs-devel@gnu.org
>> Date: Mon, 11 Jan 2021 09:59:29 +0000
>> 
>> > It's your call.  I just envision many users to expect native-comp
>> > support and asking why isn't it there.
>> 
>> No, you misunderstand me. I would plan to make Emacs-28
>> --with-nativecomp. But if I am doing this for the windows build, it is
>> reasonable to assume that other packagers will be doing for their
>> builds.
>
> Not necessarily: on platforms other than MS-Windows, if Emacs was
> linked against some optional library, that library must be installed
> for Emacs to run without crashing.
>
> So on MS-Windows you could build Emacs with all the possible
> extensions, and let users decide which ones they want to install and
> use.

The installer version could support that kind of thing, but I don't want
to provide too many options for the sake of confusion. The sort of
people who do not want features can probably build Emacs for themselves.


>
>> > The gzip thing is again your call (having gzip in the package is no
>> > big deal, IMO).  Dbus is not really useful on MS-Windows, so IMO it
>> > makes no sense building with it.
>> 
>> We've been down this road before.
>
> Did we reach any conclusions?

Yes, we concluded that it was hard to know where to stop!


>
>> I could add gzip and I am happy to consider this. But what other
>> tools do we add. What about aspell (or equivalent), so we can spell
>> check? Git so we can version? If I add them, then we get more
>> "out-of-the-box" behaviour. But where do we stop.
>
> We stop where you decide to stop.  (And the utility of aspell is much
> smaller than that of gzip, btw.  And the complexity of aspell
> installation is OTOH significantly greater.)  There's no need for any
> consistency here.  "Consistency is the hobgoblin", and all that.

I think that this is wrong. Gzip is minimal use on Windows, except for
info files, and the total install file size is 18Mb vs 5Mb.

Compare to aspell (or hunspell) for the people installing Emacs to use
Org mode? Or git for those who want to use magit. I think it would be
even better (i.e. potentially quicker!) if we could access these as
libraries.

Still, if the number of total downloads has decreased, adding a set of
command line utilities as extra options in the installer might be worth
considering. I will cogitate.

Phil

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Corwin Brust]

Thank you much for working on this!

On Sun, Jan 10, 2021 at 9:43 AM Phillip Lord <phillip.lord@russet.org.uk> wrote:
>
> I have pushed some binaries up to alpha in case anyone wants to give
> them a trial.
>

I found a problem and thought I'd been check in case it is not
situated safely between my keyboard and chair   Also, the emacsclient
issue I mentioned on r/emacs was a self-inflicted wound; if I run it
with a property shortcut incantation (including -Q) it works great!

When I start emacs using the same configuration I use daily with Emacs
27.1 and Emacs 26.3, I get a backtrace.  Staring at this in wonder, it
looks like it could be related to use-package, which features heavily
in my setup.

tia and please do let me know what additional help I might be.

Here's the backtrace:

Debugger entered--Lisp error: (invalid-read-syntax "Invalid byte-code object")
  read(get-file-char)
  require(use-package-core)
  byte-code("\300\301!\210\300\302!\210\300\303!\210\300\304!\210\300\305!\210\306\307\310\311\312$\210\313\314!\207"
[require use-package-core use-package-bind-key use-package-diminish
use-package-delight use-package-ensure autoload
use-package-jump-to-package-form "use-package-jump" nil t provide
use-package] 5)
  require(use-package)
  (progn (require 'cl) (require 'use-package))
  eval((progn (require 'cl) (require 'use-package)) t)
  #f(compiled-function (&rest body) "Like `progn', but evaluates the
body at compile time if you're compiling.\nThus, the result of the
body appears to the compiler as a quoted\nconstant.  In interpreted
code, this is entirely equivalent to\n`progn', except that the value
of the expression may be (but is\nnot necessarily) computed at load
time if eager macro expansion\nis enabled." #<bytecode
-0x173efc045404a52b>)((require 'cl) (require 'use-package))
  (eval-when-compile (require 'cl) (require 'use-package))
  eval-buffer(#<buffer  *load*-573669> nil
"d:/projects/dotfiles/elisp/init-package-managment...." nil t)  ;
Reading at buffer position 924
  load-with-code-conversion("d:/projects/dotfiles/elisp/init-package-managment...."
"d:/projects/dotfiles/elisp/init-package-managment...." nil t)
  require(init-package-managment)
  (let ((pkg (car --dolist-tail--))) (require pkg) (setq
--dolist-tail-- (cdr --dolist-tail--)))
  (while --dolist-tail-- (let ((pkg (car --dolist-tail--))) (require
pkg) (setq --dolist-tail-- (cdr --dolist-tail--))))
  (let ((--dolist-tail-- '(init-local init-package-managment
init-editor init-org init-elisp init-projectile init-vc init-company
init-fun init-bindkey init-theme))) (while --dolist-tail-- (let ((pkg
(car --dolist-tail--))) (require pkg) (setq --dolist-tail-- (cdr
--dolist-tail--)))))
  eval-buffer(#<buffer  *load*-904898> nil
"d:/projects/dotfiles/elisp/init.el" nil t)  ; Reading at buffer
position 3546
  load-with-code-conversion("d:/projects/dotfiles/elisp/init.el"
"d:/projects/dotfiles/elisp/init.el" nil nil)
  load("d:/projects/dotfiles/elisp/init")
  eval-buffer(#<buffer  *load*> nil "c:/Users/corwi/.emacs" nil t)  ;
Reading at buffer position 414
  load-with-code-conversion("c:/Users/corwi/.emacs" "c:/Users/corwi/.emacs" t t)
  load("~/.emacs" noerror nomessage)
  startup--load-user-init-file(#f(compiled-function () #<bytecode
0x5a5c7670ff81e72>) #f(compiled-function () #<bytecode
0xc085514c3215d0c>) t)
  command-line()
  normal-top-level()


Regards,
Corwin
corwin.bru.st

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Phillip Lord]

Can you do a compare and contrast with this snapshot version?

https://alpha.gnu.org/gnu/emacs/pretest/windows/emacs-28/emacs-28.0.50-snapshot-2021-01-04-x86_64.zip

This was built at about the same time as the dll-only version. I doubt
that this problem is caused by the new package, but this will tell us
for sure.

Phil

Corwin Brust <corwin@bru.st> writes:

> Thank you much for working on this!
>
> On Sun, Jan 10, 2021 at 9:43 AM Phillip Lord
> <phillip.lord@russet.org.uk> wrote:
>>
>> I have pushed some binaries up to alpha in case anyone wants to give
>> them a trial.
>>
>
> I found a problem and thought I'd been check in case it is not
> situated safely between my keyboard and chair   Also, the emacsclient
> issue I mentioned on r/emacs was a self-inflicted wound; if I run it
> with a property shortcut incantation (including -Q) it works great!
>
> When I start emacs using the same configuration I use daily with Emacs
> 27.1 and Emacs 26.3, I get a backtrace.  Staring at this in wonder, it
> looks like it could be related to use-package, which features heavily
> in my setup.
>
> tia and please do let me know what additional help I might be.
>
> Here's the backtrace:
>
> Debugger entered--Lisp error: (invalid-read-syntax "Invalid byte-code
> object")
>   read(get-file-char)
>   require(use-package-core)
>   byte-code("\300\301!\210\300\302!\210\300\303!\210\300\304!\210\300\305!\210\306\307\310\311\312$\210\313\314!\207"
> [require use-package-core use-package-bind-key use-package-diminish
> use-package-delight use-package-ensure autoload
> use-package-jump-to-package-form "use-package-jump" nil t provide
> use-package] 5)
>   require(use-package)
>   (progn (require 'cl) (require 'use-package))
>   eval((progn (require 'cl) (require 'use-package)) t)
>   #f(compiled-function (&rest body) "Like `progn', but evaluates the
> body at compile time if you're compiling.\nThus, the result of the
> body appears to the compiler as a quoted\nconstant.  In interpreted
> code, this is entirely equivalent to\n`progn', except that the value
> of the expression may be (but is\nnot necessarily) computed at load
> time if eager macro expansion\nis enabled." #<bytecode
> -0x173efc045404a52b>)((require 'cl) (require 'use-package))
>   (eval-when-compile (require 'cl) (require 'use-package))
>   eval-buffer(#<buffer  *load*-573669> nil
> "d:/projects/dotfiles/elisp/init-package-managment...." nil t)  ;
> Reading at buffer position 924
>   load-with-code-conversion("d:/projects/dotfiles/elisp/init-package-managment...."
> "d:/projects/dotfiles/elisp/init-package-managment...." nil t)
>   require(init-package-managment)
>   (let ((pkg (car --dolist-tail--))) (require pkg) (setq
> --dolist-tail-- (cdr --dolist-tail--)))
>   (while --dolist-tail-- (let ((pkg (car --dolist-tail--))) (require
> pkg) (setq --dolist-tail-- (cdr --dolist-tail--))))
>   (let ((--dolist-tail-- '(init-local init-package-managment
> init-editor init-org init-elisp init-projectile init-vc init-company
> init-fun init-bindkey init-theme))) (while --dolist-tail-- (let ((pkg
> (car --dolist-tail--))) (require pkg) (setq --dolist-tail-- (cdr
> --dolist-tail--)))))
>   eval-buffer(#<buffer  *load*-904898> nil
> "d:/projects/dotfiles/elisp/init.el" nil t)  ; Reading at buffer
> position 3546
>   load-with-code-conversion("d:/projects/dotfiles/elisp/init.el"
> "d:/projects/dotfiles/elisp/init.el" nil nil)
>   load("d:/projects/dotfiles/elisp/init")
>   eval-buffer(#<buffer  *load*> nil "c:/Users/corwi/.emacs" nil t)  ;
> Reading at buffer position 414
>   load-with-code-conversion("c:/Users/corwi/.emacs"
> "c:/Users/corwi/.emacs" t t)
>   load("~/.emacs" noerror nomessage)
>   startup--load-user-init-file(#f(compiled-function () #<bytecode
> 0x5a5c7670ff81e72>) #f(compiled-function () #<bytecode
> 0xc085514c3215d0c>) t)
>   command-line()
>   normal-top-level()
>
>
> Regards,
> Corwin
> corwin.bru.st

Re: [feature/dll-only-windows] A new windows build, comments wanted

[Corwin Brust]

You are right!

On Tue, Jan 12, 2021 at 3:48 AM Phillip Lord <phillip.lord@russet.org.uk> wrote:
> Can you do a compare and contrast with this snapshot version?
>
> https://alpha.gnu.org/gnu/emacs/pretest/windows/emacs-28/emacs-28.0.50-snapshot-2021-01-04-x86_64.zip
>
> This was built at about the same time as the dll-only version. I doubt
> that this problem is caused by the new package, but this will tell us
> for sure.

Same backtrace, at a glance.  Open a new bug for this, then?

Debugger entered--Lisp error: (invalid-read-syntax "Invalid byte-code object")
  read(get-file-char)
  require(use-package-core)
  byte-code("\300\301!\210\300\302!\210\300\303!\210\300\304!\210\300\305!\210\306\307\310\311\312$\210\313\314!\207"
[require use-package-core use-package-bind-key use-package-diminish
use-package-delight use-package-ensure autoload
use-package-jump-to-package-form "use-package-jump" nil t provide
use-package] 5)
  require(use-package)
  (progn (require 'cl) (require 'use-package))
  eval((progn (require 'cl) (require 'use-package)) t)
  #f(compiled-function (&rest body) "Like `progn', but evaluates the
body at compile time if you're compiling.\nThus, the result of the
body appears to the compiler as a quoted\nconstant.  In interpreted
code, this is entirely equivalent to\n`progn', except that the value
of the expression may be (but is\nnot necessarily) computed at load
time if eager macro expansion\nis enabled." #<bytecode
-0x173efc045404914c>)((require 'cl) (require 'use-package))
  (eval-when-compile (require 'cl) (require 'use-package))
  eval-buffer(#<buffer  *load*-275982> nil
"d:/projects/dotfiles/elisp/init-package-managment...." nil t)  ;
Reading at buffer position 924
  load-with-code-conversion("d:/projects/dotfiles/elisp/init-package-managment...."
"d:/projects/dotfiles/elisp/init-package-managment...." nil t)
  require(init-package-managment)
  (let ((pkg (car --dolist-tail--))) (require pkg) (setq
--dolist-tail-- (cdr --dolist-tail--)))
  (while --dolist-tail-- (let ((pkg (car --dolist-tail--))) (require
pkg) (setq --dolist-tail-- (cdr --dolist-tail--))))
  (let ((--dolist-tail-- '(init-local init-package-managment
init-editor init-org init-elisp init-projectile init-vc init-company
init-fun init-bindkey init-theme))) (while --dolist-tail-- (let ((pkg
(car --dolist-tail--))) (require pkg) (setq --dolist-tail-- (cdr
--dolist-tail--)))))
  eval-buffer(#<buffer  *load*-477369> nil
"d:/projects/dotfiles/elisp/init.el" nil t)  ; Reading at buffer
position 3546
  load-with-code-conversion("d:/projects/dotfiles/elisp/init.el"
"d:/projects/dotfiles/elisp/init.el" nil nil)
  load("d:/projects/dotfiles/elisp/init")
  eval-buffer(#<buffer  *load*> nil "c:/Users/corwi/.emacs" nil t)  ;
Reading at buffer position 414
  load-with-code-conversion("c:/Users/corwi/.emacs" "c:/Users/corwi/.emacs" t t)
  load("~/.emacs" noerror nomessage)
  startup--load-user-init-file(#f(compiled-function () #<bytecode
0x5a5c782f8c782d2>) #f(compiled-function () #<bytecode
0xc085514c3215d0c>) t)
  command-line()
  normal-top-level()


> > Regards,
> > Corwin
> > corwin.bru.st

[feature/internal-msys] thoughts of a more function windows package

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> The gzip thing is again your call (having gzip in the package is no
>> big deal, IMO).
>
> BTW, we don't really need a `gzip` executable as long as we have the
> `zlib-decompress-region` function, right?




Following on from this discussion, I have had a thought about a better
way of producing a more function windows download and have come up with
this idea, which is just to install msys inside the Emacs install
directory.

The plan would be during the installation run the newly installed
Emacs. This would download, unpack and configure msys2, as well as
install a variety of related packages (git, find, gzip, hunspell,
aspell, what ever). For those using the zip package of Emacs, they would
achieve the same thing by running a command.

I have given this a quick bash. It could be achieved without bloating
the Emacs download; I'd have to had xz to the Emacs installer to allow
me to unpack msys, but otherwise it would have no implications. I would
not need to host either msys2 nor its source on the Gnu FTP because,
this would all come from msys2. The installation process would be Emacs
driven, which would be easier for me to write. Emacs would remain using
the DLLs we ship with it and so stable, while the underlying msys2 would
be independently updateable. And, to all intents and purposes, it would
look like a normal windows app; packaged, bundled and installed in
directory.

What I have so far:

1) w32-msys-install

which updates msys and installs git as well as adds a loader to site-start.el

2) w32-msys-site-start.el

which updates the path and so

To be fully functional 1) would need to download msys and unpack it and
I'd need to hook it into the installer.

It would solve the question of what should I package -- I'd add some
basic utilities, and thereafter, the user could add what ever they
wanted.


Thoughts? Does this seem worth following up?

Phil

Re: [feature/internal-msys] thoughts of a more function windows package

[Stephen Leake]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Following on from this discussion, I have had a thought about a better
> way of producing a more function windows download and have come up with
> this idea, which is just to install msys inside the Emacs install
> directory.

Works for me, as long as it detects my current msys install and does not
modify it without asking my permission.

-- 
-- Stephe

Re: [feature/internal-msys] thoughts of a more function windows package

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Date: Wed, 20 Jan 2021 19:29:58 +0000
> 
> The plan would be during the installation run the newly installed
> Emacs. This would download, unpack and configure msys2, as well as
> install a variety of related packages (git, find, gzip, hunspell,
> aspell, what ever). For those using the zip package of Emacs, they would
> achieve the same thing by running a command.

Could you please elaborate on how will this work for users?  E.g., how
about describing the steps a user who installs Emacs should do to
install these additions?  I don't think I have a clear picture of what
you intend to do, and the files you committed are just the starting
point, so they don't answer this question.

Btw, I don't think I understand why you insist on using 'concat' to
generate files names from 2 components: that's what expand-file-name
is for, and it does that better.

One other not about the code you committed is that modifying PATH from
within Emacs is not generally a good idea, it has some subtleties.

Re: [feature/internal-msys] thoughts of a more function windows package

[Phillip Lord]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> Following on from this discussion, I have had a thought about a better
>> way of producing a more function windows download and have come up with
>> this idea, which is just to install msys inside the Emacs install
>> directory.
>
> Works for me, as long as it detects my current msys install and does not
> modify it without asking my permission.

I have tested this; you can have two msys installations on a machine and
they just ignore each other. In practice, someone who already has an
msys installation wouldn't want this, although there is no reason why
any technology that we put in place for managing an internal msys
shouldn't also work for a preinstalled one.

For example, there might be a command "install recommended packages"
which would just update your local msys an add those packages that work
with Emacs. Likewise, if there is an expectation that Emacs will be used
in this way, we might modify Emacs to work "properly"; for example, M-x
find tends to break on Windows because it finds the wrong "find"
command.

Phil

Re: [feature/internal-msys] thoughts of a more function windows package

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Date: Wed, 20 Jan 2021 19:29:58 +0000
>> 
>> The plan would be during the installation run the newly installed
>> Emacs. This would download, unpack and configure msys2, as well as
>> install a variety of related packages (git, find, gzip, hunspell,
>> aspell, what ever). For those using the zip package of Emacs, they would
>> achieve the same thing by running a command.
>
> Could you please elaborate on how will this work for users?  E.g., how
> about describing the steps a user who installs Emacs should do to
> install these additions?  I don't think I have a clear picture of what
> you intend to do, and the files you committed are just the starting
> point, so they don't answer this question.

Considering just the .exe installer, at the end of the installation
process, we would run Emacs with menu bar, scroll bar and everything
switch off and run the command "w32-msys-internal-install-dialog".

Emacs would say "do you want do install a handy collection of
tools". User says "y", then Emacs downloads the latest msys2-date.xz,
then unpacks it with xz. Emacs will run msys2.exe to initialize
(hopefully without poping up windows if this will work). Then Emacs will
run pacman to update and install. The user will see Emacs saying

Would you like to install msys, and a handy set of tools (Y/n)
Installing msys...done
Updating msys...done
Installing git...done

When it's finished, the user starts Emacs and does vc-dir and it works,
instead of seeing "git not found" errors. For those using the zip file,
the user would have to launch "M-x w32-msys-install" manually.

At a later date, I might add some cuter things. So, for example we might
install:

https://github.com/brotzeit/arch-packer

or equivalent. We might create "internal-msys" package on ELPA which
contains a standard set of packages and Emacs fixes to make it work, so
it would be freely updatable. At which point the w32-msys-install
command would install the ELPA package before doing anything else.

There are also issues that when Emacs is removed msys (and any
configuration) would get blitzed, so I'd have to think about how to
persist msys configuration between two installs of Emacs. Currently the
file organisation looks like this:

c:/Program Files/Emacs/emacs-28.1/bin
c:/Program Files/Emacs/emacs-28.1/share
c:/Program Files/Emacs/emacs-28.1/msys

(etc)

but 

c:/Program Files/Emacs/emacs-28.1/bin
c:/Program Files/Emacs/emacs-28.1/share
c:/Program Files/Emacs/msys

would mean that installing Emacs-28.2 would work naturally.

There are also some issues about user persmissions; I only ever run
Windows as administrator, so there will be some thought given as to how
this would work. Neither Emacs or msys require this AFAICT.



> Btw, I don't think I understand why you insist on using 'concat' to
> generate files names from 2 components: that's what expand-file-name
> is for, and it does that better.

Because in my own code, I tend to use f.el and I have got used to having
a clean, consistent and comprehensive API, so I tend to forget the
underlying Emacs primitives.


> One other not about the code you committed is that modifying PATH from
> within Emacs is not generally a good idea, it has some subtleties.

Any alternative that achieves the same thing?

Phil

Re: [feature/internal-msys] thoughts of a more function windows package

[Stephen Leake]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> Following on from this discussion, I have had a thought about a better
>> way of producing a more function windows download and have come up with
>> this idea, which is just to install msys inside the Emacs install
>> directory.
>
> Works for me, as long as it detects my current msys install and does not
> modify it without asking my permission.

Actually, a better way to proceed in this direction is to update the
current emacs mingw64 package; it is currently at 26.

-- 
-- Stephe

Re: [feature/internal-msys] thoughts of a more function windows package

[phillip.lord]

On 2021-01-21 18:22, Stephen Leake wrote:
> Stephen Leake <stephen_leake@stephe-leake.org> writes:
> 
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>> 
>>> Following on from this discussion, I have had a thought about a 
>>> better
>>> way of producing a more function windows download and have come up 
>>> with
>>> this idea, which is just to install msys inside the Emacs install
>>> directory.
>> 
>> Works for me, as long as it detects my current msys install and does 
>> not
>> modify it without asking my permission.
> 
> Actually, a better way to proceed in this direction is to update the
> current emacs mingw64 package; it is currently at 26.

Is it?

https://packages.msys2.org/package/mingw-w64-x86_64-emacs?repo=mingw64

report 27.1

Phil

Re: [feature/internal-msys] thoughts of a more function windows package

[Óscar Fuentes]

Stephen Leake <stephen_leake@stephe-leake.org> writes:

> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>
>>> Following on from this discussion, I have had a thought about a better
>>> way of producing a more function windows download and have come up with
>>> this idea, which is just to install msys inside the Emacs install
>>> directory.
>>
>> Works for me, as long as it detects my current msys install and does not
>> modify it without asking my permission.
>
> Actually, a better way to proceed in this direction is to update the
> current emacs mingw64 package; it is currently at 26.

The MSYS2/MinGW64 Emacs package? It was updated to 27.1 one day after
the release.

Re: [feature/internal-msys] thoughts of a more function windows package

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: emacs-devel@gnu.org
> Date: Thu, 21 Jan 2021 16:44:02 +0000
> 
> > Could you please elaborate on how will this work for users?  E.g., how
> > about describing the steps a user who installs Emacs should do to
> > install these additions?  I don't think I have a clear picture of what
> > you intend to do, and the files you committed are just the starting
> > point, so they don't answer this question.
> 
> Considering just the .exe installer, at the end of the installation
> process, we would run Emacs with menu bar, scroll bar and everything
> switch off and run the command "w32-msys-internal-install-dialog".

"We" being the installation process?  IOW, this will start
automatically at the end of the normal installation?

> Emacs would say "do you want do install a handy collection of
> tools". User says "y", then Emacs downloads the latest msys2-date.xz,
> then unpacks it with xz. Emacs will run msys2.exe to initialize
> (hopefully without poping up windows if this will work). Then Emacs will
> run pacman to update and install. The user will see Emacs saying
> 
> Would you like to install msys, and a handy set of tools (Y/n)
> Installing msys...done
> Updating msys...done
> Installing git...done

First, why Git?  This is a "normal" user, not an Emacs developer,
right?  Git is a huge package (and comes with Vim on top of that), so
maybe reconsider that part.

Next, by "msys" you mean all those non-native programs that depend on
msys-1.0.dll?  That's again meant for MinGW developers, not "normal"
users.

And finally, which packages will pacman install? are you going to
provide some list of packages, and if so, what will be there?

(Btw, pacman can ask questions and prompt the user for confirmation,
but the way you invoke it in w32-msys-run doesn't seem to be prepared
for such interaction?)

One other aspect that bothers me is that if the user already has some
parts of MSYS/MinGW64 installed, they will now have two places with
DLLs, which is a step towards "DLL hell".

> We might create "internal-msys" package on ELPA which
> contains a standard set of packages and Emacs fixes to make it work, so
> it would be freely updatable. At which point the w32-msys-install
> command would install the ELPA package before doing anything else.

Is it really possible for us to distribute MSYS, given its license?

> > Btw, I don't think I understand why you insist on using 'concat' to
> > generate files names from 2 components: that's what expand-file-name
> > is for, and it does that better.
> 
> Because in my own code, I tend to use f.el and I have got used to having
> a clean, consistent and comprehensive API, so I tend to forget the
> underlying Emacs primitives.

Then I think f.el has a big problem.

> > One other not about the code you committed is that modifying PATH from
> > within Emacs is not generally a good idea, it has some subtleties.
> 
> Any alternative that achieves the same thing?

I think the only good idea here is to tell the user to amend PATH by
adding such-and-such directories to it.  I don't like installers that
futz with my PATH, and would hate it if Emacs did that to others.
It's very easy to get that wrong, especially on Windows.

Re: [feature/internal-msys] thoughts of a more function windows package

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> Considering just the .exe installer, at the end of the installation
>> process, we would run Emacs with menu bar, scroll bar and everything
>> switch off and run the command "w32-msys-internal-install-dialog".
>
> "We" being the installation process?  IOW, this will start
> automatically at the end of the normal installation?


Yep.


>> Emacs would say "do you want do install a handy collection of
>> tools". User says "y", then Emacs downloads the latest msys2-date.xz,
>> then unpacks it with xz. Emacs will run msys2.exe to initialize
>> (hopefully without poping up windows if this will work). Then Emacs will
>> run pacman to update and install. The user will see Emacs saying
>> 
>> Would you like to install msys, and a handy set of tools (Y/n)
>> Installing msys...done
>> Updating msys...done
>> Installing git...done
>
> First, why Git?  This is a "normal" user, not an Emacs developer,
> right?  Git is a huge package (and comes with Vim on top of that), so
> maybe reconsider that part.

That's open for question. I'd have to think about the packages I would
want to use. However, for me, git is one of the first commands that
Emacs complains about the absence of; this is because I use Emacs on my
windows build machine, of course, which contains the Emacs git
repo. But, also because most IDEs come with the ability to interact with
git off the shelf. I think Windows users will expect it.

Yep, it's big. I get this before installation:

319 MB (335,214,637 bytes)

and this after installation.

826 MB (866,624,644 bytes)

My guess is that it's the co-install of perl that is causing this issue
rather than vim. But, openssh comes for free with this. And "huge" is a
relative term. An installation under 1Gb seems reasonable to me.

> Next, by "msys" you mean all those non-native programs that depend on
> msys-1.0.dll?  That's again meant for MinGW developers, not "normal"
> users.

Yes. Because it's got all the packages and tools and as far as I can
see, they work with Emacs.


> And finally, which packages will pacman install? are you going to
> provide some list of packages, and if so, what will be there?

Yes. Initially hard coded into the Emacs install, we might turn the list
of msys packages into an ELPA package. That way, we could update the
list of tools supported, without updating the Emacs distribution.


> (Btw, pacman can ask questions and prompt the user for confirmation,
> but the way you invoke it in w32-msys-run doesn't seem to be prepared
> for such interaction?)

That's why I use "--no-confirm". I'm looking for the minimum here that
does something usable.


> One other aspect that bothers me is that if the user already has some
> parts of MSYS/MinGW64 installed, they will now have two places with
> DLLs, which is a step towards "DLL hell".

Yes, it is. And indeed, the DLLs distributed in the Emacs distribution
might well get duplicated in the msys install. But they would be in one
place. My presumption is that people who use msys already would not want
this, and it would not be forced upon them.


>> We might create "internal-msys" package on ELPA which
>> contains a standard set of packages and Emacs fixes to make it work, so
>> it would be freely updatable. At which point the w32-msys-install
>> command would install the ELPA package before doing anything else.
>
> Is it really possible for us to distribute MSYS, given its license?

You misunderstand, my poor wording. The ELPA package would not contain
msys, but would contain list of msys package names and supporting
code. Msys would come from the msys2 repo.


>> > Btw, I don't think I understand why you insist on using 'concat' to
>> > generate files names from 2 components: that's what expand-file-name
>> > is for, and it does that better.
>> 
>> Because in my own code, I tend to use f.el and I have got used to having
>> a clean, consistent and comprehensive API, so I tend to forget the
>> underlying Emacs primitives.
>
> Then I think f.el has a big problem.

I was being slightly provocative; I shall divert the discussion no
further.



>> > One other not about the code you committed is that modifying PATH from
>> > within Emacs is not generally a good idea, it has some subtleties.
>> 
>> Any alternative that achieves the same thing?
>
> I think the only good idea here is to tell the user to amend PATH by
> adding such-and-such directories to it.  I don't like installers that
> futz with my PATH, and would hate it if Emacs did that to others.
> It's very easy to get that wrong, especially on Windows.


There has to be a better way that this. Editing environment variables by
hand is so last century. Of course, I do not want to edit the path
globally. If I can do it in Emacs, then I would have to think of a
different solution.

Phil

Re: [feature/internal-msys] thoughts of a more function windows package

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: emacs-devel@gnu.org
> Date: Thu, 21 Jan 2021 21:37:57 +0000
> 
> Yep, it's big. I get this before installation:
> 
> 319 MB (335,214,637 bytes)
> 
> and this after installation.
> 
> 826 MB (866,624,644 bytes)
> 
> My guess is that it's the co-install of perl that is causing this issue
> rather than vim.

It's Vim, and Perl, and Tcl.  And at some point I'd expect them to
include Python as well, to support those Git commands which rely on
it.

> But, openssh comes for free with this. And "huge" is a
> relative term. An installation under 1Gb seems reasonable to me.

I thought this started as an attempt to make the Emacs installation
smaller...

If the rationale is to provide a full development environment on top
of MS-Windows, then indeed the size will be much larger, and you will
need to include a lot of packages there.  Just be sure to spell this
out (including the size, perhaps) when you ask the user whether he or
she wants to install that.

> > Next, by "msys" you mean all those non-native programs that depend on
> > msys-1.0.dll?  That's again meant for MinGW developers, not "normal"
> > users.
> 
> Yes. Because it's got all the packages and tools and as far as I can
> see, they work with Emacs.

They mostly work, until they don't.  Like with Cygwin, there are
subtle incompatibilities, mainly in file names and in communications
with subprocesses and response to "signals".  Encoding defaults are
also different.

> > (Btw, pacman can ask questions and prompt the user for confirmation,
> > but the way you invoke it in w32-msys-run doesn't seem to be prepared
> > for such interaction?)
> 
> That's why I use "--no-confirm". I'm looking for the minimum here that
> does something usable.

What happens when pacman wants to replace Bash from which it was
launched, or update itself or some of its component DLLs?

> > I think the only good idea here is to tell the user to amend PATH by
> > adding such-and-such directories to it.  I don't like installers that
> > futz with my PATH, and would hate it if Emacs did that to others.
> > It's very easy to get that wrong, especially on Windows.
> 
> There has to be a better way that this.

The only way I know of is to distribute a program that writes into the
Registry.  Don't forget that there are system-wide variables and
variables specific to the current user.  And some systems have the
former locked down and the latter requires a UAC elevation.  Good luck
(you will need it) with successfully negotiating all these obstacles.

Re: [feature/internal-msys] thoughts of a more function windows package

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Cc: emacs-devel@gnu.org
>> Date: Thu, 21 Jan 2021 21:37:57 +0000
>> 
>> Yep, it's big. I get this before installation:
>> 
>> 319 MB (335,214,637 bytes)
>> 
>> and this after installation.
>> 
>> 826 MB (866,624,644 bytes)
>> 
>> My guess is that it's the co-install of perl that is causing this issue
>> rather than vim.
>
> It's Vim, and Perl, and Tcl.  And at some point I'd expect them to
> include Python as well, to support those Git commands which rely on
> it.

Yep, they don't ship a "git-core" without all this baggage
unfortunately.


>> But, openssh comes for free with this. And "huge" is a
>> relative term. An installation under 1Gb seems reasonable to me.
>
> I thought this started as an attempt to make the Emacs installation
> smaller...
>
> If the rationale is to provide a full development environment on top
> of MS-Windows, then indeed the size will be much larger, and you will
> need to include a lot of packages there.  Just be sure to spell this
> out (including the size, perhaps) when you ask the user whether he or
> she wants to install that.

Yes, a small install for easy downloads, but with the option for a
bigger one. The small install is still important, though, because it
will also mean a small upgrade from one emacs to the next.


>> > Next, by "msys" you mean all those non-native programs that depend on
>> > msys-1.0.dll?  That's again meant for MinGW developers, not "normal"
>> > users.
>> 
>> Yes. Because it's got all the packages and tools and as far as I can
>> see, they work with Emacs.
>
> They mostly work, until they don't.  Like with Cygwin, there are
> subtle incompatibilities, mainly in file names and in communications
> with subprocesses and response to "signals".  Encoding defaults are
> also different.

That's true for the msys2 commands but not the mingw64 ones? In the end,
not all of the tools need for Emacs are in mingw64 part of msys2 and
msys2 comes with pacman.

>> > (Btw, pacman can ask questions and prompt the user for confirmation,
>> > but the way you invoke it in w32-msys-run doesn't seem to be prepared
>> > for such interaction?)
>> 
>> That's why I use "--no-confirm". I'm looking for the minimum here that
>> does something usable.
>
> What happens when pacman wants to replace Bash from which it was
> launched, or update itself or some of its component DLLs?

That happens straight away on first launch. I don't know yet, but am
working on it.


>
>> > I think the only good idea here is to tell the user to amend PATH by
>> > adding such-and-such directories to it.  I don't like installers that
>> > futz with my PATH, and would hate it if Emacs did that to others.
>> > It's very easy to get that wrong, especially on Windows.
>> 
>> There has to be a better way that this.
>
> The only way I know of is to distribute a program that writes into the
> Registry.  Don't forget that there are system-wide variables and
> variables specific to the current user.  And some systems have the
> former locked down and the latter requires a UAC elevation.  Good luck
> (you will need it) with successfully negotiating all these obstacles.

Would it be easier to have Emacs allow me to successfully update PATH
during run?

Phil

Re: [feature/internal-msys] thoughts of a more function windows package

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: emacs-devel@gnu.org
> Date: Fri, 22 Jan 2021 16:14:33 +0000
> 
> >> > Next, by "msys" you mean all those non-native programs that depend on
> >> > msys-1.0.dll?  That's again meant for MinGW developers, not "normal"
> >> > users.
> >> 
> >> Yes. Because it's got all the packages and tools and as far as I can
> >> see, they work with Emacs.
> >
> > They mostly work, until they don't.  Like with Cygwin, there are
> > subtle incompatibilities, mainly in file names and in communications
> > with subprocesses and response to "signals".  Encoding defaults are
> > also different.
> 
> That's true for the msys2 commands but not the mingw64 ones?

Yes.  That's why I asked about msys-1.0.dll: the programs that depend
on that aren't mingw64 (native) programs.

> > The only way I know of is to distribute a program that writes into the
> > Registry.  Don't forget that there are system-wide variables and
> > variables specific to the current user.  And some systems have the
> > former locked down and the latter requires a UAC elevation.  Good luck
> > (you will need it) with successfully negotiating all these obstacles.
> 
> Would it be easier to have Emacs allow me to successfully update PATH
> during run?

Update how?

Re: [feature/internal-msys] thoughts of a more function windows package

[Stephen Leake]

phillip.lord@russet.org.uk writes:

> On 2021-01-21 18:22, Stephen Leake wrote:
>> Stephen Leake <stephen_leake@stephe-leake.org> writes:
>> 
>>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>> 
>>>> Following on from this discussion, I have had a thought about a
>>>> better
>>>> way of producing a more function windows download and have come up
>>>> with
>>>> this idea, which is just to install msys inside the Emacs install
>>>> directory.
>>> Works for me, as long as it detects my current msys install and
>>> does not
>>> modify it without asking my permission.
>> Actually, a better way to proceed in this direction is to update the
>> current emacs mingw64 package; it is currently at 26.
>
> Is it?
>
> https://packages.msys2.org/package/mingw-w64-x86_64-emacs?repo=mingw64
>
> report 27.1

ah, my local mingw64 cache is out of date. Sorry.

-- 
-- Stephe

Re: [feature/internal-msys] thoughts of a more function windows package

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> > They mostly work, until they don't.  Like with Cygwin, there are
>> > subtle incompatibilities, mainly in file names and in communications
>> > with subprocesses and response to "signals".  Encoding defaults are
>> > also different.
>> 
>> That's true for the msys2 commands but not the mingw64 ones?
>
> Yes.  That's why I asked about msys-1.0.dll: the programs that depend
> on that aren't mingw64 (native) programs.


But, mingw64 does not have all the packages I need.

How do people use Emacs on windows? I mean, do they install find, ls,
git, aspell and all the rest by hand? It's been a long time since I have
done it.


>> > The only way I know of is to distribute a program that writes into the
>> > Registry.  Don't forget that there are system-wide variables and
>> > variables specific to the current user.  And some systems have the
>> > former locked down and the latter requires a UAC elevation.  Good luck
>> > (you will need it) with successfully negotiating all these obstacles.
>> 
>> Would it be easier to have Emacs allow me to successfully update PATH
>> during run?
>
> Update how?

Using some magic that doesn't exist at the current time. I mean a way
like did (by fiddling with setenv) that doesn't suffer the problems that
it causes.

Phil

Re: [feature/internal-msys] thoughts of a more function windows package

[Óscar Fuentes]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
>>> > They mostly work, until they don't.  Like with Cygwin, there are
>>> > subtle incompatibilities, mainly in file names and in communications
>>> > with subprocesses and response to "signals".  Encoding defaults are
>>> > also different.
>>> 
>>> That's true for the msys2 commands but not the mingw64 ones?
>>
>> Yes.  That's why I asked about msys-1.0.dll: the programs that depend
>> on that aren't mingw64 (native) programs.
>
>
> But, mingw64 does not have all the packages I need.

Ok, here goes a data point:

> How do people use Emacs on windows?

I install MSYS2 and execute a shell script that installs this packages:

universal-ctags-git ag putty-ssh aspell aspell-en aspell-es diffutils

(the scripts prepends the names listed above with the corresponding
mingw-w64-i686/x86_64 prefix). Then I install emacs with `pacman`,
either from MSYS2 binary repository or from the packages I build.

> I mean, do they install find,

I don't use it. I don't use it on GNU/Linux either (I work 99% of the
time on GNU/Linux).

> ls,

Is it necessary?

> git,

Git for Windows.

> aspell

See above.

> and all the rest by hand?

On my case, only Git for Windows and MSYS2 are installed by hand.

BTW, Git for Windows is also based on MSYS2: it installs the required
pieces that are required for running the parts of git that still depend
on POSIX.

It theory, instead of Git for Windows I could also run MSYS2's own
git.exe setting magit-git-executable and vc-git-program, without adding
MSYS2 /bin directory to PATH, but last time I tried it was somewhat
slower than Git for Windows, which is slow enough itself. Also, the
later comes with some goodies built-in, such as git-svn. MSYS2 git
package requires about 30 MB, while Git for Windows 240.

I execute runemacs.exe from wathever/msys64/mingw64/bin. emacs.el sets
PATH and exec-path:

(defun anade-a-path (path)
  (setenv "PATH" (concat (getenv "PATH") path-separator dir))
  (setq exec-path (append exec-path (list dir))))

(anade-a-path (file-name-directory (car command-line-args)))

With this I can execute programs on emacs.exe directory (that is,
whatever/msys64/mingw64/bin) with no problems.

Re: [feature/internal-msys] thoughts of a more function windows package

[Phillip Lord]

Óscar Fuentes <ofv@wanadoo.es> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> But, mingw64 does not have all the packages I need.
>
> Ok, here goes a data point:
>
>> How do people use Emacs on windows?
>
> I install MSYS2 and execute a shell script that installs this
> packages:
>
> universal-ctags-git ag putty-ssh aspell aspell-en aspell-es diffutils
>
> (the scripts prepends the names listed above with the corresponding
> mingw-w64-i686/x86_64 prefix). Then I install emacs with `pacman`,
> either from MSYS2 binary repository or from the packages I build.

In your case, you are using msys effectively as an installer, to install
mingw64 packages. Likewise, I presume, with Emacs? This seems to come in
both an msys2 and mingw64 package.

Any reason for putty rather than openssh?


>> I mean, do they install find,
>
> I don't use it. I don't use it on GNU/Linux either (I work 99% of the
> time on GNU/Linux).

Oh, I'm a find junkie.

>> ls,
>
> Is it necessary?

Not sure. I thought dired used it, at least if it's there but maybe not.


>> git,
>
> Git for Windows.
>
>> aspell
>
> See above.
>
>> and all the rest by hand?
>
> On my case, only Git for Windows and MSYS2 are installed by hand.
>
> BTW, Git for Windows is also based on MSYS2: it installs the required
> pieces that are required for running the parts of git that still depend
> on POSIX.
>
> It theory, instead of Git for Windows I could also run MSYS2's own
> git.exe setting magit-git-executable and vc-git-program, without adding
> MSYS2 /bin directory to PATH, but last time I tried it was somewhat
> slower than Git for Windows, which is slow enough itself. Also, the
> later comes with some goodies built-in, such as git-svn. MSYS2 git
> package requires about 30 MB, while Git for Windows 240.

Yes, I hear that magit is difficult to use on Windows. I don't know if
that is fixable within Emacs, or if it is fundamental to Windows.



> I execute runemacs.exe from wathever/msys64/mingw64/bin. emacs.el sets
> PATH and exec-path:
>
> (defun anade-a-path (path)
>   (setenv "PATH" (concat (getenv "PATH") path-separator dir))
>   (setq exec-path (append exec-path (list dir))))
>
> (anade-a-path (file-name-directory (car command-line-args)))
>
> With this I can execute programs on emacs.exe directory (that is,
> whatever/msys64/mingw64/bin) with no problems.

That's a good data point! Thank you.

Phil

Re: [feature/internal-msys] thoughts of a more function windows package

[Óscar Fuentes]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Óscar Fuentes <ofv@wanadoo.es> writes:
>
>> Phillip Lord <phillip.lord@russet.org.uk> writes:
>>
>>> But, mingw64 does not have all the packages I need.
>>
>> Ok, here goes a data point:
>>
>>> How do people use Emacs on windows?
>>
>> I install MSYS2 and execute a shell script that installs this
>> packages:
>>
>> universal-ctags-git ag putty-ssh aspell aspell-en aspell-es diffutils
>>
>> (the scripts prepends the names listed above with the corresponding
>> mingw-w64-i686/x86_64 prefix). Then I install emacs with `pacman`,
>> either from MSYS2 binary repository or from the packages I build.
>
> In your case, you are using msys effectively as an installer, to install
> mingw64 packages. Likewise, I presume, with Emacs? This seems to come in
> both an msys2 and mingw64 package.
>
> Any reason for putty rather than openssh?

There is no mingw64 package of openssh and putty is "made for Windows",
which means that everything works out of the box (I'm not talking about
Tramp, which I don't use.)

>>> I mean, do they install find,
>>
>> I don't use it. I don't use it on GNU/Linux either (I work 99% of the
>> time on GNU/Linux).
>
> Oh, I'm a find junkie.

I also use find quite often while using a shell. For Emacs, ag (the
silver searcher) makes rgrep & co. irrelevant and my Dired usage is
quite basic.

>>> ls,
>>
>> Is it necessary?
>
> Not sure. I thought dired used it, at least if it's there but maybe
> not.

If there is no ls available, Emacs uses its own facilities. That applies
to Eshell, Dired, etc.

>>> git,
>>
>> Git for Windows.
>>
>>> aspell
>>
>> See above.
>>
>>> and all the rest by hand?
>>
>> On my case, only Git for Windows and MSYS2 are installed by hand.
>>
>> BTW, Git for Windows is also based on MSYS2: it installs the required
>> pieces that are required for running the parts of git that still depend
>> on POSIX.
>>
>> It theory, instead of Git for Windows I could also run MSYS2's own
>> git.exe setting magit-git-executable and vc-git-program, without adding
>> MSYS2 /bin directory to PATH, but last time I tried it was somewhat
>> slower than Git for Windows, which is slow enough itself. Also, the
>> later comes with some goodies built-in, such as git-svn. MSYS2 git
>> package requires about 30 MB, while Git for Windows 240.
>
> Yes, I hear that magit is difficult to use on Windows.

No, not difficult, just slow.

> I don't know if that is fixable within Emacs, or if it is fundamental
> to Windows.

Launching processes is slower in Windows than in GNU/Linux, plus Git is
heavily optimized for GNU/Linux, plus (and this is not really a "plus"
but a "times") Magit does a lot of git calls, about 50 runs of git
whenever the status buffer is updated, last time I checked.

>> I execute runemacs.exe from wathever/msys64/mingw64/bin. emacs.el sets
>> PATH and exec-path:
>>
>> (defun anade-a-path (path)
>>   (setenv "PATH" (concat (getenv "PATH") path-separator dir))
>>   (setq exec-path (append exec-path (list dir))))
>>
>> (anade-a-path (file-name-directory (car command-line-args)))
>>
>> With this I can execute programs on emacs.exe directory (that is,
>> whatever/msys64/mingw64/bin) with no problems.
>
> That's a good data point! Thank you.

You're welcome.

Re: [feature/internal-msys] thoughts of a more function windows package

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: emacs-devel@gnu.org
> Date: Sun, 24 Jan 2021 22:13:07 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > They mostly work, until they don't.  Like with Cygwin, there are
> >> > subtle incompatibilities, mainly in file names and in communications
> >> > with subprocesses and response to "signals".  Encoding defaults are
> >> > also different.
> >> 
> >> That's true for the msys2 commands but not the mingw64 ones?
> >
> > Yes.  That's why I asked about msys-1.0.dll: the programs that depend
> > on that aren't mingw64 (native) programs.
> 
> But, mingw64 does not have all the packages I need.

Are we talking about you personally, or are we talking about Emacs
users?

If the former, then MinGW64 are not the only source of good ports of
Free Software to Windows, far from that.  If they don't have some
package, you just go out and look for it elsewhere.

If we are talking about Emacs users who will download Emacs from the
GNU sites, then I'd say give them only what MinGW64 provides, so that
they could use pacman to easily update that.  Those who need more will
have to find and install whatever they need on their own.  Telling
them to install MSYS ports instead risks exposing them to subtle
problems, so I wouldn't recommend it.

> How do people use Emacs on windows? I mean, do they install find, ls,
> git, aspell and all the rest by hand?

If MinGW doesn't provide those, what else can you do?  Me, I ported
some of the packages myself (where I found no ports that were good
enough or new for me), and installed others where I found good ports.
Almost all of my ports are available from the ezwinports site.

> >> Would it be easier to have Emacs allow me to successfully update PATH
> >> during run?
> >
> > Update how?
> 
> Using some magic that doesn't exist at the current time. I mean a way
> like did (by fiddling with setenv) that doesn't suffer the problems that
> it causes.

I don't think this magic can exist.  But if someone knows, let them
speak up.

Re: [feature/internal-msys] thoughts of a more function windows package

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Date: Sun, 24 Jan 2021 23:34:29 +0000
> Cc: emacs-devel@gnu.org
> 
> Oh, I'm a find junkie.

Me too.  Which is why I ported Findutils long ago (the available ports
were abysmally slow).  I have an updatedb job running every week until
this day.

> >> ls,
> >
> > Is it necessary?
> 
> Not sure. I thought dired used it, at least if it's there but maybe not.

I do use ls, regardless of Dired (which doesn't need it on Windows).
That's about the only Coreutils program I ported myself; for the rest
I use an old GnuWin32 port, which is quite adequate -- ls was the
single exception.

Re: [feature/internal-msys] thoughts of a more function windows package

[chad]

[-- Attachment #1: Type: text/plain, Size: 313 bytes --]

On Mon, Jan 25, 2021 at 7:24 AM Eli Zaretskii <eliz@gnu.org> wrote:

> > Oh, I'm a find junkie.
>
> Me too.
>

Tangent, but: you might be interested in fd-find, which is an
implementation of  (paraphrasing here) about 80% of find that's usually
several times faster.

   https://github.com/sharkdp/fd

Hth,
~Chad

[-- Attachment #2: Type: text/html, Size: 752 bytes --]

Re: [feature/internal-msys] thoughts of a more function windows package

[Eli Zaretskii]

> From: chad <yandros@gmail.com>
> Date: Mon, 25 Jan 2021 11:49:24 -0800
> Cc: Phillip Lord <phillip.lord@russet.org.uk>, ofv@wanadoo.es, 
> 	EMACS development team <emacs-devel@gnu.org>
> 
> Tangent, but: you might be interested in fd-find, which is an implementation of  (paraphrasing here) about
> 80% of find that's usually several times faster.
>  
>    https://github.com/sharkdp/fd

I know about it, but it isn't a 100% functional replacement of 'find',
and uses a confusingly different mini-language on top of that.

Re: [feature/internal-msys] thoughts of a more function windows package

[Richard Copley]

On Mon, 25 Jan 2021 at 15:21, Eli Zaretskii <eliz@gnu.org> wrote:
> > From: Phillip Lord <phillip.lord@russet.org.uk>
> > Eli Zaretskii <eliz@gnu.org> writes:

> > >> Would it be easier to have Emacs allow me to successfully update PATH
> > >> during run?
> > >
> > > Update how?
> >
> > Using some magic that doesn't exist at the current time. I mean a way
> > like did (by fiddling with setenv) that doesn't suffer the problems that
> > it causes.
>
> I don't think this magic can exist.  But if someone knows, let them
> speak up.

No magic is needed. Just change exec-path and the PATH environment
variable matching values, not forgetting to add Emacs' libexec dir to
exec-path. I have code in my init file to select different paths,
because different toolchains that I want to run in Emacs require
different incompatible paths. It doesn't cause any problems, and
everything behaves as you expect.

This works for me because I install emacs into the /mingw64/bin
directory of MSYS2 (which is what "make install" does by default, in
the MSYS2/MINGW64 shell), so emacs can never fail to find its required
DLLs at runtime, nor find the wrong ones. (The LoadLibrary function
always searches the exe's directory and the System32 directory.)
The installer should make sure all the necessary DLLs are in the same
directory as emacs.exe, too, for the same reason.

When running emacs from the /src directory, features that load DLLs at
runtime don't work properly unless the correct DLLs are found on the
path, but there's nothing mysterious, magical or unexpected about
that.

By the way, the MSYS2 builds of Emacs no longer apply any significant
patches. There's just one, which is intended to disable ImageMagick,
and which is superfluous but harmless. I hope we don't do anything
that breaks the ability to build and install Emacs into an existing
MSYS2 tree using "./autogen.sh && ./configure && make && make
install", as their PKGBUILD script does.

Re: [feature/internal-msys] thoughts of a more function windows package

[Stefan Monnier]

> Tangent, but: you might be interested in fd-find, which is an
> implementation of  (paraphrasing here) about 80% of find that's usually
> several times faster.

Any hint *why* it's faster?


        Stefan

Re: [feature/internal-msys] thoughts of a more function windows package

[Óscar Fuentes]

Richard Copley <rcopley@gmail.com> writes:

[snip]

> By the way, the MSYS2 builds of Emacs no longer apply any significant
> patches. There's just one, which is intended to disable ImageMagick,
> and which is superfluous but harmless.

That patch is not used. I'm not sure why I didn't delete it.

> I hope we don't do anything
> that breaks the ability to build and install Emacs into an existing
> MSYS2 tree using "./autogen.sh && ./configure && make && make
> install", as their PKGBUILD script does.

+1, although I don't expect that this is checked regularly by the Emacs
developers. Currently my Windows machines are using 27.1 as distributed
by MSYS2, but eventually I'll switch to master, building it every few
months, so when 28 is released the build should be fine.

Re: [feature/internal-msys] thoughts of a more function windows package

[chad]

[-- Attachment #1: Type: text/plain, Size: 1903 bytes --]

On Mon, Jan 25, 2021 at 12:42 PM Stefan Monnier <monnier@iro.umontreal.ca>
wrote:

> > Tangent, but: you might be interested in fd-find, which is an
> > implementation of  (paraphrasing here) about 80% of find that's usually
> > several times faster.
>
> Any hint *why* it's faster?
>

At this point I'm just passing along what I've read on the project pages,
but it claims most of the speed comes from leveraging the speed efforts of
ripgrep's component pieces (it's rust-based regex and ignore packages).
There are two relevant chunks from ripgrep's project page; a blog post with
details and benchmarks, and a summary on ripgrep's project page.

Blog post: https://blog.burntsushi.net/ripgrep/

Summarizing, ripgrep is fast because:
>
>    - It is built on top of Rust's regex engine
>    <https://github.com/rust-lang/regex>. Rust's regex engine uses finite
>    automata, SIMD and aggressive literal optimizations to make searching very
>    fast. (PCRE2 support can be opted into with the -P/--pcre2 flag.)
>
>
>    - Rust's regex library maintains performance with full Unicode support
>    by building UTF-8 decoding directly into its deterministic finite automaton
>    engine.
>
>
>    - It supports searching with either memory maps or by searching
>    incrementally with an intermediate buffer. The former is better for single
>    files and the latter is better for large directories. ripgrep chooses the
>    best searching strategy for you automatically.
>
>
>    - Applies your ignore patterns in .gitignore files using a RegexSet
>    <https://docs.rs/regex/1/regex/struct.RegexSet.html>. That means a
>    single file path can be matched against multiple glob patterns
>    simultaneously.
>
>
>    - It uses a lock-free parallel recursive directory iterator, courtesy
>    of crossbeam <https://docs.rs/crossbeam> and ignore
>    <https://docs.rs/ignore>.
>
>
Hope that helps,
~Chad

[-- Attachment #2: Type: text/html, Size: 5487 bytes --]

Re: [feature/internal-msys] thoughts of a more function windows package

[Dmitry Gutov]

On 25.01.2021 22:42, Stefan Monnier wrote:
>> Tangent, but: you might be interested in fd-find, which is an
>> implementation of  (paraphrasing here) about 80% of find that's usually
>> several times faster.
> Any hint*why*  it's faster?

In my benchmarks, it isn't. But there is at least one report with 
numbers that says otherwise (on your bug mailing list).

Re: [feature/internal-msys] thoughts of a more function windows package

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Date: Mon, 25 Jan 2021 15:42:13 -0500
> Cc: ofv@wanadoo.es, Eli Zaretskii <eliz@gnu.org>,
>  Phillip Lord <phillip.lord@russet.org.uk>,
>  EMACS development team <emacs-devel@gnu.org>
> 
> > Tangent, but: you might be interested in fd-find, which is an
> > implementation of  (paraphrasing here) about 80% of find that's usually
> > several times faster.
> 
> Any hint *why* it's faster?

It uses several threads, and uses PCRE instead of fnmatch.  The latter
is a subtle difference from what Find does, and it explains the
different results of the search.

Re: [feature/internal-msys] thoughts of a more function windows package

[Eli Zaretskii]

> From: Óscar Fuentes <ofv@wanadoo.es>
> Date: Mon, 25 Jan 2021 22:17:15 +0100
> 
> > I hope we don't do anything
> > that breaks the ability to build and install Emacs into an existing
> > MSYS2 tree using "./autogen.sh && ./configure && make && make
> > install", as their PKGBUILD script does.
> 
> +1, although I don't expect that this is checked regularly by the Emacs
> developers.

??? How else do you think I build my Emacs?

Re: [feature/internal-msys] thoughts of a more function windows package

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

>> > I hope we don't do anything
>> > that breaks the ability to build and install Emacs into an existing
>> > MSYS2 tree using "./autogen.sh && ./configure && make && make
>> > install", as their PKGBUILD script does.
>> 
>> +1, although I don't expect that this is checked regularly by the Emacs
>> developers.
>
> ??? How else do you think I build my Emacs?

I thought you used MSYS, not MSYS2.

Do you also use the MinGW-w64 libraries provided by MSYS2?

Re: [feature/internal-msys] thoughts of a more function windows package

[Eli Zaretskii]

On January 26, 2021 7:43:23 AM GMT+02:00, "Óscar Fuentes" <ofv@wanadoo.es> wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> >> > I hope we don't do anything
> >> > that breaks the ability to build and install Emacs into an
> existing
> >> > MSYS2 tree using "./autogen.sh && ./configure && make && make
> >> > install", as their PKGBUILD script does.
> >> 
> >> +1, although I don't expect that this is checked regularly by the
> Emacs
> >> developers.
> >
> > ??? How else do you think I build my Emacs?
> 
> I thought you used MSYS, not MSYS2.

That's right, I do use MSYS.  But how is that relevant to breaking the build procedure?  What am I missing?

> Do you also use the MinGW-w64 libraries provided by MSYS2?

No.

Re: [feature/internal-msys] thoughts of a more function windows package

[Óscar Fuentes]

Eli Zaretskii <eliz@gnu.org> writes:

> On January 26, 2021 7:43:23 AM GMT+02:00, "Óscar Fuentes" <ofv@wanadoo.es> wrote:
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> > I hope we don't do anything
>> >> > that breaks the ability to build and install Emacs into an
>> existing
>> >> > MSYS2 tree using "./autogen.sh && ./configure && make && make
>> >> > install", as their PKGBUILD script does.
>> >> 
>> >> +1, although I don't expect that this is checked regularly by the
>> Emacs
>> >> developers.
>> >
>> > ??? How else do you think I build my Emacs?
>> 
>> I thought you used MSYS, not MSYS2.
>
> That's right, I do use MSYS.

That's why I wrote "although I don't expect that this [building on
MSYS2] is checked regularly by the Emacs developers."

> But how is that relevant to breaking the build procedure? What am I
> missing?

Whatever works in MSYS does not necessarily works on MSYS2.

Re: [feature/internal-msys] thoughts of a more function windows package

[Eli Zaretskii]

On January 26, 2021 9:37:48 AM GMT+02:00, "Óscar Fuentes" <ofv@wanadoo.es> wrote:
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > That's right, I do use MSYS.
> 
> That's why I wrote "although I don't expect that this [building on
> MSYS2] is checked regularly by the Emacs developers."

But everybody else who builds on Windows here uses MSYS2, so I still don't understand the problem you had in mind.

> Whatever works in MSYS does not necessarily works on MSYS2.

Do you mean MSYS2 or do you mean MinGW64?  The former is supposed to work like MSYS, for the latter I always check in their sources to make sure nothing would break.  And again, others build with those tools and report (rare) problems when they happen.  And if people stop using our build procedure with MinGW64, there will be no reason to worry about it not working, right?

Re: [feature/internal-msys] thoughts of a more function windows package

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Cc: emacs-devel@gnu.org
>> Date: Sun, 24 Jan 2021 22:13:07 +0000
>> 
>> Eli Zaretskii <eliz@gnu.org> writes:
>> 
>> >> > They mostly work, until they don't.  Like with Cygwin, there are
>> >> > subtle incompatibilities, mainly in file names and in communications
>> >> > with subprocesses and response to "signals".  Encoding defaults are
>> >> > also different.
>> >> 
>> >> That's true for the msys2 commands but not the mingw64 ones?
>> >
>> > Yes.  That's why I asked about msys-1.0.dll: the programs that depend
>> > on that aren't mingw64 (native) programs.
>> 
>> But, mingw64 does not have all the packages I need.
>
> Are we talking about you personally, or are we talking about Emacs
> users?

Emacs users in general. I don't use Emacs for Windows myself, at least
not in any routine way.


> If the former, then MinGW64 are not the only source of good ports of
> Free Software to Windows, far from that.  If they don't have some
> package, you just go out and look for it elsewhere.

I am sure that this is true. But I have to scrape together tools from
several different locations, particularly if I have to host it (and thus
have the responsibilty of getting all the source), it's not going to
happen.

I might be able to use a different package manager (like chocolaty); I'd
rather use msys2 because I already use it for my build machine.

> If we are talking about Emacs users who will download Emacs from the
> GNU sites, then I'd say give them only what MinGW64 provides, so that
> they could use pacman to easily update that.  Those who need more will
> have to find and install whatever they need on their own.  Telling
> them to install MSYS ports instead risks exposing them to subtle
> problems, so I wouldn't recommend it.

Okay. In which case I'll only add the mingw64 to the path. This excludes
git which was one of my main aims to be honest. But at least we can add
a spell checker to Emacs.



>
>> How do people use Emacs on windows? I mean, do they install find, ls,
>> git, aspell and all the rest by hand?
>
> If MinGW doesn't provide those, what else can you do?  Me, I ported
> some of the packages myself (where I found no ports that were good
> enough or new for me), and installed others where I found good ports.
> Almost all of my ports are available from the ezwinports site.

Give up on these tools till a better solution appears.


>> >> Would it be easier to have Emacs allow me to successfully update PATH
>> >> during run?
>> >
>> > Update how?
>> 
>> Using some magic that doesn't exist at the current time. I mean a way
>> like did (by fiddling with setenv) that doesn't suffer the problems that
>> it causes.
>
> I don't think this magic can exist.  But if someone knows, let them
> speak up.

I don't know what the problems are that it causes, but again, if it's
not simple and we cannot provide a 90/10 solution for changing path
within a running Emacs, then I will try something that doesn't need
this.

Phil

Re: [feature/internal-msys] thoughts of a more function windows package

[martin rudalics]

 > But everybody else who builds on Windows here uses MSYS2, so I still
 > don't understand the problem you had in mind.

Not everybody else.

martin

Re: [feature/internal-msys] thoughts of a more function windows package

[Stephen Leake]

Richard Copley <rcopley@gmail.com> writes:

> On Mon, 25 Jan 2021 at 15:21, Eli Zaretskii <eliz@gnu.org> wrote:
>> > From: Phillip Lord <phillip.lord@russet.org.uk>
>> > Eli Zaretskii <eliz@gnu.org> writes:
>
>> > >> Would it be easier to have Emacs allow me to successfully update PATH
>> > >> during run?
>> > >
>> > > Update how?
>> >
>> > Using some magic that doesn't exist at the current time. I mean a way
>> > like did (by fiddling with setenv) that doesn't suffer the problems that
>> > it causes.
>>
>> I don't think this magic can exist.  But if someone knows, let them
>> speak up.
>
> No magic is needed. Just change exec-path and the PATH environment
> variable matching values, not forgetting to add Emacs' libexec dir to
> exec-path. I have code in my init file to select different paths,
> because different toolchains that I want to run in Emacs require
> different incompatible paths. It doesn't cause any problems, and
> everything behaves as you expect.

I do the same; I sometimes change PATH after init when I change
projects.

-- 
-- Stephe

Re: [feature/internal-msys] thoughts of a more function windows package

[Stephen Leake]

Óscar Fuentes <ofv@wanadoo.es> writes:

> That's why I wrote "although I don't expect that this [building on
> MSYS2] is checked regularly by the Emacs developers."

For what it's worth, I build on MSYS2.

I only build once a month or so, and I don't install.

-- 
-- Stephe

Re: [feature/internal-msys] thoughts of a more function windows package

[Eli Zaretskii]

> From: Stephen Leake <stephen_leake@stephe-leake.org>
> Date: Wed, 27 Jan 2021 06:55:10 -0800
> Cc: emacs-devel@gnu.org
> 
> Óscar Fuentes <ofv@wanadoo.es> writes:
> 
> > That's why I wrote "although I don't expect that this [building on
> > MSYS2] is checked regularly by the Emacs developers."
> 
> For what it's worth, I build on MSYS2.

I wouldn't expect the differences between MSYS and MSYS2 to be of any
importance here.  It's the same Bash and the same Coreutils.  And the
commands we use during the build procedure are very basic.

Re: [feature/internal-msys] thoughts of a more function windows package

[Nikolay Kudryavtsev]

Sorry for late reply, but I think that while bundling msys2 is a good 
idea in theory, in practice it would turn into a complete nightmare. The 
problem, just like with bundling any third party components is that you 
have to maintain them. If you bundle the latest and shiniest msys2, you 
can never be sure if it's really properly working for our use cases. And 
if you bundle some pretested version, you run into "hey, please fix bug 
A, the upstream has already fixed it", but we can't switch to upstream 
due to bug B.

Msys2 sort of missed the boat in that they took pacman, but didn't 
bother with making their own AUR.

Re: [feature/internal-msys] thoughts of a more function windows package

[Phillip Lord]

Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com> writes:

> Sorry for late reply, but I think that while bundling msys2 is a good
> idea in theory, in practice it would turn into a complete
> nightmare. The problem, just like with bundling any third party
> components is that you have to maintain them. If you bundle the latest
> and shiniest msys2, you can never be sure if it's really properly
> working for our use cases. And if you bundle some pretested version,
> you run into "hey, please fix bug A, the upstream has already fixed
> it", but we can't switch to upstream due to bug B.
>
> Msys2 sort of missed the boat in that they took pacman, but didn't
> bother with making their own AUR.


The work on this has stalled at the moment anyway, but my plan would be
to work out how to link an Emacs installation to an Msys2 installation,
by setting up paths correctly, as well as defining a set of packages
that are useful for Emacs.

There is always the risk that msys and Emacs work inconsistently since,
with this scenario, msys could update at any point that breaks things. I
don't think that there is any solution to this than to say that the last
release version of Emacs will work with a version of msys which is about
current at the time of release. What else can we do? In the case, that
basic Emacs functionality fails, people could always fall back to the
fully bundled Emacs with DLLs that is currently available.

The current situation where Emacs without msys2 lacks basic capabilities
such as git handling which many other editors have bundled is also
problematic!

Phil

Re: [feature/internal-msys] thoughts of a more function windows package

[Wayne Harris via Emacs development discussions.]

Phillip Lord <phillip.lord@russet.org.uk> writes:

> Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com> writes:
>
>> Sorry for late reply, but I think that while bundling msys2 is a good
>> idea in theory, in practice it would turn into a complete
>> nightmare. The problem, just like with bundling any third party
>> components is that you have to maintain them. If you bundle the latest
>> and shiniest msys2, you can never be sure if it's really properly
>> working for our use cases. And if you bundle some pretested version,
>> you run into "hey, please fix bug A, the upstream has already fixed
>> it", but we can't switch to upstream due to bug B.
>>
>> Msys2 sort of missed the boat in that they took pacman, but didn't
>> bother with making their own AUR.
>
> The work on this has stalled at the moment anyway, but my plan would be
> to work out how to link an Emacs installation to an Msys2 installation,
> by setting up paths correctly, as well as defining a set of packages
> that are useful for Emacs.
>
> There is always the risk that msys and Emacs work inconsistently since,
> with this scenario, msys could update at any point that breaks things. I
> don't think that there is any solution to this than to say that the last
> release version of Emacs will work with a version of msys which is about
> current at the time of release. What else can we do? In the case, that
> basic Emacs functionality fails, people could always fall back to the
> fully bundled Emacs with DLLs that is currently available.
>
> The current situation where Emacs without msys2 lacks basic capabilities
> such as git handling which many other editors have bundled is also
> problematic!

(*) Introduction

I'll share my opinion as a user of the GNU Emacs and Windows.  I'll try
to summarize my context to help you decide whether it's worth reading
the entire message.  Maybe the context of this thread is a bit off of
mine, so perhaps the difficulties I mention don't quite apply to what
Phillip Lord is considering.

(*) Summary

Unless this coupling Emacs-MSYS2 can be well done with a certain
long-run guarantee, I'd still prefer put them together with my own
hands, because this way I can guarantee the behavior of the programs I
expect to get.  So I think I'd need an assurance of always having a
certain exact (verified with a hash sum) version of msys2.

I guarantee this today myself by zipping my GNU Emacs directory and
moving it to another system when something happens that I need to move,
say when my computer crashes and I need to install another Windows.  My
guarantee is based on Windows' guarantee of backward compatibility and
behavior stability, which is why I have to run Windows.

(*) I manually put msys2 inside Emacs

--8<---------------cut here---------------start------------->8---
%pwd
c:/sys/emacs/usr/mingw

%file msys2.exe
msys2.exe: PE32+ executable (GUI) x86-64 (stripped to external PDB), for MS Windows
--8<---------------cut here---------------end--------------->8---

Having discovered msys2 recently I immediately convinced myself to
reorganize my GNU Emacs to use msys2.  (Until then I was using a 32-bit
GNU Emacs and installed my own selections of Eli Zaretskii's ezwinports
--- they have been useful to me for various years. Thank you!)  I now
consider msys2 part of my GNU Emacs.

(*) Tangent 1: Windows was my last resort

Running Windows is not something that pleases me.  I'd much rather run
WindowMaker, say, to manage my windows, but I feel I'm stuck with
Windows because GNU systems don't help me with keeping all I need in a
single directory with an assurance that I can move that directory to a
similar system and still have everything work.

This is not a compliment to Windows.  If GNU systems did not come with
all of their variety and openess and freedom, I'm sure they could
provide the same status quo provided by Windows.

As I understand it, the core of issue here is what Torvalds calls the
``binary issue'' while answering these questions:

  https://youtu.be/5PmHRSeA2c8?t=1722
  https://youtu.be/5PmHRSeA2c8?t=2473
  
So, I'm lucky in that the programs I need are all well-designed and can
be easily ported from one system to another without requiring
reinstallations and reconfigurations, otherwise Windows would also be a
non-solution.  For instance, if Chrome or Word were software I need,
then Windows would be a non-solution.

(*) Tangent 2: I need long-term assurance

As an example, I've built my own OpenBSD distribution because I wanted
an assurance in the behavior of the system, besides a quick
installation.  I install it with a single command line and it asks no
questions.  It comes ready to do all the things *I* usually do.  The
source code of everything is exactly where I expect it to be and so on.
It allows me to quickly do all the experiments I do with a minimum of
technological struggle.  (Let me know if anyone would like a copy of
such thing, by the way.  It's OpenBSD 4.5, source code included.  Sadly,
I didn't include the GNU Emacs, though my notes say that was the very
next step.)

Re: [feature/internal-msys] thoughts of a more function windows package

[Phillip Lord]

Wayne Harris via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
>> Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com> writes:
>>
>>> Sorry for late reply, but I think that while bundling msys2 is a good
>>> idea in theory, in practice it would turn into a complete
>>> nightmare. The problem, just like with bundling any third party
>>> components is that you have to maintain them. If you bundle the latest
>>> and shiniest msys2, you can never be sure if it's really properly
>>> working for our use cases. And if you bundle some pretested version,
>>> you run into "hey, please fix bug A, the upstream has already fixed
>>> it", but we can't switch to upstream due to bug B.

>> There is always the risk that msys and Emacs work inconsistently since,
>> with this scenario, msys could update at any point that breaks things. I
>> don't think that there is any solution to this than to say that the last
>> release version of Emacs will work with a version of msys which is about
>> current at the time of release. What else can we do? In the case, that
>> basic Emacs functionality fails, people could always fall back to the
>> fully bundled Emacs with DLLs that is currently available.
>>
>> The current situation where Emacs without msys2 lacks basic capabilities
>> such as git handling which many other editors have bundled is also
>> problematic!
>
> (*) Introduction
>
> I'll share my opinion as a user of the GNU Emacs and Windows.  I'll try
> to summarize my context to help you decide whether it's worth reading
> the entire message.  Maybe the context of this thread is a bit off of
> mine, so perhaps the difficulties I mention don't quite apply to what
> Phillip Lord is considering.

Always useful, to get feedback since I rarely use Emacs on windows
myself.


> (*) Summary
>
> Unless this coupling Emacs-MSYS2 can be well done with a certain
> long-run guarantee, I'd still prefer put them together with my own
> hands, because this way I can guarantee the behavior of the programs I
> expect to get.  So I think I'd need an assurance of always having a
> certain exact (verified with a hash sum) version of msys2.
>

I can certainly appreciate that. I sit somewhere in the middle: I use
Emacs build from master, but most of my emacs packages are specific
versions, rather than running from everyones heads.

But, like it or lump it, msys2 doesn't do that. They have a versioned,
hash summed installer, but after that it just updates to the latest
version, with no specific release pattern (or a rolling release if you
prefer). I want to try and avoid replicating what msys2 already does
which, conversely, means I can only do what msys2 does.


> As an example, I've built my own OpenBSD distribution because I wanted
> an assurance in the behavior of the system, besides a quick
> installation.  I install it with a single command line and it asks no
> questions.  It comes ready to do all the things *I* usually do.

Something close to this, I think we could achieve. Install Emacs, have
it ask "do you want to link to an msys2 installation? Do you want to
install it? Do you want to update it with Emacs standard packages".

So three questions, but not none.

Phil

Re: [feature/internal-msys] thoughts of a more function windows package

[Nikolay Kudryavtsev]

You technically can make your own msys distribution, by choosing a base 
package as old as you want and then installing extras from the package 
archives manually downloaded from msys2 binary cache.

But yeah, I don't think anyone would force anyone to use the 
msys2-bundled installation, I just wanted to note that maintaining such 
distribution seems like too much work for the benefit it provides.

In the ideal world we'll have something like a bundled configure-like 
package that would test for those POSIX utilities and then show what 
you're missing and maybe install them for you.

Re: [feature/internal-msys] thoughts of a more function windows package

[Nikolay Kudryavtsev]

You might be interested in GNU Guix(or Nixos), since it provides exactly 
that ability to write one configuration script that you can later use to 
completely setup a fresh machine.

Re: [feature/internal-msys] thoughts of a more function windows package

[Óscar Fuentes]

Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com> writes:

> You might be interested in GNU Guix(or Nixos), since it provides
> exactly that ability to write one configuration script that you can
> later use to completely setup a fresh machine.

Since we are discussing Windows support, those tools are irrelevant,
aren't they?

Re: [feature/internal-msys] thoughts of a more function windows package

[Wayne Harris via Emacs development discussions.]

Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com> writes:

> You might be interested in GNU Guix(or Nixos), since it provides
> exactly that ability to write one configuration script that you can
> later use to completely setup a fresh machine.

Thank you for sharing that!  I just read the introduction of GNU Guix.
What a marvelous work!  I always wondered if things could be done this
way.  The fact no tool ever was made me think it would be too difficult
to do.  The fact it's written in Guile pleases me even more.

Re: [feature/internal-msys] thoughts of a more function windows package

[Nikolay Kudryavtsev]

I was just mentioning them for Wayne, since his frustration with the 
POSIX way of distributing software is very familiar to me.

Re: [feature/internal-msys] thoughts of a more function windows package

[Wayne Harris via Emacs development discussions.]

Phillip Lord <phillip.lord@russet.org.uk> writes:

[...]

>> (*) Summary
>>
>> Unless this coupling Emacs-MSYS2 can be well done with a certain
>> long-run guarantee, I'd still prefer put them together with my own
>> hands, because this way I can guarantee the behavior of the programs I
>> expect to get.  So I think I'd need an assurance of always having a
>> certain exact (verified with a hash sum) version of msys2.
>
> I can certainly appreciate that. I sit somewhere in the middle: I use
> Emacs build from master, but most of my emacs packages are specific
> versions, rather than running from everyones heads.

How do you do that?  It seems you take the .exe but keep .el separate?
That sounds crazy to me.  How would I know all my new .exe still works
with my very old .el files?

> But, like it or lump it, msys2 doesn't do that. They have a versioned,
> hash summed installer, but after that it just updates to the latest
> version, with no specific release pattern (or a rolling release if you
> prefer). [...]

That's indeed a big problem for people who do work dependent on computer
tools --- probably anyone who does any serious work on computers.  I
also see many non-professional users, say, who just hate the surprise of
updates.  (They might have all the time in the world to learn new
behaviors, but they just hate it.  Windows forums are full of angry
customers.)

That's why I loved the Guix-news posted by Nikolay Kudryavtsev.  That's
what I like about computer systems --- repetitive behavior with
mathematical guarantee.  It's incredible that people tolerate computers
with human-like behavior, ``mood-dependent'' say.

>> As an example, I've built my own OpenBSD distribution because I wanted
>> an assurance in the behavior of the system, besides a quick
>> installation.  I install it with a single command line and it asks no
>> questions.  It comes ready to do all the things *I* usually do.
>
> Something close to this, I think we could achieve. Install Emacs, have
> it ask "do you want to link to an msys2 installation? Do you want to
> install it? Do you want to update it with Emacs standard packages".
>
> So three questions, but not none.

That sounds nice.  If I had to start my ``GNU Emacs system'' from
scratch, this step would be a nice one.  The GNU Emacs on Windows is
handicapped without lots of other UNIX programs.  I wrote this years
ago:

--8<---------------cut here---------------start------------->8---
;; (*) On being in the wild, without a map

;; When you're on Windows, you're on your own.  You have to provide
;; everything, that is, you need to bring the UNIX part that we can
;; replicate on Windows.  (Thankfully, the EMACS shell replaces the UNIX
;; shell, even on Windows.)

;; Use this GNU debugger
(when (string= system-type "windows-nt")
    (setq gdb-command-name "~/mingw/usr/bin/gdb.exe"))

[...]

;; Use this diff program
(when (string= system-type "windows-nt")
  (setq ediff-diff-program "~/mingw/usr/bin/diff.exe"))

[...]
--8<---------------cut here---------------end--------------->8---

Re: [feature/internal-msys] thoughts of a more function windows package

[Phillip Lord]

Wayne Harris via "Emacs development discussions." <emacs-devel@gnu.org>
writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>
> [...]
>
>> I can certainly appreciate that. I sit somewhere in the middle: I use
>> Emacs build from master, but most of my emacs packages are specific
>> versions, rather than running from everyones heads.
>
> How do you do that?  It seems you take the .exe but keep .el separate?
> That sounds crazy to me.  How would I know all my new .exe still works
> with my very old .el files?

Oh, sorry, I didn't describe it well. I use Emacs and all the lisp files
in the main repo. My external packages mostly come from melpa-stable and
elpa which are tagged versions. I sync my computers with unison and keep
the packages installed using use-package; the versions formally aren't
linked between machines, but tools like straight.el would do that for
you.


>> But, like it or lump it, msys2 doesn't do that. They have a versioned,
>> hash summed installer, but after that it just updates to the latest
>> version, with no specific release pattern (or a rolling release if you
>> prefer). [...]
>
> That's indeed a big problem for people who do work dependent on computer
> tools --- probably anyone who does any serious work on computers.  I
> also see many non-professional users, say, who just hate the surprise of
> updates.  (They might have all the time in the world to learn new
> behaviors, but they just hate it.  Windows forums are full of angry
> customers.)

Perhaps. Clearly msys2 did it for a reasons, and likewise a lot of Emacs
users install from melpa which mostly runs on a dirty head or rolling
release if you prefer. It always sounded like a nightmare to me, but it
seems to work for many people.


>
> That's why I loved the Guix-news posted by Nikolay Kudryavtsev.  That's
> what I like about computer systems --- repetitive behavior with
> mathematical guarantee.  It's incredible that people tolerate computers
> with human-like behavior, ``mood-dependent'' say.
>
>>> As an example, I've built my own OpenBSD distribution because I wanted
>>> an assurance in the behavior of the system, besides a quick
>>> installation.  I install it with a single command line and it asks no
>>> questions.  It comes ready to do all the things *I* usually do.
>>
>> Something close to this, I think we could achieve. Install Emacs, have
>> it ask "do you want to link to an msys2 installation? Do you want to
>> install it? Do you want to update it with Emacs standard packages".
>>
>> So three questions, but not none.
>
> That sounds nice.  If I had to start my ``GNU Emacs system'' from
> scratch, this step would be a nice one.  The GNU Emacs on Windows is
> handicapped without lots of other UNIX programs.


Yes, I agree.

Phil