Bloat in Windows (revisited)

Bloat in Windows (revisited)

[Phillip Lord]

A while back, we had some discussion about the Emacs packages for
windows.

https://lists.gnu.org/archive/html/emacs-devel/2019-04/msg00719.html

I have now implemented the first part of this (no debug symbols in
release builds); at least hopefully, as it's hard to test for sure till
the first release of Emacs-27.

I have also been playing around with the problem of msys2 pulling in a
fairly wide set of dependencies. Instead of using `pactree` to determine
the dependency tree, I use `pacman` directly and crawl the tree myself.

This has a slightly increased maintainence load (there are two packages
with different names in different places for some reason). But it means
that I can exclude any subtree I like. Currently:

mingw-w64-x86_64-librsvg

brings in

mingw-w64-x86_64-glib2

which brings in lots of stuff including python. Excluding this results
in a much smaller dependencies package (around 40Mb as opposed to
140Mb). Haven't build the final files yet, but I would guess that this
would bring the installer version down to 50Mb -- not quite "eight Mb and
constantly swapping", but pretty small.

This also opens up the question of whether we *should* include any
executables that would add value to Emacs on Windows; I hesitate to go
too far down this route, mostly because maintaining an mini MSYS2
distribution seems the wrong thing to do.

But some basic functionality that other editors get from the OS comes
from external packages. I am thinking gzip, find, grep and aspell as
essential. Things like latex and python, I would exclude. Then there is
some middle ground -- I use msmtp for sending email, for instance.

The branch is here. Comments welcome.

http://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/smaller-windows

Phil

Re: Bloat in Windows (revisited)

[Tak Kunihiro]

> But some basic functionality that other editors get from the OS comes
> from external packages. I am thinking gzip, find, grep and aspell as
> essential. Things like latex and python, I would exclude.

Thank you for working on this.

1. I think it is a very good idea.
2. Emacs may find Window's "find.exe" instead of the "find.exe", without
   configuration.  Put instruction on (info "(efaq-w32) Top")?

Re: Bloat in Windows (revisited)

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Wed, 05 Jun 2019 22:46:11 +0100
> 
> This also opens up the question of whether we *should* include any
> executables that would add value to Emacs on Windows; I hesitate to go
> too far down this route, mostly because maintaining an mini MSYS2
> distribution seems the wrong thing to do.
> 
> But some basic functionality that other editors get from the OS comes
> from external packages. I am thinking gzip, find, grep and aspell as
> essential.

Beware: you are on a slippery slope.  What about Diff, what about
'man', what about bzip2 and lzip and xz?  And if you provide a speller
(I suggest Hunspell, btw, not aspell), then what about dictionaries?
Etc. etc.

Re: Bloat in Windows (revisited)

[Stefan Monnier]

> Beware: you are on a slippery slope.  What about Diff, what about
> 'man', what about bzip2 and lzip and xz?  And if you provide a speller
> (I suggest Hunspell, btw, not aspell), then what about dictionaries?
> Etc. etc.

Agreed.  What might make sense is to catch the error signaled when an
executable is not found and turn it into a clear message pointing the
user to simple instructions for how to install those tools via MSYS2
(or whichever other way).


        Stefan

Re: Bloat in Windows (revisited)

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> Beware: you are on a slippery slope.  What about Diff, what about
>> 'man', what about bzip2 and lzip and xz?  And if you provide a speller
>> (I suggest Hunspell, btw, not aspell), then what about dictionaries?
>> Etc. etc.
>
> Agreed.  What might make sense is to catch the error signaled when an
> executable is not found and turn it into a clear message pointing the
> user to simple instructions for how to install those tools via MSYS2
> (or whichever other way).


Absolutely, this is a slippery slope, but then to break the metaphor,
you need to the view from the top of a slope is always better.

I think many editors would have "integrated" find tools even if this is
just a call out to the OS GUI tool for doing the same. Likewise, spell
checker. A quick "grep" for executable-find gives 300 hits. Many are
languages (i.e. in progmodes), many are text processing commands or
viewers. After that, we do not have that large a list.

I don't think that there are simple instructions on how to install these
tools, other than "re-install this version of Emacs over msys2 (or just
install Emacs via msys2) then add the packages with pacman". This is why
I am asking the question.

On the other question, does excluding mingw-w64-x86_64-glib2 and subtree
seem sensible? If so, I will add this master.

Phil

Re: Bloat in Windows (revisited)

[Stefan Monnier]

> I think many editors would have "integrated" find tools even if this is
> just a call out to the OS GUI tool for doing the same. Likewise, spell
> checker.

Yes, I agree with the general desire.  I just proposed a hypothetical
way to make the slope less slippery by reducing the difference between
the tools that are bundled and those that aren't.

> A quick "grep" for executable-find gives 300 hits.  Many are
> languages (i.e. in progmodes), many are text processing commands or
> viewers. After that, we do not have that large a list.

[ Not sure what is the relation between executable-find and this
  discussion.  I was thinking about catching the error you get from
  start/call-process and then looking up some ad-hoc list of
  known commands and their corresponding MSYS2 package.  ]

> I don't think that there are simple instructions on how to install these
> tools, other than "re-install this version of Emacs over msys2 (or just
> install Emacs via msys2) then add the packages with pacman". This is why
> I am asking the question.

I'm too far from Windows to know anything about that.
So maybe indeed there's no good solution here.

I was hoping we could write some Elisp that can download&install
a minimal MSYS2, and another that can ask MSYS2 to install some tool.

> On the other question, does excluding mingw-w64-x86_64-glib2 and subtree
> seem sensible?

I'm too far from Windows to know anything about that ;-)


        Stefan

Re: Bloat in Windows (revisited)

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Thu, 06 Jun 2019 21:14:48 +0100
> Cc: emacs-devel@gnu.org
> 
> On the other question, does excluding mingw-w64-x86_64-glib2 and subtree
> seem sensible?

Yes.

Re: Bloat in Windows (revisited)

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> I think many editors would have "integrated" find tools even if this is
>> just a call out to the OS GUI tool for doing the same. Likewise, spell
>> checker.
>
> Yes, I agree with the general desire.  I just proposed a hypothetical
> way to make the slope less slippery by reducing the difference between
> the tools that are bundled and those that aren't.
>
>> A quick "grep" for executable-find gives 300 hits.  Many are
>> languages (i.e. in progmodes), many are text processing commands or
>> viewers. After that, we do not have that large a list.
>
> [ Not sure what is the relation between executable-find and this
>   discussion.  I was thinking about catching the error you get from
>   start/call-process and then looking up some ad-hoc list of
>   known commands and their corresponding MSYS2 package.  ]

Oh, "executable-find" is just a cheap and cheerful way of giving some
idea of what executables Emacs might well look for.


>> I don't think that there are simple instructions on how to install these
>> tools, other than "re-install this version of Emacs over msys2 (or just
>> install Emacs via msys2) then add the packages with pacman". This is why
>> I am asking the question.
>
> I'm too far from Windows to know anything about that.
> So maybe indeed there's no good solution here.
>
> I was hoping we could write some Elisp that can download&install
> a minimal MSYS2, and another that can ask MSYS2 to install some tool.


I think this would be effectively rewriting pacman in elisp which seems
like a bad idea. Pre-bundling seems the only sensible solution. People
who want to install new dependencies already know how to do this, I
think -- which is install the no-deps Emacs over an existing MSYS.

>> On the other question, does excluding mingw-w64-x86_64-glib2 and subtree
>> seem sensible?
>
> I'm too far from Windows to know anything about that ;-)

Me too, alas.

Re: Bloat in Windows (revisited)

[Nikolay Kudryavtsev]

[-- Attachment #1: Type: text/plain, Size: 693 bytes --]

Hello, Phillip.

I was wondering about this bloat too. So the gist of my question is - do 
we have to bundle dependencies with their optional stuff due to some 
reason, like licensing requirements? For example do we really have to 
bundle everything in Xpm-nox library distribution, or just the dll 
itself would do, since for running we only need the dll?

Also, some time ago with manual testing I figured lists of dlls we need 
get fully-fledged Emacs running for each toolchain, to use in a script. 
See here 
<https://github.com/sg2002/ms-windows-builder.el/blob/master/ms-windows-builder-config.el#L272>. 
Maybe it's gonna help you in your quest.

-- 
Best Regards,
Nikolay Kudryavtsev


[-- Attachment #2: Type: text/html, Size: 1012 bytes --]

Re: Bloat in Windows (revisited)

[Nikolay Kudryavtsev]

[-- Attachment #1: Type: text/plain, Size: 404 bytes --]

> I was hoping we could write some Elisp that can download&install
> a minimal MSYS2, and another that can ask MSYS2 to install some tool.
I actually implemented this here 
<https://github.com/sg2002/ms-windows-builder.el>. The problem is that 
we need wget(or equivalent) at the very least. There are probably ways 
to do with powershell or bitsadmin, but ugh...

-- 
Best Regards,
Nikolay Kudryavtsev


[-- Attachment #2: Type: text/html, Size: 792 bytes --]

Re: Bloat in Windows (revisited)

[Óscar Fuentes]

Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com> writes:

>> I was hoping we could write some Elisp that can download&install
>> a minimal MSYS2, and another that can ask MSYS2 to install some tool.
> I actually implemented this here
> <https://github.com/sg2002/ms-windows-builder.el>. The problem is that
> we need wget(or equivalent) at the very least. There are probably ways
> to do with powershell or bitsadmin, but ugh...

Since your script is an Elisp program, can't you use Emacs itself to
download those files?

Re: Bloat in Windows (revisited)

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Date: Mon, 10 Jun 2019 11:52:55 +0300
> 
> I was wondering about this bloat too. So the gist of my question is - do we have to bundle dependencies with
> their optional stuff due to some reason, like licensing requirements?

No, the only reason is that "Someone" has to figure out which parts
are necessary for Emacs and which aren't.  Assuming, that is, that no
one will use these binary distributions for any purpose other than
running Emacs.

Re: Bloat in Windows (revisited)

[Nikolay Kudryavtsev]

Yeah, you're right, we can use Emacs as a download manager too, I don't 
think I even seriously thought of doing that. Probably gonna implement 
it there, at least as an option. Would make it less async, but one small 
step closer to making "s" in Emacs stand for "self-replicating".

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Bloat in Windows (revisited)

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
>> Date: Mon, 10 Jun 2019 11:52:55 +0300
>> 
>> I was wondering about this bloat too. So the gist of my question is - do we have to bundle dependencies with
>> their optional stuff due to some reason, like licensing requirements?
>
> No, the only reason is that "Someone" has to figure out which parts
> are necessary for Emacs and which aren't.  Assuming, that is, that no
> one will use these binary distributions for any purpose other than
> running Emacs.


And, unfortunately, in the absence of a test set which tests the compile
time functionality of Emacs, it's hard to do this. I don't use Emacs on
Windows myself, so am left with asking users.

An example here is libwinpthreads which was or is required for operation
of libXpm, at least the DLLs from msys2 (which we do not use for the "no
dependencies" download of Emacs).

In answer to the question about licensing, identifying exactly what
source is required to fulfil GPL is also a little hard, so having too
big a dependency list here strikes me as a reasonably solution.

At this point, the debug snapshot release installer is 50Mb. The
non-debug stripped version should be smaller still. Both of these are
smaller than the source tarball for Emacs-24.5. At this point, the cost
benefit to a smaller distribution size still seems a little moot.

Phil

Re: Bloat in Windows (revisited)

[Phillip Lord]

Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com> writes:

>> I was hoping we could write some Elisp that can download&install
>> a minimal MSYS2, and another that can ask MSYS2 to install some tool.
> I actually implemented this here
> <https://github.com/sg2002/ms-windows-builder.el>. The problem is that
> we need wget(or equivalent) at the very least. There are probably ways
> to do with powershell or bitsadmin, but ugh...


Or python which is how I do it.

I did see this file a while back. It doesn't bootstrap though -- I'd
need an installation of Emacs to build Emacs which I don't always
have.

And I don't know if it builds source, which I need for the Emacs
distribution. It was actually this that took the time to make a with
dependencies release of Emacs. The logic for creating the binaries I
just copied from Chris Zheng's distribution.

Phil

Re: Bloat in Windows (revisited)

[Nikolay Kudryavtsev]

Yes, so far it's windows only, but I'm definitely planning to get Linux 
to Windows cross-compilation working. And building sources too. You 
probably have some notes on doing those two things, would you be willing 
to share them with me?

Bootstraping is another thing I was planing to implement at some point, 
probably using Guile.

Also Phillip, you you ever need any help with testing on Windows, feel 
free to email me directly. I'd probably be willing to help with that.

Another thing I was thinking about in the long run, is turning that 
script into a more generic build tool, so that you could provide scripts 
for building not just Emacs, but other GNU and similar software. For 
example a script for Guile would be nice, since there's no official 
Windows distribution and then making plugins for different Unix package 
managers would be great.

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Bloat in Windows (revisited)

[Eli Zaretskii]

> From: Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com>
> Date: Thu, 13 Jun 2019 20:27:25 +0300
> Cc: emacs-devel@gnu.org
> 
> Yes, so far it's windows only, but I'm definitely planning to get Linux 
> to Windows cross-compilation working.

Emacs cannot be cross-compiled.  But maybe you mean to start a VM
running Windows or Wine?

Re: Bloat in Windows (revisited)

[Nikolay Kudryavtsev]

I meant in the very loose sense of creating Windows builds from GNU, 
that's how Phillip makes his builds right now, in the absence of 
Windows, right? Or some VM is used?

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Bloat in Windows (revisited)

[Phillip Lord]

Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com> writes:

> Yes, so far it's windows only, but I'm definitely planning to get
> Linux to Windows cross-compilation working. And building sources
> too. You probably have some notes on doing those two things, would you
> be willing to share them with me?


Sure. The code is all in admin/nt/dist-build, as are the setup
instructions.

> Bootstraping is another thing I was planing to implement at some
> point, probably using Guile.
>
> Also Phillip, you you ever need any help with testing on Windows, feel
> free to email me directly. I'd probably be willing to help with that.

It's always good to get testing -- either of the releases or the
snapshots. I post about those here, first.


> Another thing I was thinking about in the long run, is turning that
> script into a more generic build tool, so that you could provide
> scripts for building not just Emacs, but other GNU and similar
> software. For example a script for Guile would be nice, since there's
> no official Windows distribution and then making plugins for different
> Unix package managers would be great.


A variation on my build scripts would do that, I think. Currently, I
just build out of source and then zip everything up. Most of it is not
emacs specific; with some fiddling, it would work for any autoconf'd
package.

At the end of the day, though, I think software like guile is probably
better support within an msys2 distribution with a package manager.

Phil

Re: Bloat in Windows (revisited)

[Phillip Lord]

Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com> writes:

> I meant in the very loose sense of creating Windows builds from GNU,
> that's how Phillip makes his builds right now, in the absence of
> Windows, right? Or some VM is used?

No, builds are all done on windows; originally, a machine in my front
room, now a VM. Emacs doesn't cross compile; even if it did, I would not
want to go to the effort of cross-compiling all the deps also.

Phil

Re: Bloat in Windows (revisited)

[Nikolay Kudryavtsev]

I see. You asking for help with Windows made me assume too much. Since 
MinGW64 can be ran on Unix, I assumed that you managed to do just that. 
Actually have anyone ever tried that? Is there some big problem that 
would prevent this from working, or it's just a lot of work in adapting 
scripts and getting dependencies compiling?

-- 
Best Regards,
Nikolay Kudryavtsev

Re: Bloat in Windows (revisited)

[Phillip Lord]

Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com> writes:

> I see. You asking for help with Windows made me assume too much. Since
> MinGW64 can be ran on Unix, I assumed that you managed to do just
> that. Actually have anyone ever tried that? Is there some big problem
> that would prevent this from working, or it's just a lot of work in
> adapting scripts and getting dependencies compiling?


Eli can probably answer this much better than I. For Emacs-26, there is
unexec functionality which is going to be platform specific, although I
presume that problem goes away with the portable dumper which is (I
guess) portable.

After that, the basic problem is that Emacs is build using Emacs; first
temacs, then bootstrap-emacs. So, you have to be able to run Emacs on
the platform that you are building it on, which conflicts with the idea
of cross compliation.

Phil

Re: Bloat in Windows (revisited)

[Eli Zaretskii]

> From: phillip.lord@russet.org.uk (Phillip Lord)
> Date: Mon, 17 Jun 2019 16:59:50 +0100
> Cc: emacs-devel@gnu.org
> 
> Nikolay Kudryavtsev <nikolay.kudryavtsev@gmail.com> writes:
> 
> > I see. You asking for help with Windows made me assume too much. Since
> > MinGW64 can be ran on Unix, I assumed that you managed to do just
> > that. Actually have anyone ever tried that? Is there some big problem
> > that would prevent this from working, or it's just a lot of work in
> > adapting scripts and getting dependencies compiling?
> 
> Eli can probably answer this much better than I. For Emacs-26, there is
> unexec functionality which is going to be platform specific, although I
> presume that problem goes away with the portable dumper which is (I
> guess) portable.
> 
> After that, the basic problem is that Emacs is build using Emacs; first
> temacs, then bootstrap-emacs. So, you have to be able to run Emacs on
> the platform that you are building it on, which conflicts with the idea
> of cross compliation.

That's right: any program that needs to be run as part of its build
breaks cross-building, because cross-building produces binaries that
cannot be run on the host where they are compiled.

This is not Windows specific, this is a known limitation of Emacs: it
cannot be cross-built.  (Most, if not all, of the dependency packages
Phillip builds and bundles _can_ be cross-compiled, though.)

Re: Bloat in Windows (revisited)

[Stefan Monnier]

> That's right: any program that needs to be run as part of its build
> breaks cross-building, because cross-building produces binaries that
> cannot be run on the host where they are compiled.

Tho sometimes this can be fixed by building those tools twice:
- once for the target
- once for the host (to run it during the build).

E.g. I think we could build almost all of Emacs's files via
cross-compilation, the only exception being the dumped Emacs (i.e. the
.pdmp files): we could build a "host-temacs" which we can use to compile
the .elc files and then a "target-temacs" for the actual
intended target.  But generating the target .pdmp file requires running
the target-temacs.


        Stefan

Re: Bloat in Windows (revisited)

[Óscar Fuentes]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> That's right: any program that needs to be run as part of its build
>> breaks cross-building, because cross-building produces binaries that
>> cannot be run on the host where they are compiled.
>
> Tho sometimes this can be fixed by building those tools twice:
> - once for the target
> - once for the host (to run it during the build).
>
> E.g. I think we could build almost all of Emacs's files via
> cross-compilation, the only exception being the dumped Emacs (i.e. the
> .pdmp files): we could build a "host-temacs" which we can use to compile
> the .elc files and then a "target-temacs" for the actual
> intended target.  But generating the target .pdmp file requires running
> the target-temacs.

For the case GNU/Linux -> Windows, Wine could be of help for generating
the .pdmp file.

Furthermore, since .elc files are cross-platform (right?) an existent
Emacs install could be used if its version is compatlible.