Process to build standalone Emacs + deps in Windows

Process to build standalone Emacs + deps in Windows

[Juan José García Ripoll]

Hi,

I am trying to debug the process of building a distributable Emacs with
dependencies on Windows 64 bits. I have seen some sketch of
documentation under nt/ but it does not include support for graphics and
image formats, which the official distribution for Windows includes.

What are the steps or the scripts used to create the *.zip files that
are found in the official repository? Is there a simple guide I can use
to reproduce it?

Best,

-- 
Juan José García Ripoll

Quantum Information and Foundations Group
http://quinfog.hbar.es - http://juanjose.garcia.ripoll

Re: Process to build standalone Emacs + deps in Windows

[Eli Zaretskii]

> From: juanjose.garciaripoll@gmail.com (Juan José García
>  Ripoll)
> Date: Sun, 22 Mar 2020 14:03:19 +0100
> 
> I am trying to debug the process of building a distributable Emacs with
> dependencies on Windows 64 bits. I have seen some sketch of
> documentation under nt/ but it does not include support for graphics and
> image formats, which the official distribution for Windows includes.
> 
> What are the steps or the scripts used to create the *.zip files that
> are found in the official repository? Is there a simple guide I can use
> to reproduce it?

Did you look at the files in admin/nt/ ?

Re: Process to build standalone Emacs + deps in Windows

[Juan José García-Ripoll]

Eli Zaretskii <eliz@gnu.org> writes:

>> I am trying to debug the process of building a distributable Emacs
>> with dependencies on Windows 64 bits. [...] What are the steps or the
>> scripts used to create the *.zip files that are found in the official
>> repository? Is there a simple guide I can use to reproduce it?
>
> Did you look at the files in admin/nt/ ?

No, I was focused on the 'nt/' folder and did  not expect other
locations. Thanks for the hint.

Now, to motivate my intentions a bit better, I have built Emacs from
savannah's git repository following a different set of instructions
(https://gist.github.com/nauhygon/f3b44f51b34e89bc54f8). These
instructions are hacky (cp /mingw64/bin/*.dll $target/bin).  However,
this seems to work, which shows that, out of the dependencies that Emacs
tracks down during its build process, many can be dropped. The resulting
directory is also leaner

$ du -smc scoop/apps/emacs/26.3 # standard distribution
758     scoop/apps/emacs/26.3
$ du -smc scoop/apps/emacs/27 # from git
207     scoop/apps/emacs/27

The extra space seems to be:
- Executables that come with the linked in libraries, such as
tiff2bw.exe, minizip.exe, imginfo.exe, etc.
- Dependencies that are needed by the libraries in development mode, but
not in actual use, such as the python library brought in by glib,
together with all their associated directories.
- Secondary dependencies of those other libraries, such as tcl/tk
brought in by python (once more, executables, DLL's and supporting
directories)

On a side note, what kind of solution would be acceptable?
- Do we need to keep all the utilities supplied by the libraries Emacs
links to?
- Is a manual filter of second-level dependencies ok?
- Is there a catalogue of utilities that Emacs expect laying around in
the OS? I see the dependencies file contains bzip2.exe, xz.exe and other
tools which are not coming from first-level dependencies.

Finally, I am cleared by the FSF to contribute patches to AucTeX. Would
I require a validation also to contribute fixes to Emacs?

Cheers,

-- 
Juan José García Ripoll

Quantum Information and Foundations Group
http://quinfog.hbar.es - http://juanjose.garciaripoll.com

Re: Process to build standalone Emacs + deps in Windows

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Date: Sun, 22 Mar 2020 15:38:10 +0100
> 
> Now, to motivate my intentions a bit better, I have built Emacs from
> savannah's git repository following a different set of instructions
> (https://gist.github.com/nauhygon/f3b44f51b34e89bc54f8). These
> instructions are hacky (cp /mingw64/bin/*.dll $target/bin).  However,
> this seems to work, which shows that, out of the dependencies that Emacs
> tracks down during its build process, many can be dropped. The resulting
> directory is also leaner

What do you mean by "dependencies that Emacs tracks down during its
build process"?  I'm not sure how what happens during the build is
relevant to what happens at run time.

> On a side note, what kind of solution would be acceptable?
> - Do we need to keep all the utilities supplied by the libraries Emacs
> links to?

No, only those that are needed.  But figuring out which ones are those
is not simple.  The basic problem here is exacerbated by the fact that
MSYS2 produces notoriously fat binary distros, where every possible
optional dependency has been included, whether it does or doesn't make
sense on Windows (or at all), and every possible product is included
(which of course enlarges the list of dependencies and, recursively,
their dependencies).  Figuring out what's needed requires a lot of
tedious reading of documentation and some research.

> - Is a manual filter of second-level dependencies ok?

It depends on who does the job.  Some don't mind manually removing
stuff, others don't have time for that.  Since this is work done by
volunteers, and Emacs as a project doesn't provide "official"
MS-Windows binaries, it isn't for me to make the decision here.

> - Is there a catalogue of utilities that Emacs expect laying around in
> the OS? I see the dependencies file contains bzip2.exe, xz.exe and other
> tools which are not coming from first-level dependencies.

That's a separate issue.  Various Emacs features use external
programs, and where do you stop is up to you.  For example, DocView
will not work 100% without Ghostscript, AFAIK -- do we need to provide
it as part of Emacs?  Or what about TeX/LaTeX, used by some features?
Or Groff (required for formatting man pages, something "M-x man" would
like to do).  Etc. etc.  These are very large packages, and some might
say you will end up with a very large Emacs installation if you
include all of them.

> Finally, I am cleared by the FSF to contribute patches to AucTeX. Would
> I require a validation also to contribute fixes to Emacs?

Your assignment is for Emacs, so you are good there.

Thanks.

Re: Process to build standalone Emacs + deps in Windows

[Phillip Lord]

Juan José García-Ripoll <juanjose.garciaripoll@gmail.com> writes:

> Now, to motivate my intentions a bit better, I have built Emacs from
> savannah's git repository following a different set of instructions
> (https://gist.github.com/nauhygon/f3b44f51b34e89bc54f8). These
> instructions are hacky (cp /mingw64/bin/*.dll $target/bin).  However,
> this seems to work, which shows that, out of the dependencies that Emacs
> tracks down during its build process, many can be dropped. The resulting
> directory is also leaner
>
> $ du -smc scoop/apps/emacs/26.3 # standard distribution
> 758     scoop/apps/emacs/26.3
> $ du -smc scoop/apps/emacs/27 # from git
> 207     scoop/apps/emacs/27
>
> The extra space seems to be:
> - Executables that come with the linked in libraries, such as
> tiff2bw.exe, minizip.exe, imginfo.exe, etc.
> - Dependencies that are needed by the libraries in development mode, but
> not in actual use, such as the python library brought in by glib,
> together with all their associated directories.
> - Secondary dependencies of those other libraries, such as tcl/tk
> brought in by python (once more, executables, DLL's and supporting
> directories)


Yes, all of this is true. I *mostly* just let MSYS2 decide what goes in,
since it provides the full dependency list. There are a few exceptions
which decrease the install size significantly (such as python). Combined
with executable installer option, the overall download size is now at
50Mb -- this is incidentally, only a tad more than the no-deps zip.

I am inclined to believe that at 50Mb, it's really not something worth
worrying about.


> On a side note, what kind of solution would be acceptable?
> - Do we need to keep all the utilities supplied by the libraries Emacs
> links to?
> - Is a manual filter of second-level dependencies ok?
> - Is there a catalogue of utilities that Emacs expect laying around in
> the OS? I see the dependencies file contains bzip2.exe, xz.exe and other
> tools which are not coming from first-level dependencies.

No, there is no such catalogue. It all comes from MSYS2. The only list I
maintain is in build-deps-zip.py which looks like this:

mingw-w64-x86_64-giflib
mingw-w64-x86_64-gnutls
mingw-w64-x86_64-lcms2
mingw-w64-x86_64-libjpeg-turbo
mingw-w64-x86_64-libpng
mingw-w64-x86_64-librsvg
mingw-w64-x86_64-libtiff
mingw-w64-x86_64-libxml2
mingw-w64-x86_64-xpm-nox

Incidentally, @Eli have I missed up here? Should this not include
Harfbuzz? I have a local branch to which I have added it, but it's not
on Master or Emacs-27.

There has been discussion previously both on decreasing the size of the
download (by removing the full transitive dependency list) and also on
adding new dependencies by adding essential utilities. I even added a
feature branch for the latter

http://git.savannah.gnu.org/cgit/emacs.git/log/?h=feature/windows-with-utils

As you can see, it has not had many contributors.

Still, I am open to new ideas and contributors, but limited in time to
support these contributions.

Phil

Re: Process to build standalone Emacs + deps in Windows

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Date: Sun, 22 Mar 2020 18:54:48 +0000
> Cc: emacs-devel@gnu.org
> 
> Incidentally, @Eli have I missed up here? Should this not include
> Harfbuzz? I have a local branch to which I have added it, but it's not
> on Master or Emacs-27.

It definitely should.  I asked you on at least two occasions why the
Emacs you build doesn't include HarfBuzz, but got no answer.  I guess
I have it now ;-)

Re: Process to build standalone Emacs + deps in Windows

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Date: Sun, 22 Mar 2020 18:54:48 +0000
>> Cc: emacs-devel@gnu.org
>> 
>> Incidentally, @Eli have I missed up here? Should this not include
>> Harfbuzz? I have a local branch to which I have added it, but it's not
>> on Master or Emacs-27.
>
> It definitely should.  I asked you on at least two occasions why the
> Emacs you build doesn't include HarfBuzz, but got no answer.  I guess
> I have it now ;-)

Yeah, I added it to the HarfBuzz branch. But then I changed
build-dep-zips to use a different logic so that I could selectively drop
some of the transitive dependencies because of the really large size the
download had. I guess it did not survive the merge of the HarfBuzz branch.

So, what should I do? Clearly it needs to go in. I guess as we are still
pre-release, I should add to Emacs-27; I can rebuild the deps files now,
before I fix it for the Emacs-27 cycle?

Sorry I missed your earlier emails; I struggle to keep up with email am
afraid.

Phil

Re: Process to build standalone Emacs + deps in Windows

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: juanjose.garciaripoll@gmail.com,  emacs-devel@gnu.org
> Date: Sun, 22 Mar 2020 21:02:06 +0000
> 
> So, what should I do? Clearly it needs to go in. I guess as we are still
> pre-release, I should add to Emacs-27; I can rebuild the deps files now,
> before I fix it for the Emacs-27 cycle?

Yes, please.

> Sorry I missed your earlier emails; I struggle to keep up with email am
> afraid.

No sweat.

Re: Process to build standalone Emacs + deps in Windows

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Cc: juanjose.garciaripoll@gmail.com,  emacs-devel@gnu.org
>> Date: Sun, 22 Mar 2020 21:02:06 +0000
>> 
>> So, what should I do? Clearly it needs to go in. I guess as we are still
>> pre-release, I should add to Emacs-27; I can rebuild the deps files now,
>> before I fix it for the Emacs-27 cycle?
>
> Yes, please.


Want me to push these up to alpha with the same name (emacs-27.0.90-) on
top of the existing ones or rename them? (emacs-27.0.90-2-).

Phil

Re: Process to build standalone Emacs + deps in Windows

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Cc: juanjose.garciaripoll@gmail.com,  emacs-devel@gnu.org
> Date: Mon, 23 Mar 2020 23:36:36 +0000
> 
> Want me to push these up to alpha with the same name (emacs-27.0.90-) on
> top of the existing ones or rename them? (emacs-27.0.90-2-).

A new name would be preferable, thanks.

Re: Process to build standalone Emacs + deps in Windows

[Phillip Lord]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Phillip Lord <phillip.lord@russet.org.uk>
>> Cc: juanjose.garciaripoll@gmail.com,  emacs-devel@gnu.org
>> Date: Mon, 23 Mar 2020 23:36:36 +0000
>> 
>> Want me to push these up to alpha with the same name (emacs-27.0.90-) on
>> top of the existing ones or rename them? (emacs-27.0.90-2-).
>
> A new name would be preferable, thanks.

All done.

Re: Process to build standalone Emacs + deps in Windows

[Juan José García-Ripoll]

Phillip Lord <phillip.lord@russet.org.uk> writes:
> Yes, all of this is true. I *mostly* just let MSYS2 decide what goes in,
> since it provides the full dependency list. There are a few exceptions
> which decrease the install size significantly (such as python). Combined
> with executable installer option, the overall download size is now at
> 50Mb -- this is incidentally, only a tad more than the no-deps zip.

You are right. Indeed, I did not realize that you had already shrunk the
size so much!

- I checked the 26.3 release and that one was 758Mb with emacs and
  dependencies.

- I just built dependencies with the method in the emacs-27 prerelease and
  x86_64 is just 128 Mb. Python is no longer installed and it is not listed as
  dependency by glib2.

Nevertheless, we could make further improvements:

- We can save further 37Mb by removing the linking libraries *.a from
  mingw64/lib

- Another 26Mb can be removed by eliminating the mingw64/share/doc and
  mingw64/share/gtk-doc folders. I would only leave share/doc/licenses for
  obvious reasons

- Since we are not shipping libraries, mingw64/include should also probably go.

- Since we are pulling in compression utilities, we may just as well tell the
  configuration to compress the *.el files. Windows is the only platform where
  this does not happen
  
- I would remove all utilities (*.exe) that are not strictly needed. I suspect
  the *tls*.exe tools are required. We could leave the those, the compressor
  and remove the rest.

With all this, except the removal of *.exe files, we go down to 52Mb (60%
reduction) uncompressed data, as reported by unzip. I know that space is not
much of an issue in this world, but I think it is fair to keep just what is
needed, plus it makes Emacs even smaller than most Electron apps around. First
impressions matter.

Cheers,

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es

Re: Process to build standalone Emacs + deps in Windows

[Juan José García-Ripoll]

Juan José García-Ripoll <juanjose.garciaripoll@gmail.com> writes:
> Nevertheless, we could make further improvements: [...]
> With all this, except the removal of *.exe files, we go down to 52Mb (60%
> reduction) uncompressed data, as reported by unzip. I know that space is not
> much of an issue in this world, but I think it is fair to keep just what is
> needed, plus it makes Emacs even smaller than most Electron apps around. First
> impressions matter.

I attach a patch file that changes build-dep-zips.py with the following
fixes

1. It no longer assumes that LANG is en_US. I am in Spain and pacman
defaults to the system's locale, breaking the algorithm that
build-dep-zips.py uses to identify dependencies.

2. Cleaning is a standalone operation. Formerly it would try to build
all dependencies prior to cleaning. Since I have msys only for 64-bits
that would fail in my system.

3. It shows what package pulls what. It has been helpful in
understanding the dependencies.

4. It creates a zip file where many of the superfluous directories and
executables have been removed. I would go for removing even more, since
it seems that Emacs is built with zlib and we could get rid of
b*zip*.exe, xz.exe, etc.

Overall the dependencies are now about 42Mb uncompressed. As I said, it
can be slightly smaller if we get rid of further command line tools, but
I am currently worried that some of the TLS routines may be used.

Some questions:

a) Is there a way to test the resulting standalone distribution to
verify that Emacs is running properly?

b) I realized that Cairo is pulled in as a dependency, but Emacs is
built without support for Cairo. Is this intentional?

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es

Re: Process to build standalone Emacs + deps in Windows

[Juan José García-Ripoll]

[-- Attachment #1: Type: text/plain, Size: 279 bytes --]

Juan José García-Ripoll <juanjose.garciaripoll@gmail.com> writes:
> I attach a patch file that changes build-dep-zips.py with the following
> fixes

As usual, the missing attachment below.

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: build-dep-zips.diff --]
[-- Type: text/x-patch, Size: 3642 bytes --]

diff --git a/admin/nt/dist-build/build-dep-zips.py b/admin/nt/dist-build/build-dep-zips.py
index 33ed4b6..6390eae 100755
--- a/admin/nt/dist-build/build-dep-zips.py
+++ b/admin/nt/dist-build/build-dep-zips.py
@@ -83,6 +83,11 @@ def immediate_deps(pkg):
     dependencies = [d.split(">")[0] for d in dependencies if d]
     dependencies = [d for d in dependencies if not d == "None"]

+    if dependencies:
+        print(f"Package {pkg} depends on:")
+        for d in dependencies:
+            print(f" -> {d}")
+
     dependencies = [MUNGE_DEP_PKGS.get(d, d) for d in dependencies]
     return dependencies

@@ -107,7 +112,8 @@ def extract_deps():

 def gather_deps(deps, arch, directory):

-    os.mkdir(arch)
+    if not os.path.exists(arch):
+        os.mkdir(arch)
     os.chdir(arch)

     ## Replace the architecture with the correct one
@@ -135,11 +141,56 @@ def gather_deps(deps, arch, directory):
     print("Copying dependencies: {}".format(arch))
     check_output_maybe(["rsync", "-R"] + deps_files + ["."])

+    ## exclude files
+    excludes=['lib/*.a', # No files to link against
+              'lib/*/*.exe', # No hidden executables
+              'libexec/*/*.exe',
+              'include/*', # No development files
+              'lib/cmake/*',
+              'lib/pkgconfig/*',
+              'lib/python*/*',
+              'share/aclocal/*',
+              'share/gettext/*',
+              'share/doc/*', # No documentation from libraries
+              'share/gtk-doc/*',
+              'share/man/*',
+              'share/info/*',
+              'bin/*tiff*.exe', # No graphic manipulation tools
+              'bin/img*.exe',
+              'bin/*gif*.exe',
+              'bin/*jpg*.exe',
+              'bin/*jpeg*.exe',
+              'bin/*png*.exe',
+              'bin/*svg*.exe',
+              'bin/*xpm*.exe',
+              'bin/*icc.exe',
+              'bin/fax2*.exe',
+              'bin/pango-*.exe',
+              'bin/gdk-*.exe',
+              'bin/jasper.exe',
+              'bin/fc-*.exe', # No font configuration tools
+              'bin/gr2fonttest.exe',
+              'bin/hb-*.exe',
+              'bin/msg*.exe', # No tools to manipulate gettext catalogues
+              'bin/*gettext.exe'
+    ]
+    ## Files that might have been excluded by those rules
+    includes=[]
+    ## Build command line argument for zip
+    if excludes:
+        excludes = '-x '+' '.join(excludes).replace('*','\\*')
+    else:
+        excludes = ''
+    if includes:
+        includes = '-i '+' '.join(includes).replace('*','\\*')
+    else:
+        includes=''
+
     ## And package them up
     os.chdir(directory)
     print("Zipping: {}".format(arch))
-    check_output_maybe("zip -9r ../../emacs-{}-{}{}-deps.zip *"
-                       .format(EMACS_MAJOR_VERSION, DATE, arch),
+    check_output_maybe("zip -9r ../../emacs-{}-{}{}-deps.zip * {} {}"
+                       .format(EMACS_MAJOR_VERSION, DATE, arch, excludes, includes),
                        shell=True)
     os.chdir("../../")

@@ -255,6 +306,17 @@ def clean():
 args = parser.parse_args()
 do_all=not (args.c or args.r or args.f or args.t)

+if( args.c ):
+    clean()
+    exit(0)
+
+# The dependency extraction relies on an English version of pacman
+# We need to configure the locale to match the expectations
+if 'LANG' in os.environ:
+    os_lang = os.environ['LANG']
+    if (len(os_lang) > 2) and (os_lang[0:2] != 'en'):
+        os.environ['LANG']='en_US'
+
 deps=extract_deps()

 DRY_RUN=args.d
@@ -277,6 +339,3 @@ def clean():

 if( do_all or args.r ):
     gather_source(deps)
-
-if( args.c ):
-    clean()

Re: Process to build standalone Emacs + deps in Windows

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Date: Wed, 25 Mar 2020 15:54:05 +0100
> 
> +              'libexec/*/*.exe',

This is not necessarily wise.  It could be TRT with the current
dependencies, but might lose in the long run.  E.g., suppose someone
decides to add GNU Findutils to the bundle.

> +              'share/doc/*', # No documentation from libraries
> +              'share/gtk-doc/*',
> +              'share/man/*',
> +              'share/info/*',

Questionable.  Why deprive the users from being able to read the docs
of the dependencies?  Some of that could be very relevant.  E.g., what
about the docs of bzip2.exe, xz.exe, etc.? they might come in handy
for setting up all kinds of *-switches defcustoms.

> +              'bin/fc-*.exe', # No font configuration tools

If some dependency is configured to access fonts via Fontconfig (is it
HarfBuzz?), then this is a mistake, because some of those utilities
are needed to maintain the font cache.

Thanks.

Re: Process to build standalone Emacs + deps in Windows

[Juan José García-Ripoll]

[-- Attachment #1: Type: text/plain, Size: 1885 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Juan José García-Ripoll
>>  <juanjose.garciaripoll@gmail.com>
>> Date: Wed, 25 Mar 2020 15:54:05 +0100
>> 
>> +              'libexec/*/*.exe',
>
> This is not necessarily wise.  It could be TRT with the current
> dependencies, but might lose in the long run.  E.g., suppose someone
> decides to add GNU Findutils to the bundle.

Understood. Can we remove the ones that are not used?

>> +              'share/doc/*', # No documentation from libraries
>> +              'share/gtk-doc/*',
>> +              'share/man/*',
>> +              'share/info/*',
>
> Questionable.  Why deprive the users from being able to read the docs
> of the dependencies?  Some of that could be very relevant.  E.g., what
> about the docs of bzip2.exe, xz.exe, etc.? they might come in handy
> for setting up all kinds of *-switches defcustoms.

Ok. This relates to an earlier email where I asked about what of those
tools one should keep, but not completely.

Many of those directories contain manuals and documentation for
libraries (e.g. C/python interfaces to gettext, libxml, etc), not for
tools (bzip2, bunzip2, etc). Since users do not have exposed access to
those libraries, it makes as much sense to bundle them with emacs as it
would make it to bundle the documentation of the C library.

>> +              'bin/fc-*.exe', # No font configuration tools
>
> If some dependency is configured to access fonts via Fontconfig (is it
> HarfBuzz?), then this is a mistake, because some of those utilities
> are needed to maintain the font cache.

Thanks. So far I have been using emacs-27 w/o them. But maybe it is
because I built emacs before HarfBuzz was added to build-dep*.py? In any
case, I can remove it.

Does the newly attached script look more reasonable?

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: build-dep-zips.diff --]
[-- Type: text/x-patch, Size: 3667 bytes --]

diff --git a/admin/nt/dist-build/build-dep-zips.py b/admin/nt/dist-build/build-dep-zips.py
index 33ed4b6..118fde2 100755
--- a/admin/nt/dist-build/build-dep-zips.py
+++ b/admin/nt/dist-build/build-dep-zips.py
@@ -83,6 +83,11 @@ def immediate_deps(pkg):
     dependencies = [d.split(">")[0] for d in dependencies if d]
     dependencies = [d for d in dependencies if not d == "None"]

+    if dependencies:
+        print(f"Package {pkg} depends on:")
+        for d in dependencies:
+            print(f" -> {d}")
+
     dependencies = [MUNGE_DEP_PKGS.get(d, d) for d in dependencies]
     return dependencies

@@ -107,7 +112,8 @@ def extract_deps():

 def gather_deps(deps, arch, directory):

-    os.mkdir(arch)
+    if not os.path.exists(arch):
+        os.mkdir(arch)
     os.chdir(arch)

     ## Replace the architecture with the correct one
@@ -135,11 +141,56 @@ def gather_deps(deps, arch, directory):
     print("Copying dependencies: {}".format(arch))
     check_output_maybe(["rsync", "-R"] + deps_files + ["."])

+    ## exclude files
+    excludes=['lib/*.a', # No files to link against
+              'lib/*/*.exe', # No hidden executables
+              'libexec/p11-kit/*.exe',
+              'include/*', # No development files
+              'lib/cmake/*',
+              'lib/pkgconfig/*',
+              'lib/python*/*',
+              'share/aclocal/*',
+              'share/gettext/*',
+              'share/doc/lib*', # No documentation from libraries
+              'share/doc/gettext/*',
+              'share/gtk-doc/*',
+              'share/man/man3/*',
+              'share/man/man5/*',
+              'bin/*tiff*.exe', # No graphic manipulation tools
+              'bin/img*.exe',
+              'bin/*gif*.exe',
+              'bin/*jpg*.exe',
+              'bin/*jpeg*.exe',
+              'bin/*png*.exe',
+              'bin/*svg*.exe',
+              'bin/*xpm*.exe',
+              'bin/*icc.exe',
+              'bin/fax2*.exe',
+              'bin/pango-*.exe',
+              'bin/gdk-*.exe',
+              'bin/jasper.exe',
+              'bin/gr2fonttest.exe', # No font visualization tools
+              'bin/hb-*.exe',
+              'bin/msg*.exe', # No tools to manipulate gettext catalogues
+              'bin/*gettext.exe'
+    ]
+    ## Files that might have been excluded by those rules
+    includes=[]
+    ## Build command line argument for zip
+    if excludes:
+        excludes = '-x '+' '.join(excludes).replace('*','\\*')
+    else:
+        excludes = ''
+    if includes:
+        includes = '-i '+' '.join(includes).replace('*','\\*')
+    else:
+        includes=''
+
     ## And package them up
     os.chdir(directory)
     print("Zipping: {}".format(arch))
-    check_output_maybe("zip -9r ../../emacs-{}-{}{}-deps.zip *"
-                       .format(EMACS_MAJOR_VERSION, DATE, arch),
+    check_output_maybe("zip -9r ../../emacs-{}-{}{}-deps.zip * {} {}"
+                       .format(EMACS_MAJOR_VERSION, DATE, arch, excludes, includes),
                        shell=True)
     os.chdir("../../")

@@ -255,6 +306,17 @@ def clean():
 args = parser.parse_args()
 do_all=not (args.c or args.r or args.f or args.t)

+if( args.c ):
+    clean()
+    exit(0)
+
+# The dependency extraction relies on an English version of pacman
+# We need to configure the locale to match the expectations
+if 'LANG' in os.environ:
+    os_lang = os.environ['LANG']
+    if (len(os_lang) > 2) and (os_lang[0:2] != 'en'):
+        os.environ['LANG']='en_US'
+
 deps=extract_deps()

 DRY_RUN=args.d
@@ -277,6 +339,3 @@ def clean():

 if( do_all or args.r ):
     gather_source(deps)
-
-if( args.c ):
-    clean()

Re: Process to build standalone Emacs + deps in Windows

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Date: Wed, 25 Mar 2020 17:41:35 +0100
> 
> >> +              'libexec/*/*.exe',
> >
> > This is not necessarily wise.  It could be TRT with the current
> > dependencies, but might lose in the long run.  E.g., suppose someone
> > decides to add GNU Findutils to the bundle.
> 
> Understood. Can we remove the ones that are not used?

Yes, sure.  My comment was about emptying the entire directory.

> Many of those directories contain manuals and documentation for
> libraries (e.g. C/python interfaces to gettext, libxml, etc), not for
> tools (bzip2, bunzip2, etc). Since users do not have exposed access to
> those libraries, it makes as much sense to bundle them with emacs as it
> would make it to bundle the documentation of the C library.

Again, discretion is advisable.

> >> +              'bin/fc-*.exe', # No font configuration tools
> >
> > If some dependency is configured to access fonts via Fontconfig (is it
> > HarfBuzz?), then this is a mistake, because some of those utilities
> > are needed to maintain the font cache.
> 
> Thanks. So far I have been using emacs-27 w/o them. But maybe it is
> because I built emacs before HarfBuzz was added to build-dep*.py? In any
> case, I can remove it.

It could be that librsvg uses Fontconfig, I don't know.  (In my builds
of optional libraries, I've succeeded to avoid Fontconfig being
compiled in, but MSYS2 generally compile in everything, whether needed
or not.)

> Does the newly attached script look more reasonable?

I think so, yes.

Re: Process to build standalone Emacs + deps in Windows

[Juan José García-Ripoll]

[-- Attachment #1: Type: text/plain, Size: 1025 bytes --]

Eli Zaretskii <eliz@gnu.org> writes:
>> Thanks. So far I have been using emacs-27 w/o them. But maybe it is
>> because I built emacs before HarfBuzz was added to build-dep*.py? In any
>> case, I can remove it.
>
> It could be that librsvg uses Fontconfig, I don't know.  (In my builds
> of optional libraries, I've succeeded to avoid Fontconfig being
> compiled in, but MSYS2 generally compile in everything, whether needed
> or not.)

The new script shows the implicit dependencies
  rsvg -> Pango
     Pango -> Cairo and Fontconfig
     Cairo -> Fontconfig

It seems therefore a very much confined dependency that can be tested. TBH, I
have not had a look at SVG files in a few weeks, so it may well be that I broke
something by removing fc-*.exe locally (not in the python script). I will
check.

Incidentally, here goes a patch to allow build-zips.sh just build zips (with
build-zips.sh -z) and avoid building the installer.

Cheers

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: build-zips.diff --]
[-- Type: text/x-patch, Size: 1331 bytes --]

diff --git a/admin/nt/dist-build/build-zips.sh b/admin/nt/dist-build/build-zips.sh
index 8eaa3a9..1342604 100755
--- a/admin/nt/dist-build/build-zips.sh
+++ b/admin/nt/dist-build/build-zips.sh
@@ -102,6 +102,7 @@
 SNAPSHOT=
 CACHE=

+INSTALLER=1
 BUILD=1
 BUILD_32=1
 BUILD_64=1
@@ -110,7 +111,7 @@ CONFIG=
 CFLAGS="-O2 -static"
 INSTALL_TARGET="install-strip"

-while getopts "36gb:hnsiV:" opt; do
+while getopts "36gb:hnsizV:" opt; do
   case $opt in
     3)
         BUILD_32=1
@@ -152,8 +153,12 @@ INSTALL_TARGET=
         echo "  -6 64 bit build only"
         echo "  -g git update and worktree only"
         echo "  -i build installer only"
+        echo "  -z only build zip files"
         exit 0
         ;;
+    z)
+        INSTALLER=0
+        ;;
     \?)
         echo "Invalid option: -$OPTARG" >&2
         ;;
@@ -223,7 +228,10 @@ OF_VERSION=
     then
         build_zip x86_64 /mingw64/lib/pkgconfig x86_64-w64-mingw32
     fi
-    build_installer x86_64
+    if (($INSTALLER))
+    then
+        build_installer x86_64
+    fi
 fi

 ## Do the 64 bit build first, because we reset some environment
@@ -234,5 +242,8 @@ OF_VERSION=
     then
         build_zip i686 /mingw32/lib/pkgconfig i686-w64-mingw32
     fi
-    build_installer i686
+    if (($INSTALLER))
+    then
+        build_installer i686
+    fi
 fi

Re: Process to build standalone Emacs + deps in Windows

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Date: Wed, 25 Mar 2020 18:22:47 +0100
> 
> > It could be that librsvg uses Fontconfig, I don't know.  (In my builds
> > of optional libraries, I've succeeded to avoid Fontconfig being
> > compiled in, but MSYS2 generally compile in everything, whether needed
> > or not.)
> 
> The new script shows the implicit dependencies
>   rsvg -> Pango
>      Pango -> Cairo and Fontconfig
>      Cairo -> Fontconfig

Right.  My librsvg is built without the Fontconfig dependencies, but
AFAIR it required some investigation and non-default configure
switches.

Re: Process to build standalone Emacs + deps in Windows

[Phillip Lord]

Juan José García-Ripoll <juanjose.garciaripoll@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>
> Incidentally, here goes a patch to allow build-zips.sh just build zips (with
> build-zips.sh -z) and avoid building the installer.


I'd rather not add that at the moment. I suspect that there are only two
people who have ever run build-zips.sh and one of them (me) doesn't need
it.

The number of options on build-zips.sh is too large anyway, and
documenting it is on my list of things to do. I have a sneaking
suspicion that I have changed it, broken something else, then fixed it
again several times at different points in the Emacs release cycle. With
a gap of years between major releases, it's hard to get right for all
the different stages.

Phil

Re: Process to build standalone Emacs + deps in Windows

[Juan José García-Ripoll]

Phillip Lord <phillip.lord@russet.org.uk> writes:
> Juan José García-Ripoll <juanjose.garciaripoll@gmail.com> writes:
>
>> Eli Zaretskii <eliz@gnu.org> writes:
>>
>> Incidentally, here goes a patch to allow build-zips.sh just build zips (with
>> build-zips.sh -z) and avoid building the installer.
>
> I'd rather not add that at the moment. I suspect that there are only two
> people who have ever run build-zips.sh and one of them (me) doesn't need
> it.

I think that build-zips.sh is actually far more useful than the current
build setup that is by default in the nt/ directory. I personally do not
use the latter, because it forces me to have a full installation of
Msys/Mingw laying around.

So the fact that only two (maybe now three) people use it is not because
it is not useful, but because it is not visible. The consequence of this
is the type of hacky instructions that are out there in the internet,
advising people to build emacs, install it in a directory and copy all
of mingw64's *.dll.

Instead, Emacs on Windows should have, by default, a build process that
allows users to build their own standalone installation, without
imposing them to use makensis.

My 2c.

Juanjo

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es

Re: Process to build standalone Emacs + deps in Windows

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Date: Thu, 26 Mar 2020 14:16:54 +0100
> 
> Emacs on Windows should have, by default, a build process that
> allows users to build their own standalone installation, without
> imposing them to use makensis.

It does.  It's just that the build process we support doesn't include
building and packaging optional libraries, that is definitely outside
of the scope of the Emacs project.  I'm okay with hosting contributed
scripts that do more than what we are supposed to do, but I don't
think it's fair to take that for granted and demand that we provide
some QoS regarding those scripts.

Thanks.

Re: Process to build standalone Emacs + deps in Windows

[Phillip Lord]

Juan José García-Ripoll <juanjose.garciaripoll@gmail.com> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>> Juan José García-Ripoll <juanjose.garciaripoll@gmail.com> writes:
>>
>>> Eli Zaretskii <eliz@gnu.org> writes:
>>>
>>> Incidentally, here goes a patch to allow build-zips.sh just build zips (with
>>> build-zips.sh -z) and avoid building the installer.
>>
>> I'd rather not add that at the moment. I suspect that there are only two
>> people who have ever run build-zips.sh and one of them (me) doesn't need
>> it.
>
> I think that build-zips.sh is actually far more useful than the current
> build setup that is by default in the nt/ directory. I personally do not
> use the latter, because it forces me to have a full installation of
> Msys/Mingw laying around.
>
> So the fact that only two (maybe now three) people use it is not because
> it is not useful, but because it is not visible. The consequence of this
> is the type of hacky instructions that are out there in the internet,
> advising people to build emacs, install it in a directory and copy all
> of mingw64's *.dll.
>
> Instead, Emacs on Windows should have, by default, a build process that
> allows users to build their own standalone installation, without
> imposing them to use makensis.


build-zips does require a MSYS2 installation and a fairly odd directory
structure. It doesn't hassle me because I only use my windows machine
for building Emacs.

build-zips could do with cleaning up and making more generic, but I lack
the time, especially as it's quite difficult to test.

Phil

Re: Process to build standalone Emacs + deps in Windows

[Phillip Lord]

Juan José García-Ripoll <juanjose.garciaripoll@gmail.com> writes:

> Juan José García-Ripoll <juanjose.garciaripoll@gmail.com> writes:
>> I attach a patch file that changes build-dep-zips.py with the following
>> fixes
>
> As usual, the missing attachment below.


In reply to two of your emails, sorry:

+    if dependencies:
+        print(f"Package {pkg} depends on:")
+        for d in dependencies:
+            print(f" -> {d}")
+

I wouldn't add this. It's only useful some of the time, not in normal
usage, and anyway "pactree" does the same thing better.

def gather_deps(deps, arch, directory):

-    os.mkdir(arch)
+    if not os.path.exists(arch):
+        os.mkdir(arch)
     os.chdir(arch)

This is wrong, I think. IIRC, my script crashes out at this point if
"arch" exists. This is needed, otherwise I could run build-dep-zips.py
twice at different times and both old and new dependencies would get
added. This forces cleanliness.


+    ## exclude files
+    excludes=['lib/*.a', # No files to link against
+              'lib/*/*.exe', # No hidden executables
+              'libexec/*/*.exe',
+              'include/*', # No development files
+              'lib/cmake/*',
+              'lib/pkgconfig/*',
+              'lib/python*/*',
+              'share/aclocal/*',
+              'share/gettext/*',
+              'share/doc/*', # No documentation from libraries
+              'share/gtk-doc/*',
+              'share/man/*',
+              'share/info/*',
+              'bin/*tiff*.exe', # No graphic manipulation tools
+              'bin/img*.exe',
+              'bin/*gif*.exe',
+              'bin/*jpg*.exe',
+              'bin/*jpeg*.exe',
+              'bin/*png*.exe',
+              'bin/*svg*.exe',
+              'bin/*xpm*.exe',
+              'bin/*icc.exe',
+              'bin/fax2*.exe',
+              'bin/pango-*.exe',
+              'bin/gdk-*.exe',
+              'bin/jasper.exe',
+              'bin/fc-*.exe', # No font configuration tools
+              'bin/gr2fonttest.exe',
+              'bin/hb-*.exe',
+              'bin/msg*.exe', # No tools to manipulate gettext catalogues
+              'bin/*gettext.exe'
+    ]

This list would need to go near the front, so we have a clean
configuration section. It also displays the key problem -- it's really
long and non obvious so a maintenance burden

If we just did a few big top level directories, would that not solve all
the problems.




+# The dependency extraction relies on an English version of pacman
+# We need to configure the locale to match the expectations
+if 'LANG' in os.environ:
+    os_lang = os.environ['LANG']
+    if (len(os_lang) > 2) and (os_lang[0:2] != 'en'):
+        os.environ['LANG']='en_US'
+




I am vaguely curious what it looks like in Spanish. I thought the output
was fairly mechanistic.


- Since we are pulling in compression utilities, we may just as well tell the
  configuration to compress the *.el files. Windows is the only platform where
  this does not happen

No, because I generate the no-deps version first, then add the deps. The
-no-deps version doesn't have the compression utilities. Of course, we
do not need the *.el files at all.


  a) Is there a way to test the resulting standalone distribution to
  verify that Emacs is running properly?

No. The only testing I do is to start Emacs and see if it starts and
looks okay (literally looks okay -- the icons can get broken). The lack
of a test suite that I can run is what keeps me cautious, especially as
I don't use this build. I wrote a robot testing framework for
org-drill.el; we could do something similar for testing this version of
Emacs.


  b) I realized that Cairo is pulled in as a dependency, but Emacs is
  built without support for Cairo. Is this intentional?

No, it will be a transitive dependency of something. pactree will
reveal!

Phil

Re: Process to build standalone Emacs + deps in Windows

[Stefan Monnier]

> +    excludes=['lib/*.a', # No files to link against
[...]
> This list would need to go near the front, so we have a clean
> configuration section. It also displays the key problem -- it's really
> long and non obvious so a maintenance burden

I had a clean and simple solution, but I wrote it in a small margin
and now I can't read it any more, sorry.

> +# The dependency extraction relies on an English version of pacman
> +# We need to configure the locale to match the expectations
> +if 'LANG' in os.environ:
> +    os_lang = os.environ['LANG']
> +    if (len(os_lang) > 2) and (os_lang[0:2] != 'en'):
> +        os.environ['LANG']='en_US'

I think the `C` locale is the obvious choice for these kinds of situations.

> Of course, we do not need the *.el files at all.

I really hope we keep them there.  Even if many users won't look at
them, the fact that they're so easy to get to is almost a defining
feature of Emacs.


        Stefan

Re: Process to build standalone Emacs + deps in Windows

[Juan José García-Ripoll]

Stefan Monnier <monnier@iro.umontreal.ca> writes:
>> Of course, we do not need the *.el files at all.
>
> I really hope we keep them there.  Even if many users won't look at
> them, the fact that they're so easy to get to is almost a defining
> feature of Emacs.

I concur that the *.el files _are_ needed. Right now I am debugging 3
regressions in Emacs-27 and a bug in 26.3. I can do this with the
installed copy of Emacs without needing to install the sources (even
though I have them in other directories) because I can inspect the
code. Moreover, the current information system about variables and
functions is dramatically improved by having them around: I can use C-h
v and name a variable and go to the code and see how it is used,
etc. Without the *.el files Emacs becomes just a black box, like VS
Studio.

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es

Re: Process to build standalone Emacs + deps in Windows

[Phillip Lord]

Stefan Monnier <monnier@iro.umontreal.ca> writes:

>> +    excludes=['lib/*.a', # No files to link against
> [...]
>> This list would need to go near the front, so we have a clean
>> configuration section. It also displays the key problem -- it's really
>> long and non obvious so a maintenance burden
>
> I had a clean and simple solution, but I wrote it in a small margin
> and now I can't read it any more, sorry.

Move to vim perhaps?


>> +# The dependency extraction relies on an English version of pacman
>> +# We need to configure the locale to match the expectations
>> +if 'LANG' in os.environ:
>> +    os_lang = os.environ['LANG']
>> +    if (len(os_lang) > 2) and (os_lang[0:2] != 'en'):
>> +        os.environ['LANG']='en_US'
>
> I think the `C` locale is the obvious choice for these kinds of situations.
>
>> Of course, we do not need the *.el files at all.
>
> I really hope we keep them there.  Even if many users won't look at
> them, the fact that they're so easy to get to is almost a defining
> feature of Emacs.

Yeah. I had a vague memory that the default out-of-source build didn't
put the .el files there, and that I'd done that deliberately for the
windows build. But there's nothing in build-zips.sh to suggest that this
is true, so I guess my memory is wrong.

Phil

Re: Process to build standalone Emacs + deps in Windows

[Juan José García-Ripoll]

Phillip Lord <phillip.lord@russet.org.uk> writes:
> In reply to two of your emails, sorry:
>
> +    if dependencies:
> +        print(f"Package {pkg} depends on:")
> +        for d in dependencies:
> +            print(f" -> {d}")
> +
>
> I wouldn't add this. It's only useful some of the time, not in normal
> usage, and anyway "pactree" does the same thing better.

Right now the scripts do not provide this information but do emit a line
"Adding *package name*" I personally find it more informative to at
least instruct why the package is added. If pactree is more useful, it
is not used anywhere in the current code.

> +    ## exclude files
> +    excludes=['lib/*.a', # No files to link against
> +              'lib/*/*.exe', # No hidden executables
> +              'libexec/*/*.exe',
>  [...]
>
> This list would need to go near the front, so we have a clean
> configuration section. It also displays the key problem -- it's really
> long and non obvious so a maintenance burden

Not really. The *.exe can be deduced from the pacman files.

> If we just did a few big top level directories, would that not solve all
> the problems.

I proposed to remove complete directories but it seems it is not an option.

> +# The dependency extraction relies on an English version of pacman
> +# We need to configure the locale to match the expectations
> +if 'LANG' in os.environ:
> +    os_lang = os.environ['LANG']
> +    if (len(os_lang) > 2) and (os_lang[0:2] != 'en'):
> +        os.environ['LANG']='en_US'
> I am vaguely curious what it looks like in Spanish. I thought the output
> was fairly mechanistic.

Your filter relies on the English word "depends on"
    ## Extract the "Depends On" line
    depends_on = [x for x in package_info if x.startswith("Depends On")][0]
This is heavily locale-dependent

> - Since we are pulling in compression utilities, we may just as well tell the
>   configuration to compress the *.el files. Windows is the only platform where
>   this does not happen
>
> No, because I generate the no-deps version first, then add the deps. The
> -no-deps version doesn't have the compression utilities. Of course, we
> do not need the *.el files at all.

The no-deps version does not have the compression utilities but it is
not intended to be used by itself, as it depends on all the *.dll Hence,
one may argue that it can be built to support compression.

>   b) I realized that Cairo is pulled in as a dependency, but Emacs is
>   built without support for Cairo. Is this intentional?
>
> No, it will be a transitive dependency of something. pactree will
> reveal!

My simple lines instructing what depends on what showed this: it is
pulled in by rsvg. There were some earlier emails about this.

Cheers,

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es

Re: Process to build standalone Emacs + deps in Windows

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Date: Thu, 26 Mar 2020 14:24:15 +0100
> 
> The no-deps version does not have the compression utilities but it is
> not intended to be used by itself, as it depends on all the *.dll

If this is indeed so, it's a bug in the build procedure.  All the
optional DLLs (a.k.a. "dependencies") are loaded dynamically at run
time, and if some DLL is unavailable, the corresponding features will
fail, but Emacs will still run "by itself" and be a useful editor.
That is not what "depends on DLLs" mean on Windows: it means a program
will refuse to start if the DLL cannot be found.  For Emacs, this is
only true for a few core DLLs that are available on any Windows system.

Re: Process to build standalone Emacs + deps in Windows

[Juan José García-Ripoll]

Eli Zaretskii <eliz@gnu.org> writes:
> If this is indeed so, it's a bug in the build procedure.  All the
> optional DLLs (a.k.a. "dependencies") are loaded dynamically at run
> time, and if some DLL is unavailable, the corresponding features will
> fail, but Emacs will still run "by itself" and be a useful editor.

Sorry, it seems I misunderstood the *-no-deps concept deeply. I had
always assumed that *-no-deps whas a zip to be installed on top of the
*-deps one.

So, just to clarify, the *-no-deps is a standalone executable that works
(I checked it) but only guarantees that the editor works. Other
components, such as Gnus, the web browser, etc, may fail due to missing
tools (e.g. Gnus fails because gnu-tls is missing).

Cheers,

-- 
Juan José García Ripoll
http://juanjose.garciaripoll.com
http://quinfog.hbar.es

Re: Process to build standalone Emacs + deps in Windows

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Date: Sat, 28 Mar 2020 17:01:52 +0100
> 
> So, just to clarify, the *-no-deps is a standalone executable that works
> (I checked it) but only guarantees that the editor works. Other
> components, such as Gnus, the web browser, etc, may fail due to missing
> tools (e.g. Gnus fails because gnu-tls is missing).

Yes, that's right.

Re: Process to build standalone Emacs + deps in Windows

[Phillip Lord]

Juan José García-Ripoll <juanjose.garciaripoll@gmail.com> writes:

> Eli Zaretskii <eliz@gnu.org> writes:
>> If this is indeed so, it's a bug in the build procedure.  All the
>> optional DLLs (a.k.a. "dependencies") are loaded dynamically at run
>> time, and if some DLL is unavailable, the corresponding features will
>> fail, but Emacs will still run "by itself" and be a useful editor.
>
> Sorry, it seems I misunderstood the *-no-deps concept deeply. I had
> always assumed that *-no-deps whas a zip to be installed on top of the
> *-deps one.


Other way around. If you unpack the deps zip (emacs-27-x86_64-deps.zip) and
the no-deps emacs (emacs-27.0.90-2-x86_64-no-deps.zip) into the same
structure, you would get the same as emacs-27.0.90-2-x86_64.zip.

For early versions of Emacs, (emacs-25) we had emacs-25.zip and then a
"deps" file that you unpacked. But I wanted the "with-dependencies" to
be the one with the simplest name, because that's what people would
choose most often.

I think -no-deps is probably most useful for people who want to unpack
it over a msys2 installation that they already have. Unfortunately I
don't have information and who and what people are actually using.

Phil

Re: Process to build standalone Emacs + deps in Windows

[Eli Zaretskii]

> From: Phillip Lord <phillip.lord@russet.org.uk>
> Date: Sat, 28 Mar 2020 19:36:05 +0000
> Cc: emacs-devel@gnu.org
> 
> I think -no-deps is probably most useful for people who want to unpack
> it over a msys2 installation that they already have.

It should also be useful for someone who wants to determine by
themselves which optional libraries they install and which they don't.

Re: Process to build standalone Emacs + deps in Windows

[Phillip Lord]

Juan José García-Ripoll <juanjose.garciaripoll@gmail.com> writes:

> Phillip Lord <phillip.lord@russet.org.uk> writes:
>> In reply to two of your emails, sorry:
>>
>> +    if dependencies:
>> +        print(f"Package {pkg} depends on:")
>> +        for d in dependencies:
>> +            print(f" -> {d}")
>> +
>>
>> I wouldn't add this. It's only useful some of the time, not in normal
>> usage, and anyway "pactree" does the same thing better.
>
> Right now the scripts do not provide this information but do emit a line
> "Adding *package name*" I personally find it more informative to at
> least instruct why the package is added. If pactree is more useful, it
> is not used anywhere in the current code.

pactree gives a graphical picture of all the dependencies. It's not used
in the current code, since I refactored it to drop a lot of
dependencies. But, as a tool it will still tell you what depends on what.


>> +    ## exclude files
>> +    excludes=['lib/*.a', # No files to link against
>> +              'lib/*/*.exe', # No hidden executables
>> +              'libexec/*/*.exe',
>>  [...]
>>
>> This list would need to go near the front, so we have a clean
>> configuration section. It also displays the key problem -- it's really
>> long and non obvious so a maintenance burden
>
> Not really. The *.exe can be deduced from the pacman files.


Not sure I understand here to be honest.


>> If we just did a few big top level directories, would that not solve all
>> the problems.
>
> I proposed to remove complete directories but it seems it is not an option.
>
>> +# The dependency extraction relies on an English version of pacman
>> +# We need to configure the locale to match the expectations
>> +if 'LANG' in os.environ:
>> +    os_lang = os.environ['LANG']
>> +    if (len(os_lang) > 2) and (os_lang[0:2] != 'en'):
>> +        os.environ['LANG']='en_US'
>> I am vaguely curious what it looks like in Spanish. I thought the output
>> was fairly mechanistic.
>
> Your filter relies on the English word "depends on"
>     ## Extract the "Depends On" line
>     depends_on = [x for x in package_info if x.startswith("Depends On")][0]
> This is heavily locale-dependent

Oh yeah, that's funny.



>> - Since we are pulling in compression utilities, we may just as well tell the
>>   configuration to compress the *.el files. Windows is the only platform where
>>   this does not happen
>>
>> No, because I generate the no-deps version first, then add the deps. The
>> -no-deps version doesn't have the compression utilities. Of course, we
>> do not need the *.el files at all.
>
> The no-deps version does not have the compression utilities but it is
> not intended to be used by itself, as it depends on all the *.dll Hence,
> one may argue that it can be built to support compression.


The no-deps version should work fine all by itself. Indeed, this used to
be the only form of Emacs distribution for windows. It just won't do
some things.


>
>>   b) I realized that Cairo is pulled in as a dependency, but Emacs is
>>   built without support for Cairo. Is this intentional?
>>
>> No, it will be a transitive dependency of something. pactree will
>> reveal!
>
> My simple lines instructing what depends on what showed this: it is
> pulled in by rsvg. There were some earlier emails about this.


Yes, I know.

Phil

Re: Process to build standalone Emacs + deps in Windows

[Eli Zaretskii]

> From: Juan José García-Ripoll
>  <juanjose.garciaripoll@gmail.com>
> Date: Wed, 25 Mar 2020 15:43:24 +0100
> 
> 4. It creates a zip file where many of the superfluous directories and
> executables have been removed. I would go for removing even more, since
> it seems that Emacs is built with zlib and we could get rid of
> b*zip*.exe, xz.exe, etc.

No, zlib doesn't replace bzip2 and xz, you need additional libraries
for that (and Emacs currently cannot use such libraries directly
anyway).

> b) I realized that Cairo is pulled in as a dependency, but Emacs is
> built without support for Cairo. Is this intentional?

It's a side effect of the fact that librsvg needs Cairo.  We don't
(yet) have support for drawing with Cairo on MS-Windows directly from
Emacs.

Re: Process to build standalone Emacs + deps in Windows

[Phillip Lord]

Juan José García-Ripoll <juanjose.garciaripoll@gmail.com> writes:

> Now, to motivate my intentions a bit better, I have built Emacs from
> savannah's git repository following a different set of instructions
> (https://gist.github.com/nauhygon/f3b44f51b34e89bc54f8). These
> instructions are hacky (cp /mingw64/bin/*.dll $target/bin).

Incidentally, those instructions are "modified from emacs-w64 Wiki page
by zklhp." It what the Emacs-w64 instructions that I used way back when
as the basis for the build-zips.sh file which makes the 64 bit
build. What goes around, comes around.

Phil

Re: Process to build standalone Emacs + deps in Windows

[phillip.lord]

On 2020-03-22 13:03, juanjose.garciaripoll@gmail.com wrote:
> Hi,
> 
> I am trying to debug the process of building a distributable Emacs with
> dependencies on Windows 64 bits. I have seen some sketch of
> documentation under nt/ but it does not include support for graphics 
> and
> image formats, which the official distribution for Windows includes.
> 
> What are the steps or the scripts used to create the *.zip files that
> are found in the official repository? Is there a simple guide I can use
> to reproduce it?
> 
> Best,


The documentation is in admin/nt/dist-build/README-scripts

It's a little long-winded to set up, I am afraid, but once you have the 
right directory structure, it's one command.

Let me know if the instructions still work. I mostly run them on a 
single machine that I keep for the purpose, so I have not had to run 
them through de-novo for quite a while.

Phil