bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 4286 bytes --]

The script nt/admin/dist-build/build-deps-zips.py needs help.  This is
the script that I use to collect and package dependencies and sources
for dependencies on Microsoft Windows, as part of releasing Emacs
binaries for Windows.  It is a python script that runs under MSYS2
MSYS console (not MinGW).

Neither the version currently in the emacs-29 nor in the master
branches will work for the given Emacs version without changes.  The
attached patch would make emacs-29 match what I am using locally.

In addition to other changes, the patch reflects my current "transformation map"
approach to deal with MSYS source package paths change, which seems to
be happening quite a bit upstream.

In case it may not be clear, my process is to run the script
after updating local MSYS packages that are dependencies (optional or
no), or edit and run it when Emacs' dependencies have changed.

The patch reflects the script as I have been using it during the Emacs
29 release process.  I'm sure there's general room for improvement
(editing this script is literally my only python coding credit), I'm
opening this bug report because bug#65188 (a packaging error preventing
WEBP from working for people using the Windows binaries) has called
attention to the importance of having additional eyes on build tooling
(especially when it so far contains hard-coded lists of upstream deps).

In GNU Emacs 29.1 (build 2, x86_64-w64-mingw32) of 2023-08-02 built on
 AVALON
Windowing system distributor 'Microsoft Corp.', version 10.0.19045
System Description: Microsoft Windows 10 Home (v10.0.2009.19045.3324)

Configured using:
 'configure --with-modules --without-dbus --with-native-compilation=aot
 --without-compress-install --with-tree-sitter CFLAGS=-O2'

Configured features:
ACL GIF GMP GNUTLS HARFBUZZ JPEG JSON LCMS2 LIBXML2 MODULES NATIVE_COMP
NOTIFY W32NOTIFY PDUMPER PNG RSVG SOUND SQLITE3 THREADS TIFF
TOOLKIT_SCROLL_BARS TREE_SITTER WEBP XPM ZLIB

Important settings:
  value of $LANG: ENU
  locale-coding-system: cp1252

Major mode: Lisp Interaction

Minor modes in effect:
  tooltip-mode: t
  global-eldoc-mode: t
  eldoc-mode: t
  show-paren-mode: t
  electric-indent-mode: t
  mouse-wheel-mode: t
  tool-bar-mode: t
  menu-bar-mode: t
  file-name-shadow-mode: t
  global-font-lock-mode: t
  font-lock-mode: t
  blink-cursor-mode: t
  line-number-mode: t
  indent-tabs-mode: t
  transient-mark-mode: t
  auto-composition-mode: t
  auto-encryption-mode: t
  auto-compression-mode: t

Load-path shadows:
None found.

Features:
(shadow sort mail-extr emacsbug message mailcap yank-media puny dired
dired-loaddefs rfc822 mml mml-sec password-cache epa derived epg rfc6068
epg-config gnus-util text-property-search time-date mm-decode mm-bodies
mm-encode mail-parse rfc2231 mailabbrev gmm-utils mailheader cl-loaddefs
comp comp-cstr warnings icons subr-x rx cl-seq cl-macs gv cl-extra
help-mode bytecomp byte-compile cl-lib sendmail rfc2047 rfc2045
ietf-drums mm-util mail-prsvr mail-utils rmc iso-transl tooltip cconv
eldoc paren electric uniquify ediff-hook vc-hooks lisp-float-type
elisp-mode mwheel dos-w32 ls-lisp disp-table term/w32-win w32-win
w32-vars term/common-win tool-bar dnd fontset image regexp-opt fringe
tabulated-list replace newcomment text-mode lisp-mode prog-mode register
page tab-bar menu-bar rfn-eshadow isearch easymenu timer select
scroll-bar mouse jit-lock font-lock syntax font-core term/tty-colors
frame minibuffer nadvice seq simple cl-generic indonesian philippine
cham georgian utf-8-lang misc-lang vietnamese tibetan thai tai-viet lao
korean japanese eucjp-ms cp51932 hebrew greek romanian slovak czech
european ethiopic indian cyrillic chinese composite emoji-zwj charscript
charprop case-table epa-hook jka-cmpr-hook help abbrev obarray oclosure
cl-preloaded button loaddefs theme-loaddefs faces cus-face macroexp
files window text-properties overlay sha1 md5 base64 format env
code-pages mule custom widget keymap hashtable-print-readable backquote
threads w32notify w32 lcms2 multi-tty make-network-process
native-compile emacs)

Memory information:
((conses 16 80415 13861)
 (symbols 48 7175 0)
 (strings 32 21269 1764)
 (string-bytes 1 617449)
 (vectors 16 16398)
 (vector-slots 8 334731 15794)
 (floats 8 29 46)
 (intervals 56 238 0)
 (buffers 984 10))

[-- Attachment #2: 0001-Fix-Windows-build-dependancy-packaging-for-Emacs-29-.patch --]
[-- Type: application/octet-stream, Size: 5790 bytes --]

From 932320cda4d7633cb48bb89163fb9cf62c5e208d Mon Sep 17 00:00:00 2001
From: Corwin Brust <corwin@bru.st>
Date: Thu, 10 Aug 2023 07:19:41 -0500
Subject: [PATCH] ; Fix Windows build dependancy packaging for Emacs 29 and 30

* nt/admin/dist-build/build-deps-zips.py (script): add webp,
Xpm, Xpm-noX4, treesitter, and sqlite4, bump EMACS_MAJOR_VERSION,
remove unneeded imports, change vendor slug from msys64 to mingw64,
skip some ancient certificates, add SRC_EXT to map transformation
source package name transformations vs historical convention.
---
 admin/nt/dist-build/build-dep-zips.py | 84 ++++++++++++++++++---------
 1 file changed, 57 insertions(+), 27 deletions(-)

diff --git a/admin/nt/dist-build/build-dep-zips.py b/admin/nt/dist-build/build-dep-zips.py
index 71105a071ec..a7c116d9b41 100755
--- a/admin/nt/dist-build/build-dep-zips.py
+++ b/admin/nt/dist-build/build-dep-zips.py
@@ -20,13 +20,11 @@
 import os
 import shutil
 import re
-import functools
-import operator
 
 from subprocess import check_output
 
 ## Constants
-EMACS_MAJOR_VERSION="28"
+EMACS_MAJOR_VERSION="29"
 
 # This list derives from the features we want Emacs to compile with.
 PKG_REQ='''mingw-w64-x86_64-giflib
@@ -37,9 +35,13 @@
 mingw-w64-x86_64-libjpeg-turbo
 mingw-w64-x86_64-libpng
 mingw-w64-x86_64-librsvg
+mingw-w64-x86_64-libwebp
 mingw-w64-x86_64-libtiff
 mingw-w64-x86_64-libxml2
-mingw-w64-x86_64-xpm-nox'''.split()
+mingw-w64-x86_64-gmp
+mingw-w64-x86_64-xpm-nox
+mingw-w64-x86_64-tree-sitter
+mingw-w64-x86_64-sqlite3'''.split()
 
 DLL_REQ='''libgif
 libgnutls
@@ -49,9 +51,14 @@
 libturbojpeg
 libpng
 librsvg
+libwebp
 libtiff
 libxml
-libXpm'''.split()
+libgmp
+libXpm
+libXpm-noX4
+libtree-sitter
+libsqlite3-0'''.split()
 
 
 ## Options
@@ -103,7 +110,7 @@ def ntldd_munge(out):
 
         ## if it's the former, we want it, if its the later we don't
         splt = dep.split()
-        if len(splt) > 2 and "msys64" in splt[2]:
+        if len(splt) > 2 and "mingw64" in splt[2]:
             print("Adding dep", splt[0])
             rtn.append(splt[0].split(".")[0])
 
@@ -114,26 +121,45 @@ def ntldd_munge(out):
 ## Packages to fiddle with
 ## Source for gcc-libs is part of gcc
 SKIP_SRC_PKGS=["mingw-w64-gcc-libs"]
-SKIP_DEP_PKGS=frozenset(["mingw-w64-x86_64-glib2"])
+SKIP_DEP_PKGS=["mingw-w64-glib2" "mingw-w64-ca-certificates-20211016-3"]
 MUNGE_SRC_PKGS={"mingw-w64-libwinpthread-git":"mingw-w64-winpthreads-git"}
 MUNGE_DEP_PKGS={
     "mingw-w64-x86_64-libwinpthread":"mingw-w64-x86_64-libwinpthread-git",
     "mingw-w64-x86_64-libtre": "mingw-w64-x86_64-libtre-git",
 }
+SRC_EXT={
+    "mingw-w64-freetype": ".src.tar.zst",
+    "mingw-w64-fribidi": ".src.tar.zst",
+    "mingw-w64-glib2": ".src.tar.zst",
+    "mingw-w64-harfbuzz": ".src.tar.zst",
+    "mingw-w64-libunistring": ".src.tar.zst",
+    "mingw-w64-winpthreads-git": ".src.tar.zst",
+    "mingw-w64-ca-certificates": ".src.tar.zst",
+    "mingw-w64-libxml2": ".src.tar.zst",
+    "mingw-w64-ncurses": ".src.tar.zst",
+    "mingw-w64-openssl": ".src.tar.zst",
+    "mingw-w64-pango": ".src.tar.zst",
+    "mingw-w64-python": ".src.tar.zst",
+    "mingw-w64-sqlite3": ".src.tar.zst",
+    "mingw-w64-xpm-nox": ".src.tar.zst",
+    "mingw-w64-xz": ".src.tar.zst",
+}
 
 ## Currently no packages seem to require this!
 ARCH_PKGS=[]
 SRC_REPO="https://repo.msys2.org/mingw/sources"
 
 
-def immediate_deps(pkgs):
-    package_info = check_output(["pacman", "-Si"] + pkgs).decode("utf-8").splitlines()
+def immediate_deps(pkg):
+    package_info = check_output(["pacman", "-Si", pkg]).decode("utf-8").split("\n")
+
+    ## Extract the "Depends On" line
+    depends_on = [x for x in package_info if x.startswith("Depends On")][0]
+    ## Remove "Depends On" prefix
+    dependencies = depends_on.split(":")[1]
 
-    ## Extract the packages listed for "Depends On:" lines.
-    dependencies = [line.split(":")[1].split() for line in package_info
-                    if line.startswith("Depends On")]
-    ## Flatten dependency lists from multiple packages into one list.
-    dependencies = functools.reduce(operator.iconcat, dependencies, [])
+    ## Split into dependencies
+    dependencies = dependencies.strip().split(" ")
 
     ## Remove > signs TODO can we get any other punctuation here?
     dependencies = [d.split(">")[0] for d in dependencies if d]
@@ -149,18 +175,16 @@ def extract_deps():
     print( "Extracting deps" )
 
     # Get a list of all dependencies needed for packages mentioned above.
-    pkgs = set(PKG_REQ)
-    newdeps = pkgs
-    print("adding...")
-    while True:
-        subdeps = frozenset(immediate_deps(list(newdeps)))
-        newdeps = subdeps - SKIP_DEP_PKGS - pkgs
-        if not newdeps:
-            break
-        print('\n'.join(newdeps))
-        pkgs |= newdeps
+    pkgs = PKG_REQ[:]
+    n = 0
+    while n < len(pkgs):
+        subdeps = immediate_deps(pkgs[n])
+        for p in subdeps:
+            if not (p in pkgs or p in SKIP_DEP_PKGS):
+                pkgs.append(p)
+        n = n + 1
 
-    return list(pkgs)
+    return sorted(pkgs)
 
 
 def download_source(tarball):
@@ -208,7 +232,13 @@ def gather_source(deps):
         ## Switch names if necessary
         pkg_name = MUNGE_SRC_PKGS.get(pkg_name,pkg_name)
 
-        tarball = "{}-{}.src.tar.gz".format(pkg_name,pkg_version)
+        ## src archive is usually a .tar.gz
+        if pkg_name in SRC_EXT.keys():
+            src_ext = SRC_EXT[pkg_name]
+        else:
+            src_ext = ".src.tar.gz"
+
+        tarball = "{}-{}{}".format(pkg_name,pkg_version,src_ext)
 
         download_source(tarball)
 
@@ -257,7 +287,7 @@ def clean():
 
 if( args.l ):
     print("List of dependencies")
-    print( deps )
+    print( extract_deps() )
     exit(0)
 
 if args.s:
-- 
2.41.0.windows.1

bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Thu, 10 Aug 2023 07:40:48 -0500
> 
> The script nt/admin/dist-build/build-deps-zips.py needs help.  This is
> the script that I use to collect and package dependencies and sources
> for dependencies on Microsoft Windows, as part of releasing Emacs
> binaries for Windows.  It is a python script that runs under MSYS2
> MSYS console (not MinGW).
> 
> Neither the version currently in the emacs-29 nor in the master
> branches will work for the given Emacs version without changes.  The
> attached patch would make emacs-29 match what I am using locally.
> 
> In addition to other changes, the patch reflects my current "transformation map"
> approach to deal with MSYS source package paths change, which seems to
> be happening quite a bit upstream.
> 
> In case it may not be clear, my process is to run the script
> after updating local MSYS packages that are dependencies (optional or
> no), or edit and run it when Emacs' dependencies have changed.
> 
> The patch reflects the script as I have been using it during the Emacs
> 29 release process.  I'm sure there's general room for improvement
> (editing this script is literally my only python coding credit), I'm
> opening this bug report because bug#65188 (a packaging error preventing
> WEBP from working for people using the Windows binaries) has called
> attention to the importance of having additional eyes on build tooling
> (especially when it so far contains hard-coded lists of upstream deps).

Would you mind providing an overview of the process by which the
script (and maybe some additional measures) collect(s) the list of the
dependency packages for the binary distro, including the main ideas
and information sources?  It is hard to glean all that from just a
patch, or even by reading the script, and I think that, given the
pains this gives, perhaps some new ideas are in order.

Thanks.

bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Corwin Brust]

On Thu, Aug 10, 2023 at 8:29 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
>
> Would you mind providing an overview of the process by which the
> script (and maybe some additional measures) collect(s) the list of the
> dependency packages for the binary distro, including the main ideas
> and information sources?  It is hard to glean all that from just a
> patch, or even by reading the script, and I think that, given the
> pains this gives, perhaps some new ideas are in order.

I will happily try my best; sorry that it cannot be immediately/today.
In any case, I completely agree with you.

bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Corwin Brust]

On Thu, Aug 10, 2023 at 8:29 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> [Provide] an overview of the process by which

nt/admin/dist-build/build-dep-zips.py - it's really just that, as far
as how I collect the deps presently

To be clear, by "deps", I'm referring to DLLs that are not built with
Emacs but which we should (optionally) include when distributing
binary versions of Emacs for Windows.

Similarly pedantically, here are the files I (might) include for a
given release of Windows binaries for Emacs:

FULL ZIP: emacs-${VER}.zip - the "full zip", includes the deps DLLs
this script packages
NO DEPS ZIP: emacs-${VER}-no-deps.zip - specifically without the deps
DLLs this script packages
INSTALLER EXE: emacs-${VERION}-installer.exe - a self-installer
including the deps DLLs (and it compresses stuff even more than zip
-9. nice)
DEPS ZIP: emacs-${MAJOR_VERSION}-${DEPS_CREATE_DATE_IF_NOT_FIRST_AND_ONLY_FOR_MAJOR_VERSION}.zip
- contains only the deps DLLs. Created when changes are needed, more
to come on this.
DEPS SOURCES ZIP: the MSYS2/MINGW64 source code for all DLLs in DEPS
ZIP. Created only when a new DEPS ZIP is created

Finally (I swear, I'm going to answer your actual question), to place
the script in context of my present "use-case" (release and
release-similar builds specifically, here):

0. PREREQ: I have various folders setup, a working MSYS/MINGW64 that
has recently built me a working Emacs, I've been watching mailing
lists, I have ACLs going for me, etc.

1. A (pre)release tarball is pushed to GNU (alpha) FTP.

2. Skip this step if a "last good" deps and deps source file exist and
it is current (i.e. when there are not known new or updated
--including optional-- dependendencies for Emacs to include for the
first time),  Otherwise, update and run the script (more on this
below, obviously) in question to create (new) deps and deps source
archives containing, respectively, a bunch of DLLs we plan to
distribute (nominally) with Emacs (this is the deps archive), and the
MSYS2/MINGW64 sources for these DLLs and their compile time
dependencies (the deps sources archive).

3. Download and unpack the tarball manually

4. Configure && make install  to ${PREFIX}. Note, I have a tedious
pipeline command I like for this; I'm not currently using
"build-zips.sh" from the same folder as the build-dep-zips.sh script
in question.

5. Create a no-deps archive, essentially:

  zip -r9 ${PREFIX} emacs-${TARGET_VERSION_NAME}-no-deps.zip

6. Unpack the deps archive created in step #2 (or the "last good", if
that step was skipped), something like:

  unzip -d ${PREFIX}/bin ${LAST_GOOD_DEPS_ZIP_FOR_EMACS_MAJOR_VERSION}

7. Create a "full" zip of Emacs (now including those "extra" DLLs),
essentially repeating step #5 with a different archive file name  (in
fact, I copy no-deps and re-add bin to it)

8. Create the installer

9. Perform certain rights and incantations to cause files to appear on
GNU (alpha) FTP servers.  Notably, include new deps files ONLY when
they have been newly created along with the release being published.
These files have historically changed only rarely within a given Emacs
major version.

> collect(s) the list of the dependency packages for the binary distro,

Ignoring my patch for the moment, the script contains two hard-coded
lists that represent our first likely source of breakage:

DLL_REQ lists specific DLLs that should be copied into ${PREFIX}/bin,
after make install, to get a "complete" Emacs installation.  During
the Emacs 29 pre-release cycle I added sqlite3 and tree-sitter to this
list, enabling those features to work "out of the box".  In some
cases, such as these two in fact, Emacs would likely function
correctly under Windows if we chose not to distribute a particular
DLL; however, I believe this is not the case for all of DLLs included
and, moreover (in my opinion) would tend to make Emacs to less viable
as a means of drawing Windows users closer to Free Software by virtue
of Just Being Better, which would be a bit sad.

PKG_REQ lists the mingw-w64-x86_64 source package name for each item
in the first list.

These two variables are coordinated lists and so obviously could be
refactored (e.g. into an associative array) that I suspect historical
raisins (meaning, perhaps once these lists were not so coordinated; I
didn't research this so far).

In addition to these "main lists" there are a few other vars/lines to
study (now taking from my patched version):

SKIP_SRC_PKGS=["mingw-w64-gcc-libs"]
SKIP_DEP_PKGS=["mingw-w64-glib2" "mingw-w64-ca-certificates-20211016-3"]

We'll come to the logic -where all of these are used-- next.

> including the main ideas and information sources[.  ...]

Ignoring as much complexity as possible by focusing on the
release(like) use-case, we can ignore the arguments and conditions
and, finally, we can reduce the the script to:

2.1 evaluate above mentioned vars
2.2 collect all DLLs mentioned in DLL_REQ
2.3 collect the DLLs that are unique dependencies the DLLs collected
in 2.2 skipping any which appear in SKIP_DEP_PKGS
2.4 collect the source for all DLLs from 2.2 and 2.3 unless the source
package is listed in SKIP_SRC_PKGS

I hope I did cover information sources well enough, but to put a fine
point on it: I watch Emacs devel, bug reports, IRC, reddit, ..., for
information leading to updates to this script. Developers and others
using Emacs on Windows are the main "information sources", at least
speaking from developing that patch :)

>
> Thanks.
>

bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Tue, 15 Aug 2023 02:39:45 -0500
> Cc: 65206@debbugs.gnu.org
> 
> 5. Create a no-deps archive, essentially:
> 
>   zip -r9 ${PREFIX} emacs-${TARGET_VERSION_NAME}-no-deps.zip
> 
> 6. Unpack the deps archive created in step #2 (or the "last good", if
> that step was skipped), something like:
> 
>   unzip -d ${PREFIX}/bin ${LAST_GOOD_DEPS_ZIP_FOR_EMACS_MAJOR_VERSION}
> 
> 7. Create a "full" zip of Emacs (now including those "extra" DLLs),
> essentially repeating step #5 with a different archive file name  (in
> fact, I copy no-deps and re-add bin to it)
> 
> 8. Create the installer
> 
> 9. Perform certain rights and incantations to cause files to appear on
> GNU (alpha) FTP servers.  Notably, include new deps files ONLY when
> they have been newly created along with the release being published.
> These files have historically changed only rarely within a given Emacs
> major version.
> 
> > collect(s) the list of the dependency packages for the binary distro,
> 
> Ignoring my patch for the moment, the script contains two hard-coded
> lists that represent our first likely source of breakage:
> 
> DLL_REQ lists specific DLLs that should be copied into ${PREFIX}/bin,
> after make install, to get a "complete" Emacs installation.  During
> the Emacs 29 pre-release cycle I added sqlite3 and tree-sitter to this
> list, enabling those features to work "out of the box".  In some
> cases, such as these two in fact, Emacs would likely function
> correctly under Windows if we chose not to distribute a particular
> DLL; however, I believe this is not the case for all of DLLs included
> and, moreover (in my opinion) would tend to make Emacs to less viable
> as a means of drawing Windows users closer to Free Software by virtue
> of Just Being Better, which would be a bit sad.
> 
> PKG_REQ lists the mingw-w64-x86_64 source package name for each item
> in the first list.
> 
> These two variables are coordinated lists and so obviously could be
> refactored (e.g. into an associative array) that I suspect historical
> raisins (meaning, perhaps once these lists were not so coordinated; I
> didn't research this so far).
> 
> In addition to these "main lists" there are a few other vars/lines to
> study (now taking from my patched version):
> 
> SKIP_SRC_PKGS=["mingw-w64-gcc-libs"]
> SKIP_DEP_PKGS=["mingw-w64-glib2" "mingw-w64-ca-certificates-20211016-3"]
> 
> We'll come to the logic -where all of these are used-- next.
> 
> > including the main ideas and information sources[.  ...]
> 
> Ignoring as much complexity as possible by focusing on the
> release(like) use-case, we can ignore the arguments and conditions
> and, finally, we can reduce the the script to:
> 
> 2.1 evaluate above mentioned vars
> 2.2 collect all DLLs mentioned in DLL_REQ
> 2.3 collect the DLLs that are unique dependencies the DLLs collected
> in 2.2 skipping any which appear in SKIP_DEP_PKGS
> 2.4 collect the source for all DLLs from 2.2 and 2.3 unless the source
> package is listed in SKIP_SRC_PKGS
> 
> I hope I did cover information sources well enough, but to put a fine
> point on it: I watch Emacs devel, bug reports, IRC, reddit, ..., for
> information leading to updates to this script. Developers and others
> using Emacs on Windows are the main "information sources", at least
> speaking from developing that patch :)

Thanks.  What I still don't think I understand is how do you make sure
you have a full list of first-order dependencies?  I understand that
you mostly build on the "last good" list from previous release, but
since the list grows from time to time, what is the procedure for
finding the new dependencies, adding them to the list, and making sure
they all are there?

I'm asking because this is exactly where the procedure broke down when
we added WebP image support in Emacs 29.

bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Corwin Brust]

On Tue, Aug 15, 2023 at 10:43 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Corwin Brust <corwin@bru.st>
> >
> > I hope I did cover information sources well enough, but to put a fine
> > point on it: I watch Emacs devel, bug reports, IRC, reddit, ..., for
> > information leading to updates to this script. Developers and others
> > using Emacs on Windows are the main "information sources", at least
> > speaking from developing that patch :)
>
> Thanks.  What I still don't think I understand is how do you make sure
> you have a full list of first-order dependencies?  I understand that
> you mostly build on the "last good" list from previous release, but
> since the list grows from time to time, what is the procedure for
> finding the new dependencies, adding them to the list, and making sure
> they all are there?
>
> I'm asking because this is exactly where the procedure broke down when
> we added WebP image support in Emacs 29.

The above quote from myself I retained is my best/only answer:

I update the script as issues are reported or when I somehow otherwise
learn that changes are needed.  I have no real process for this, and
nothing in the tooling is helping me with it, so far.

bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Tue, 15 Aug 2023 10:53:55 -0500
> Cc: 65206@debbugs.gnu.org
> 
> > Thanks.  What I still don't think I understand is how do you make sure
> > you have a full list of first-order dependencies?  I understand that
> > you mostly build on the "last good" list from previous release, but
> > since the list grows from time to time, what is the procedure for
> > finding the new dependencies, adding them to the list, and making sure
> > they all are there?
> >
> > I'm asking because this is exactly where the procedure broke down when
> > we added WebP image support in Emacs 29.
> 
> The above quote from myself I retained is my best/only answer:
> 
> I update the script as issues are reported or when I somehow otherwise
> learn that changes are needed.  I have no real process for this, and
> nothing in the tooling is helping me with it, so far.

OK, so here's a suggestion which might improve that crucial part: scan
the list in dynamic-library-alist, on lisp/term/w32-win.el.  Every
dependency that is loaded dynamically (i.e., Emacs is not linked
against it when it is built) must be in that list.  So when we add
dependencies, we add them there.

I believe that given a full and complete lest of first-order
dependencies, those which Emacs actually loads, all the higher-order
dependencies can be found by following the MSYS2 pacman etc., is that
right?

bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Corwin Brust]

On Tue, Aug 15, 2023 at 11:01 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> >
> > I update the script as issues are reported or when I somehow otherwise
> > learn that changes are needed.  I have no real process for this, and
> > nothing in the tooling is helping me with it, so far.
>
> OK, so here's a suggestion which might improve that crucial part: scan
> the list in dynamic-library-alist, on lisp/term/w32-win.el.  Every
> dependency that is loaded dynamically (i.e., Emacs is not linked
> against it when it is built) must be in that list.  So when we add
> dependencies, we add them there.

I looked at the variable.   OT1H, it serves a very different use-case
("what are valid DLL names for a given library?" in the run-time, vs
"what DLLs should be sent along with Emacs?" in the build-time).
This means that meaningful hackery would likely be needed to
contemplate removing the hard-coded list completely, or even that we
would not be able to device any means of parsing this and choosing the
correct sent among DLLs present on the build system, to include.

OTOH, and more directly to the point of this bug report:

>
> I believe that given a full and complete lest of first-order
> dependencies, those which Emacs actually loads, all the higher-order
> dependencies can be found by following the MSYS2 pacman etc., is that
> right?
>

Right, that is how the script presently works:

    package_info = check_output(["pacman", "-Si",
pkg]).decode("utf-8").split("\n")

Thus, if we are content to have the script detect, and error demanding
correction when out of sync wrt `dynamic-library-alist', I believe it
can be done.  Moreover, IMO this will definitely help.

One process note is that I would likely switch to (maybe) generating
new DEPS right after creating NO DEPS, so the script can count on
invoking the freshly compiled and locally installed Emacs.  This also
seems much easier and also more proper vs anything to parse it "the
hard way".  (Especially so, considering complexities such as format
strings and the libpng version comment on the source setting up the
default value of dynamic-module-alist.)

Does a "invokes Emacs now and errors out if stuff is missing" approach
sound right/good?
Is it too soon for me to try making a new patch that explores this idea further?

bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Tue, 15 Aug 2023 20:23:44 -0500
> Cc: 65206@debbugs.gnu.org
> 
> On Tue, Aug 15, 2023 at 11:01 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > OK, so here's a suggestion which might improve that crucial part: scan
> > the list in dynamic-library-alist, on lisp/term/w32-win.el.  Every
> > dependency that is loaded dynamically (i.e., Emacs is not linked
> > against it when it is built) must be in that list.  So when we add
> > dependencies, we add them there.
> 
> I looked at the variable.   OT1H, it serves a very different use-case
> ("what are valid DLL names for a given library?" in the run-time, vs
> "what DLLs should be sent along with Emacs?" in the build-time).
> This means that meaningful hackery would likely be needed to
> contemplate removing the hard-coded list completely, or even that we
> would not be able to device any means of parsing this and choosing the
> correct sent among DLLs present on the build system, to include.

I'm not sure I understand the reservation.  That list mentions every
single DLL that we know can be used for each optional feature.  If a
feature has more than one DLL listed, the first one is usually the
most popular, and should be tried first.  Given this, what problems do
you envision with using that list?

> Thus, if we are content to have the script detect, and error demanding
> correction when out of sync wrt `dynamic-library-alist', I believe it
> can be done.  Moreover, IMO this will definitely help.

Great, that's what I hoped to achieve: a way of verifying that your
list of first-order dependencies is complete.

> Does a "invokes Emacs now and errors out if stuff is missing" approach
> sound right/good?

I'm not sure I understand how would you force Emacs to "error out"
when we are talking about optional dependencies.  They are optional so
that Emacs could run even if they are not present.

bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Corwin Brust]

On Wed, Aug 16, 2023 at 7:08 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
> > From: Corwin Brust <corwin@bru.st>
> > Date: Tue, 15 Aug 2023 20:23:44 -0500
> > Cc: 65206@debbugs.gnu.org
> >
> > On Tue, Aug 15, 2023 at 11:01 AM Eli Zaretskii <eliz@gnu.org> wrote:
> > >
> > > the list in dynamic-library-alist, on lisp/term/w32-win.el.
>
> I'm not sure I understand the reservation.  That list mentions every
> single DLL that we know can be used for each optional feature.  If a
> feature has more than one DLL listed, the first one is usually the
> most popular, and should be tried first.

This solves my worry completely, or nearly so.

To confirm: when walking the list, I will want to take the first DLL
mentioned that actually exists for each entry. Is that right?

> Given this, what problems do you envision with using that list?

There might not be a problem (except the one we are trying to fix).
The alist contains 22 entries, while var DLL_REQ contains 14 entries.
The five on the alist but on mentioned in the script (so far) are:

gdiplus
shlwapi
gobject
gio
webpdemux - this is pretty obviously a miss in the script; it does get
however because it's required by webp which is listed in DLL_REQ

Are all of these errors with the script (so, the corresponding DLLs
should be included)?  If not, I think we will need a way for the
script to know which alist entries to skip/ignore.

> > Does a "invokes Emacs now and errors out if stuff is missing" approach
> > sound right/good?
>
> I'm not sure I understand how would you force Emacs to "error out"
> when we are talking about optional dependencies.  They are optional so
> that Emacs could run even if they are not present.
>

Oops, badly said:  I mean that my build and packaging process should
stop and report an error if it cannot create a "complete" DEPS ZIP.
Nothing affecting the Emacs run-time.

bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Wed, 16 Aug 2023 08:41:25 -0500
> Cc: 65206@debbugs.gnu.org
> 
> On Wed, Aug 16, 2023 at 7:08 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > I'm not sure I understand the reservation.  That list mentions every
> > single DLL that we know can be used for each optional feature.  If a
> > feature has more than one DLL listed, the first one is usually the
> > most popular, and should be tried first.
> 
> This solves my worry completely, or nearly so.
> 
> To confirm: when walking the list, I will want to take the first DLL
> mentioned that actually exists for each entry. Is that right?

Yes.

> There might not be a problem (except the one we are trying to fix).
> The alist contains 22 entries, while var DLL_REQ contains 14 entries.
> The five on the alist but on mentioned in the script (so far) are:
> 
> gdiplus
> shlwapi

You can ignore those: they are for Windows 9X, and they are system
DLLs.

> gobject
> gio

These are needed for librsvg.  You might get away with them because
librsvg pulls them in as second-order dependencies.

> webpdemux - this is pretty obviously a miss in the script; it does get
> however because it's required by webp which is listed in DLL_REQ

Yes, this is required by libwebp, since some of the library functions
are in lobwebpdemux.

> Are all of these errors with the script (so, the corresponding DLLs
> should be included)?  If not, I think we will need a way for the
> script to know which alist entries to skip/ignore.

You should only skip the first two, which are Windows system DLLs.

> > > Does a "invokes Emacs now and errors out if stuff is missing" approach
> > > sound right/good?
> >
> > I'm not sure I understand how would you force Emacs to "error out"
> > when we are talking about optional dependencies.  They are optional so
> > that Emacs could run even if they are not present.
> >
> 
> Oops, badly said:  I mean that my build and packaging process should
> stop and report an error if it cannot create a "complete" DEPS ZIP.
> Nothing affecting the Emacs run-time.

OK.

bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Corwin Brust]

[-- Attachment #1: Type: text/plain, Size: 2456 bytes --]

On Wed, Aug 16, 2023 at 9:49 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > To confirm: when walking the list, I will want to take the first DLL
> > mentioned that actually exists for each entry. Is that right?
>
> Yes.
>

The attached patch replaces DLL_REQ (the var holding the "starting"
list of DLLs) with an invocation of emacs batch, as described above.
Once this patch is installed, the script will require the (path to
the) nominally newly created Emacs binary as an argument.

I'm still looking at the differences between outputs from this and
from the script as from my prior patch; however, this runs without
error which will improve the situation if it is applied.  (The patch
is for emacs-29 because I expect Emacs 29.2 will be the next release
packaged.)

>
> You should only skip the first two, which are Windows system DLLs.
>
> > > > Does a "invokes Emacs now and errors out if stuff is missing" approach
> > > > sound right/good?
> > >
> > > I'm not sure I understand how would you force Emacs to "error out"
> > > when we are talking about optional dependencies.  They are optional so
> > > that Emacs could run even if they are not present.
> > >
> >
> > Oops, badly said:  I mean that my build and packaging process should
> > stop and report an error if it cannot create a "complete" DEPS ZIP.
> > Nothing affecting the Emacs run-time.
>
> OK.
>

The patch does not attempt this: I simply remove nil from the "first
present DLL" sweep.

I did verify, currently, only gdiplus and shlwapi return nil, thus the
rest mentioned here are being included (except libgccjig which the
script --still-- expressly omits).

(mapcar (lambda(lib)
(list (car lib)
       (seq-find
(lambda(file)
  (file-exists-p
   (file-name-concat "c:/msys2/mingw64/bin"
     file)))
(cdr lib))))
       dynamic-library-alist)

((gdiplus nil) (shlwapi nil) (xpm "libXpm-nox4.dll")
 (png "libpng16-16.dll") (tiff "libtiff-5.dll") (jpeg "libjpeg-8.dll")
 (gif "libgif-7.dll") (svg "librsvg-2-2.dll") (webp "libwebp-7.dll")
 (webpdemux "libwebpdemux-2.dll") (sqlite3 "libsqlite3-0.dll")
 (gdk-pixbuf "libgdk_pixbuf-2.0-0.dll") (glib "libglib-2.0-0.dll")
 (gio "libgio-2.0-0.dll") (gobject "libgobject-2.0-0.dll")
 (gnutls "libgnutls-30.dll") (libxml2 "libxml2-2.dll")
 (zlib "zlib1.dll") (lcms2 "liblcms2-2.dll") (json "libjansson-4.dll")
 (gccjit "libgccjit-0.dll") (tree-sitter "libtree-sitter.dll"))

[-- Attachment #2: 0001-Replace-hard-coded-DLL-list-with-emacs-batch-for-nt.patch --]
[-- Type: application/octet-stream, Size: 8834 bytes --]

From a44f06900efa33635e1e9e4bfcbf54c9045e0abf Mon Sep 17 00:00:00 2001
From: Corwin Brust <corwin@bru.st>
Date: Thu, 17 Aug 2023 01:28:19 -0500
Subject: [PATCH] Replace DLL list with emacs --batch for nt packaging

The script now requires the path to (the newly built) Emacs as a
command line argument.  Run the script with -h for additional options.
Emacs is invoked to print a list containing the first DLL found to
exist while walking the CDR of each item in `dynamic-library-alist',
and other critical maintenance. (Bug#65206)
* nt/admin/dist-build/build-deps-zips.py(script): remove DLL_REQ
in favor of calling emacs, change vendor slug from msys64 to mingw64,
skip some ancient certificates, add SRC_EXT to map transformation
source package name transformations vs historical convention.
---
 admin/nt/dist-build/build-dep-zips.py | 151 +++++++++++++++++---------
 1 file changed, 98 insertions(+), 53 deletions(-)

diff --git a/admin/nt/dist-build/build-dep-zips.py b/admin/nt/dist-build/build-dep-zips.py
index 71105a071ec..fdd74a60e2b 100755
--- a/admin/nt/dist-build/build-dep-zips.py
+++ b/admin/nt/dist-build/build-dep-zips.py
@@ -20,15 +20,17 @@
 import os
 import shutil
 import re
-import functools
-import operator
+import subprocess
 
 from subprocess import check_output
 
 ## Constants
-EMACS_MAJOR_VERSION="28"
+EMACS_MAJOR_VERSION="29"
 
-# This list derives from the features we want Emacs to compile with.
+# Base URI for the package sources mapped in PKG_REQ
+SRC_REPO="https://repo.msys2.org/mingw/sources"
+
+# Map items in `dynamic-library-alist' to source pakages
 PKG_REQ='''mingw-w64-x86_64-giflib
 mingw-w64-x86_64-gnutls
 mingw-w64-x86_64-harfbuzz
@@ -37,26 +39,34 @@
 mingw-w64-x86_64-libjpeg-turbo
 mingw-w64-x86_64-libpng
 mingw-w64-x86_64-librsvg
+mingw-w64-x86_64-libwebp
 mingw-w64-x86_64-libtiff
 mingw-w64-x86_64-libxml2
-mingw-w64-x86_64-xpm-nox'''.split()
-
-DLL_REQ='''libgif
-libgnutls
-libharfbuzz
-libjansson
-liblcms2
-libturbojpeg
-libpng
-librsvg
-libtiff
-libxml
-libXpm'''.split()
-
+mingw-w64-x86_64-gmp
+mingw-w64-x86_64-xpm-nox
+mingw-w64-x86_64-tree-sitter
+mingw-w64-x86_64-sqlite3'''.split()
+
+# Emacs style path to dependancy DLLs on build system
+DLL_SRC="c:/msys2/mingw64/bin"
+
+# Report first existing file for entries in dynamic-library-alist
+ELISP_PROG="""
+(message "%s" (mapconcat 'identity (remove nil
+	(mapcar (lambda(lib)
+		  (seq-find
+		   (lambda(file)
+		     (file-exists-p
+		      (file-name-concat "{}"
+					file)))
+		   (cdr lib)))
+		dynamic-library-alist)
+	) "\\n"))
+""".format(DLL_SRC)
 
 ## Options
 DRY_RUN=False
-
+NEW_EMACS="bin/emacs.exe"
 
 def check_output_maybe(*args,**kwargs):
     if(DRY_RUN):
@@ -64,15 +74,17 @@ def check_output_maybe(*args,**kwargs):
     else:
         return check_output(*args,**kwargs)
 
+####################
 ## DLL Capture
+
+# entry point
 def gather_deps():
 
     os.mkdir("x86_64")
     os.chdir("x86_64")
 
-    for dep in full_dll_dependency():
-        check_output_maybe(["cp /mingw64/bin/{}*.dll .".format(dep)],
-                           shell=True)
+    for dep in full_dll_dependency(init_deps()):
+        check_output_maybe(["cp /mingw64/bin/{} .".format(dep)], shell=True)
 
     print("Zipping")
     check_output_maybe("zip -9r ../emacs-{}-{}deps.zip *"
@@ -80,15 +92,23 @@ def gather_deps():
                        shell=True)
     os.chdir("../")
 
-## Return all Emacs dependencies
-def full_dll_dependency():
-    deps = [dll_dependency(dep) for dep in DLL_REQ]
-    return set(sum(deps, []) + DLL_REQ)
+# Return dependancies listed in Emacs
+def init_deps():
+    job_args=[NEW_EMACS, "--batch", "--eval", ELISP_PROG]
+    print("args: ", job_args)
+    return subprocess.check_output(job_args, stderr=subprocess.STDOUT
+                                   ).decode('utf-8').splitlines()
+
+# Return all second order dependencies
+def full_dll_dependency(dlls):
+    deps = [dll_dependency(dep) for dep in dlls]
+    return set(sum(deps, []) + dlls)
 
-## Dependencies for a given DLL
+# Dependencies for a given DLL
 def dll_dependency(dll):
     output = check_output(["/mingw64/bin/ntldd", "--recursive",
-                           "/mingw64/bin/{}*.dll".format(dll)]).decode("utf-8")
+                           "/mingw64/bin/{}".format(dll)]
+                          ).decode("utf-8")
     ## munge output
     return ntldd_munge(output)
 
@@ -103,9 +123,9 @@ def ntldd_munge(out):
 
         ## if it's the former, we want it, if its the later we don't
         splt = dep.split()
-        if len(splt) > 2 and "msys64" in splt[2]:
+        if len(splt) > 2 and "mingw64" in splt[2]:
             print("Adding dep", splt[0])
-            rtn.append(splt[0].split(".")[0])
+            rtn.append(splt[0])
 
     return rtn
 
@@ -114,26 +134,43 @@ def ntldd_munge(out):
 ## Packages to fiddle with
 ## Source for gcc-libs is part of gcc
 SKIP_SRC_PKGS=["mingw-w64-gcc-libs"]
-SKIP_DEP_PKGS=frozenset(["mingw-w64-x86_64-glib2"])
+SKIP_DEP_PKGS=["mingw-w64-glib2", "mingw-w64-ca-certificates-20211016-3"]
 MUNGE_SRC_PKGS={"mingw-w64-libwinpthread-git":"mingw-w64-winpthreads-git"}
 MUNGE_DEP_PKGS={
     "mingw-w64-x86_64-libwinpthread":"mingw-w64-x86_64-libwinpthread-git",
     "mingw-w64-x86_64-libtre": "mingw-w64-x86_64-libtre-git",
 }
+SRC_EXT={
+    "mingw-w64-freetype": ".src.tar.zst",
+    "mingw-w64-fribidi": ".src.tar.zst",
+    "mingw-w64-glib2": ".src.tar.zst",
+    "mingw-w64-harfbuzz": ".src.tar.zst",
+    "mingw-w64-libunistring": ".src.tar.zst",
+    "mingw-w64-winpthreads-git": ".src.tar.zst",
+    "mingw-w64-ca-certificates": ".src.tar.zst",
+    "mingw-w64-libxml2": ".src.tar.zst",
+    "mingw-w64-ncurses": ".src.tar.zst",
+    "mingw-w64-openssl": ".src.tar.zst",
+    "mingw-w64-pango": ".src.tar.zst",
+    "mingw-w64-python": ".src.tar.zst",
+    "mingw-w64-sqlite3": ".src.tar.zst",
+    "mingw-w64-xpm-nox": ".src.tar.zst",
+    "mingw-w64-xz": ".src.tar.zst",
+}
 
 ## Currently no packages seem to require this!
 ARCH_PKGS=[]
-SRC_REPO="https://repo.msys2.org/mingw/sources"
 
+def immediate_deps(pkg):
+    package_info = check_output(["pacman", "-Si", pkg]).decode("utf-8").split("\n")
 
-def immediate_deps(pkgs):
-    package_info = check_output(["pacman", "-Si"] + pkgs).decode("utf-8").splitlines()
+    ## Extract the "Depends On" line
+    depends_on = [x for x in package_info if x.startswith("Depends On")][0]
+    ## Remove "Depends On" prefix
+    dependencies = depends_on.split(":")[1]
 
-    ## Extract the packages listed for "Depends On:" lines.
-    dependencies = [line.split(":")[1].split() for line in package_info
-                    if line.startswith("Depends On")]
-    ## Flatten dependency lists from multiple packages into one list.
-    dependencies = functools.reduce(operator.iconcat, dependencies, [])
+    ## Split into dependencies
+    dependencies = dependencies.strip().split(" ")
 
     ## Remove > signs TODO can we get any other punctuation here?
     dependencies = [d.split(">")[0] for d in dependencies if d]
@@ -149,18 +186,16 @@ def extract_deps():
     print( "Extracting deps" )
 
     # Get a list of all dependencies needed for packages mentioned above.
-    pkgs = set(PKG_REQ)
-    newdeps = pkgs
-    print("adding...")
-    while True:
-        subdeps = frozenset(immediate_deps(list(newdeps)))
-        newdeps = subdeps - SKIP_DEP_PKGS - pkgs
-        if not newdeps:
-            break
-        print('\n'.join(newdeps))
-        pkgs |= newdeps
+    pkgs = PKG_REQ[:]
+    n = 0
+    while n < len(pkgs):
+        subdeps = immediate_deps(pkgs[n])
+        for p in subdeps:
+            if not (p in pkgs or p in SKIP_DEP_PKGS):
+                pkgs.append(p)
+        n = n + 1
 
-    return list(pkgs)
+    return sorted(pkgs)
 
 
 def download_source(tarball):
@@ -208,7 +243,13 @@ def gather_source(deps):
         ## Switch names if necessary
         pkg_name = MUNGE_SRC_PKGS.get(pkg_name,pkg_name)
 
-        tarball = "{}-{}.src.tar.gz".format(pkg_name,pkg_version)
+        ## src archive is usually a .tar.gz
+        if pkg_name in SRC_EXT.keys():
+            src_ext = SRC_EXT[pkg_name]
+        else:
+            src_ext = ".src.tar.gz"
+
+        tarball = "{}-{}{}".format(pkg_name,pkg_version,src_ext)
 
         download_source(tarball)
 
@@ -233,6 +274,9 @@ def clean():
 
 
 parser = argparse.ArgumentParser()
+
+parser.add_argument("emacs", help="emacs executable")
+
 parser.add_argument("-s", help="snapshot build",
                     action="store_true")
 
@@ -251,13 +295,14 @@ def clean():
 args = parser.parse_args()
 do_all=not (args.c or args.r)
 
-
+NEW_EMACS=args.emacs
 
 DRY_RUN=args.d
 
 if( args.l ):
     print("List of dependencies")
-    print( deps )
+    init_deps()
+    print( extract_deps() )
     exit(0)
 
 if args.s:
-- 
2.36.1

bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Eli Zaretskii]

> From: Corwin Brust <corwin@bru.st>
> Date: Thu, 17 Aug 2023 02:25:27 -0500
> Cc: 65206@debbugs.gnu.org
> 
> The attached patch replaces DLL_REQ (the var holding the "starting"
> list of DLLs) with an invocation of emacs batch, as described above.
> Once this patch is installed, the script will require the (path to
> the) nominally newly created Emacs binary as an argument.
> 
> I'm still looking at the differences between outputs from this and
> from the script as from my prior patch; however, this runs without
> error which will improve the situation if it is applied.  (The patch
> is for emacs-29 because I expect Emacs 29.2 will be the next release
> packaged.)

Hmm... what is the value of libjpeg-version?  It's 90 here, so the
library for JPEG is libjpeg-9.dll, not libjpeg-8.dll.

Otherwise, LGTM, thanks.

bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Corwin Brust]

On Thu, Aug 17, 2023 at 4:55 AM Eli Zaretskii <eliz@gnu.org> wrote:
>
>
> Hmm... what is the value of libjpeg-version?  It's 90 here, so the
> library for JPEG is libjpeg-9.dll, not libjpeg-8.dll.

I have 80

>
> Otherwise, LGTM, thanks.
>