bug#65206: 29.1; [windows][patch] build-deps-zips.py is broken and hard to maintain

[Corwin Brust <corwin@bru.st>]

[-- Attachment #1: Type: text/plain, Size: 2456 bytes --]

On Wed, Aug 16, 2023 at 9:49 AM Eli Zaretskii <eliz@gnu.org> wrote:
> >
> > To confirm: when walking the list, I will want to take the first DLL
> > mentioned that actually exists for each entry. Is that right?
>
> Yes.
>

The attached patch replaces DLL_REQ (the var holding the "starting"
list of DLLs) with an invocation of emacs batch, as described above.
Once this patch is installed, the script will require the (path to
the) nominally newly created Emacs binary as an argument.

I'm still looking at the differences between outputs from this and
from the script as from my prior patch; however, this runs without
error which will improve the situation if it is applied.  (The patch
is for emacs-29 because I expect Emacs 29.2 will be the next release
packaged.)

>
> You should only skip the first two, which are Windows system DLLs.
>
> > > > Does a "invokes Emacs now and errors out if stuff is missing" approach
> > > > sound right/good?
> > >
> > > I'm not sure I understand how would you force Emacs to "error out"
> > > when we are talking about optional dependencies.  They are optional so
> > > that Emacs could run even if they are not present.
> > >
> >
> > Oops, badly said:  I mean that my build and packaging process should
> > stop and report an error if it cannot create a "complete" DEPS ZIP.
> > Nothing affecting the Emacs run-time.
>
> OK.
>

The patch does not attempt this: I simply remove nil from the "first
present DLL" sweep.

I did verify, currently, only gdiplus and shlwapi return nil, thus the
rest mentioned here are being included (except libgccjig which the
script --still-- expressly omits).

(mapcar (lambda(lib)
(list (car lib)
       (seq-find
(lambda(file)
  (file-exists-p
   (file-name-concat "c:/msys2/mingw64/bin"
     file)))
(cdr lib))))
       dynamic-library-alist)

((gdiplus nil) (shlwapi nil) (xpm "libXpm-nox4.dll")
 (png "libpng16-16.dll") (tiff "libtiff-5.dll") (jpeg "libjpeg-8.dll")
 (gif "libgif-7.dll") (svg "librsvg-2-2.dll") (webp "libwebp-7.dll")
 (webpdemux "libwebpdemux-2.dll") (sqlite3 "libsqlite3-0.dll")
 (gdk-pixbuf "libgdk_pixbuf-2.0-0.dll") (glib "libglib-2.0-0.dll")
 (gio "libgio-2.0-0.dll") (gobject "libgobject-2.0-0.dll")
 (gnutls "libgnutls-30.dll") (libxml2 "libxml2-2.dll")
 (zlib "zlib1.dll") (lcms2 "liblcms2-2.dll") (json "libjansson-4.dll")
 (gccjit "libgccjit-0.dll") (tree-sitter "libtree-sitter.dll"))

[-- Attachment #2: 0001-Replace-hard-coded-DLL-list-with-emacs-batch-for-nt.patch --]
[-- Type: application/octet-stream, Size: 8834 bytes --]

From a44f06900efa33635e1e9e4bfcbf54c9045e0abf Mon Sep 17 00:00:00 2001
From: Corwin Brust <corwin@bru.st>
Date: Thu, 17 Aug 2023 01:28:19 -0500
Subject: [PATCH] Replace DLL list with emacs --batch for nt packaging

The script now requires the path to (the newly built) Emacs as a
command line argument.  Run the script with -h for additional options.
Emacs is invoked to print a list containing the first DLL found to
exist while walking the CDR of each item in `dynamic-library-alist',
and other critical maintenance. (Bug#65206)
* nt/admin/dist-build/build-deps-zips.py(script): remove DLL_REQ
in favor of calling emacs, change vendor slug from msys64 to mingw64,
skip some ancient certificates, add SRC_EXT to map transformation
source package name transformations vs historical convention.
---
 admin/nt/dist-build/build-dep-zips.py | 151 +++++++++++++++++---------
 1 file changed, 98 insertions(+), 53 deletions(-)

diff --git a/admin/nt/dist-build/build-dep-zips.py b/admin/nt/dist-build/build-dep-zips.py
index 71105a071ec..fdd74a60e2b 100755
--- a/admin/nt/dist-build/build-dep-zips.py
+++ b/admin/nt/dist-build/build-dep-zips.py
@@ -20,15 +20,17 @@
 import os
 import shutil
 import re
-import functools
-import operator
+import subprocess
 
 from subprocess import check_output
 
 ## Constants
-EMACS_MAJOR_VERSION="28"
+EMACS_MAJOR_VERSION="29"
 
-# This list derives from the features we want Emacs to compile with.
+# Base URI for the package sources mapped in PKG_REQ
+SRC_REPO="https://repo.msys2.org/mingw/sources"
+
+# Map items in `dynamic-library-alist' to source pakages
 PKG_REQ='''mingw-w64-x86_64-giflib
 mingw-w64-x86_64-gnutls
 mingw-w64-x86_64-harfbuzz
@@ -37,26 +39,34 @@
 mingw-w64-x86_64-libjpeg-turbo
 mingw-w64-x86_64-libpng
 mingw-w64-x86_64-librsvg
+mingw-w64-x86_64-libwebp
 mingw-w64-x86_64-libtiff
 mingw-w64-x86_64-libxml2
-mingw-w64-x86_64-xpm-nox'''.split()
-
-DLL_REQ='''libgif
-libgnutls
-libharfbuzz
-libjansson
-liblcms2
-libturbojpeg
-libpng
-librsvg
-libtiff
-libxml
-libXpm'''.split()
-
+mingw-w64-x86_64-gmp
+mingw-w64-x86_64-xpm-nox
+mingw-w64-x86_64-tree-sitter
+mingw-w64-x86_64-sqlite3'''.split()
+
+# Emacs style path to dependancy DLLs on build system
+DLL_SRC="c:/msys2/mingw64/bin"
+
+# Report first existing file for entries in dynamic-library-alist
+ELISP_PROG="""
+(message "%s" (mapconcat 'identity (remove nil
+	(mapcar (lambda(lib)
+		  (seq-find
+		   (lambda(file)
+		     (file-exists-p
+		      (file-name-concat "{}"
+					file)))
+		   (cdr lib)))
+		dynamic-library-alist)
+	) "\\n"))
+""".format(DLL_SRC)
 
 ## Options
 DRY_RUN=False
-
+NEW_EMACS="bin/emacs.exe"
 
 def check_output_maybe(*args,**kwargs):
     if(DRY_RUN):
@@ -64,15 +74,17 @@ def check_output_maybe(*args,**kwargs):
     else:
         return check_output(*args,**kwargs)
 
+####################
 ## DLL Capture
+
+# entry point
 def gather_deps():
 
     os.mkdir("x86_64")
     os.chdir("x86_64")
 
-    for dep in full_dll_dependency():
-        check_output_maybe(["cp /mingw64/bin/{}*.dll .".format(dep)],
-                           shell=True)
+    for dep in full_dll_dependency(init_deps()):
+        check_output_maybe(["cp /mingw64/bin/{} .".format(dep)], shell=True)
 
     print("Zipping")
     check_output_maybe("zip -9r ../emacs-{}-{}deps.zip *"
@@ -80,15 +92,23 @@ def gather_deps():
                        shell=True)
     os.chdir("../")
 
-## Return all Emacs dependencies
-def full_dll_dependency():
-    deps = [dll_dependency(dep) for dep in DLL_REQ]
-    return set(sum(deps, []) + DLL_REQ)
+# Return dependancies listed in Emacs
+def init_deps():
+    job_args=[NEW_EMACS, "--batch", "--eval", ELISP_PROG]
+    print("args: ", job_args)
+    return subprocess.check_output(job_args, stderr=subprocess.STDOUT
+                                   ).decode('utf-8').splitlines()
+
+# Return all second order dependencies
+def full_dll_dependency(dlls):
+    deps = [dll_dependency(dep) for dep in dlls]
+    return set(sum(deps, []) + dlls)
 
-## Dependencies for a given DLL
+# Dependencies for a given DLL
 def dll_dependency(dll):
     output = check_output(["/mingw64/bin/ntldd", "--recursive",
-                           "/mingw64/bin/{}*.dll".format(dll)]).decode("utf-8")
+                           "/mingw64/bin/{}".format(dll)]
+                          ).decode("utf-8")
     ## munge output
     return ntldd_munge(output)
 
@@ -103,9 +123,9 @@ def ntldd_munge(out):
 
         ## if it's the former, we want it, if its the later we don't
         splt = dep.split()
-        if len(splt) > 2 and "msys64" in splt[2]:
+        if len(splt) > 2 and "mingw64" in splt[2]:
             print("Adding dep", splt[0])
-            rtn.append(splt[0].split(".")[0])
+            rtn.append(splt[0])
 
     return rtn
 
@@ -114,26 +134,43 @@ def ntldd_munge(out):
 ## Packages to fiddle with
 ## Source for gcc-libs is part of gcc
 SKIP_SRC_PKGS=["mingw-w64-gcc-libs"]
-SKIP_DEP_PKGS=frozenset(["mingw-w64-x86_64-glib2"])
+SKIP_DEP_PKGS=["mingw-w64-glib2", "mingw-w64-ca-certificates-20211016-3"]
 MUNGE_SRC_PKGS={"mingw-w64-libwinpthread-git":"mingw-w64-winpthreads-git"}
 MUNGE_DEP_PKGS={
     "mingw-w64-x86_64-libwinpthread":"mingw-w64-x86_64-libwinpthread-git",
     "mingw-w64-x86_64-libtre": "mingw-w64-x86_64-libtre-git",
 }
+SRC_EXT={
+    "mingw-w64-freetype": ".src.tar.zst",
+    "mingw-w64-fribidi": ".src.tar.zst",
+    "mingw-w64-glib2": ".src.tar.zst",
+    "mingw-w64-harfbuzz": ".src.tar.zst",
+    "mingw-w64-libunistring": ".src.tar.zst",
+    "mingw-w64-winpthreads-git": ".src.tar.zst",
+    "mingw-w64-ca-certificates": ".src.tar.zst",
+    "mingw-w64-libxml2": ".src.tar.zst",
+    "mingw-w64-ncurses": ".src.tar.zst",
+    "mingw-w64-openssl": ".src.tar.zst",
+    "mingw-w64-pango": ".src.tar.zst",
+    "mingw-w64-python": ".src.tar.zst",
+    "mingw-w64-sqlite3": ".src.tar.zst",
+    "mingw-w64-xpm-nox": ".src.tar.zst",
+    "mingw-w64-xz": ".src.tar.zst",
+}
 
 ## Currently no packages seem to require this!
 ARCH_PKGS=[]
-SRC_REPO="https://repo.msys2.org/mingw/sources"
 
+def immediate_deps(pkg):
+    package_info = check_output(["pacman", "-Si", pkg]).decode("utf-8").split("\n")
 
-def immediate_deps(pkgs):
-    package_info = check_output(["pacman", "-Si"] + pkgs).decode("utf-8").splitlines()
+    ## Extract the "Depends On" line
+    depends_on = [x for x in package_info if x.startswith("Depends On")][0]
+    ## Remove "Depends On" prefix
+    dependencies = depends_on.split(":")[1]
 
-    ## Extract the packages listed for "Depends On:" lines.
-    dependencies = [line.split(":")[1].split() for line in package_info
-                    if line.startswith("Depends On")]
-    ## Flatten dependency lists from multiple packages into one list.
-    dependencies = functools.reduce(operator.iconcat, dependencies, [])
+    ## Split into dependencies
+    dependencies = dependencies.strip().split(" ")
 
     ## Remove > signs TODO can we get any other punctuation here?
     dependencies = [d.split(">")[0] for d in dependencies if d]
@@ -149,18 +186,16 @@ def extract_deps():
     print( "Extracting deps" )
 
     # Get a list of all dependencies needed for packages mentioned above.
-    pkgs = set(PKG_REQ)
-    newdeps = pkgs
-    print("adding...")
-    while True:
-        subdeps = frozenset(immediate_deps(list(newdeps)))
-        newdeps = subdeps - SKIP_DEP_PKGS - pkgs
-        if not newdeps:
-            break
-        print('\n'.join(newdeps))
-        pkgs |= newdeps
+    pkgs = PKG_REQ[:]
+    n = 0
+    while n < len(pkgs):
+        subdeps = immediate_deps(pkgs[n])
+        for p in subdeps:
+            if not (p in pkgs or p in SKIP_DEP_PKGS):
+                pkgs.append(p)
+        n = n + 1
 
-    return list(pkgs)
+    return sorted(pkgs)
 
 
 def download_source(tarball):
@@ -208,7 +243,13 @@ def gather_source(deps):
         ## Switch names if necessary
         pkg_name = MUNGE_SRC_PKGS.get(pkg_name,pkg_name)
 
-        tarball = "{}-{}.src.tar.gz".format(pkg_name,pkg_version)
+        ## src archive is usually a .tar.gz
+        if pkg_name in SRC_EXT.keys():
+            src_ext = SRC_EXT[pkg_name]
+        else:
+            src_ext = ".src.tar.gz"
+
+        tarball = "{}-{}{}".format(pkg_name,pkg_version,src_ext)
 
         download_source(tarball)
 
@@ -233,6 +274,9 @@ def clean():
 
 
 parser = argparse.ArgumentParser()
+
+parser.add_argument("emacs", help="emacs executable")
+
 parser.add_argument("-s", help="snapshot build",
                     action="store_true")
 
@@ -251,13 +295,14 @@ def clean():
 args = parser.parse_args()
 do_all=not (args.c or args.r)
 
-
+NEW_EMACS=args.emacs
 
 DRY_RUN=args.d
 
 if( args.l ):
     print("List of dependencies")
-    print( deps )
+    init_deps()
+    print( extract_deps() )
     exit(0)
 
 if args.s:
-- 
2.36.1