bug#40628: Optimize admin/nt dependency computation

bug#40628: Optimize admin/nt dependency computation

[Noam Postavsky]

[-- Attachment #1: Type: text/plain, Size: 335 bytes --]

Severity: wishlist
Tags: patch
X-Debbugs-CC: Phillip Lord <phillip.lord@russet.org.uk>

While checking the patch I suggested in
https://debbugs.gnu.org/40003#11, I noticed the dependency extraction
seemed pretty slow.  With the patch attached below it's much faster
(1m20.437s to 0m3.695s according to 'time ./build-dep-zips.py -l').


[-- Attachment #2: patch --]
[-- Type: text/plain, Size: 3548 bytes --]

From 5312f51e315cc147ca901d499aa3c3cc64aa3340 Mon Sep 17 00:00:00 2001
From: Noam Postavsky <npostavs@gmail.com>
Date: Tue, 14 Apr 2020 12:27:34 -0400
Subject: [PATCH] Optimize admin/nt dependency computation

admin/nt/dist-build/build-dep-zips.py (immediate_deps)
(extract_deps): Gather package dependency info in batches, rather than
one at a time.  This reduces the number of invocations of 'pacman -Si
...' to the depth of the dependency tree, rather than the number of
dependent packages.
(top-level): Don't call 'extract_deps' when given the '-l' option.
---
 admin/nt/dist-build/build-dep-zips.py | 45 ++++++++++++++-------------
 1 file changed, 23 insertions(+), 22 deletions(-)

diff --git a/admin/nt/dist-build/build-dep-zips.py b/admin/nt/dist-build/build-dep-zips.py
index b0345a42cf3..dd7dc5e99dc 100755
--- a/admin/nt/dist-build/build-dep-zips.py
+++ b/admin/nt/dist-build/build-dep-zips.py
@@ -22,6 +22,8 @@
 import os
 import shutil
 import re
+import functools
+import operator
 
 from subprocess import check_output
 
@@ -47,7 +49,7 @@
 ## Packages to fiddle with
 ## Source for gcc-libs is part of gcc
 SKIP_SRC_PKGS=["mingw-w64-gcc-libs"]
-SKIP_DEP_PKGS=["mingw-w64-x86_64-glib2"]
+SKIP_DEP_PKGS=frozenset(["mingw-w64-x86_64-glib2"])
 MUNGE_SRC_PKGS={"mingw-w64-libwinpthread-git":"mingw-w64-winpthreads-git"}
 MUNGE_DEP_PKGS={
     "mingw-w64-i686-libwinpthread":"mingw-w64-i686-libwinpthread-git",
@@ -68,16 +70,14 @@ def check_output_maybe(*args,**kwargs):
     else:
         return check_output(*args,**kwargs)
 
-def immediate_deps(pkg):
-    package_info = check_output(["pacman", "-Si", pkg]).decode("utf-8").split("\n")
+def immediate_deps(pkgs):
+    package_info = check_output(["pacman", "-Si"] + pkgs).decode("utf-8").splitlines()
 
-    ## Extract the "Depends On" line
-    depends_on = [x for x in package_info if x.startswith("Depends On")][0]
-    ## Remove "Depends On" prefix
-    dependencies = depends_on.split(":")[1]
-
-    ## Split into dependencies
-    dependencies = dependencies.strip().split(" ")
+    ## Extract the packages listed for "Depends On:" lines.
+    dependencies = [line.split(":")[1].split() for line in package_info
+                    if line.startswith("Depends On")]
+    ## Flatten dependency lists from multiple packages into one list.
+    dependencies = functools.reduce(operator.iconcat, dependencies, [])
 
     ## Remove > signs TODO can we get any other punctuation here?
     dependencies = [d.split(">")[0] for d in dependencies if d]
@@ -92,18 +92,19 @@ def extract_deps():
     print( "Extracting deps" )
 
     # Get a list of all dependencies needed for packages mentioned above.
-    pkgs = PKG_REQ[:]
+    pkgs = set(PKG_REQ)
     print("Initial pkgs", pkgs)
-    n = 0
-    while n < len(pkgs):
-        subdeps = immediate_deps(pkgs[n])
-        for p in subdeps:
-            if not (p in pkgs or p in SKIP_DEP_PKGS):
-                print("adding", p)
-                pkgs.append(p)
-        n = n + 1
-
-    return sorted(pkgs)
+    newdeps = pkgs
+    print("adding...")
+    while True:
+        subdeps = frozenset(immediate_deps(list(newdeps)))
+        newdeps = subdeps - SKIP_DEP_PKGS - pkgs
+        if not newdeps:
+            break
+        print('\n'.join(newdeps))
+        pkgs |= newdeps
+
+    return list(pkgs)
 
 def gather_deps(deps, arch, directory):
 
@@ -261,7 +262,7 @@ def clean():
 
 if( args.l ):
     print("List of dependencies")
-    print( extract_deps() )
+    print( deps )
     exit(0)
 
 if args.s:
-- 
2.23.0.windows.1

bug#40628: Optimize admin/nt dependency computation

[Lars Ingebrigtsen]

Noam Postavsky <npostavs@gmail.com> writes:

> While checking the patch I suggested in
> https://debbugs.gnu.org/40003#11, I noticed the dependency extraction
> seemed pretty slow.  With the patch attached below it's much faster
> (1m20.437s to 0m3.695s according to 'time ./build-dep-zips.py -l').
>
>>From 5312f51e315cc147ca901d499aa3c3cc64aa3340 Mon Sep 17 00:00:00 2001
> From: Noam Postavsky <npostavs@gmail.com>
> Date: Tue, 14 Apr 2020 12:27:34 -0400
> Subject: [PATCH] Optimize admin/nt dependency computation
>
> admin/nt/dist-build/build-dep-zips.py (immediate_deps)
> (extract_deps): Gather package dependency info in batches, rather than
> one at a time.  This reduces the number of invocations of 'pacman -Si
> ...' to the depth of the dependency tree, rather than the number of
> dependent packages.
> (top-level): Don't call 'extract_deps' when given the '-l' option.

This was four months ago, but the patch was apparently not applied?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40628: Optimize admin/nt dependency computation

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Noam Postavsky <npostavs@gmail.com> writes:
>
>> While checking the patch I suggested in
>> https://debbugs.gnu.org/40003#11, I noticed the dependency extraction
>> seemed pretty slow.  With the patch attached below it's much faster
>> (1m20.437s to 0m3.695s according to 'time ./build-dep-zips.py -l').

[...]

> This was four months ago, but the patch was apparently not applied?

I don't have any Windows machines -- has anybody given this patch a test
to see whether it introduces any problems?

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40628: Optimize admin/nt dependency computation

[Lars Ingebrigtsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Noam Postavsky <npostavs@gmail.com> writes:
>>
>>> While checking the patch I suggested in
>>> https://debbugs.gnu.org/40003#11, I noticed the dependency extraction
>>> seemed pretty slow.  With the patch attached below it's much faster
>>> (1m20.437s to 0m3.695s according to 'time ./build-dep-zips.py -l').
>
> [...]
>
>> This was four months ago, but the patch was apparently not applied?
>
> I don't have any Windows machines -- has anybody given this patch a test
> to see whether it introduces any problems?

I tested it myself, and it brought the generation time down from ~3
minutes to about 20 seconds, so I've applied the patch.  I don't see any
difference in the results, but then again, if failed with this both
before and after the change:

subprocess.CalledProcessError: Command 'wget -a ../download.log -O ../emacs-src-cache/mingw-w64-libtiff-4.3.0-6.src.tar.gz https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources/mingw-w64-libtiff-4.3.0-6.src.tar.gz/download' returned non-zero exit status 8.


-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

bug#40628: Optimize admin/nt dependency computation

[Andy Moreton]

On Fri 12 Nov 2021, Lars Ingebrigtsen wrote:

> Lars Ingebrigtsen <larsi@gnus.org> writes:
>
>> Lars Ingebrigtsen <larsi@gnus.org> writes:
>>
>>> Noam Postavsky <npostavs@gmail.com> writes:
>>>
>>>> While checking the patch I suggested in
>>>> https://debbugs.gnu.org/40003#11, I noticed the dependency extraction
>>>> seemed pretty slow.  With the patch attached below it's much faster
>>>> (1m20.437s to 0m3.695s according to 'time ./build-dep-zips.py -l').
>>
>> [...]
>>
>>> This was four months ago, but the patch was apparently not applied?
>>
>> I don't have any Windows machines -- has anybody given this patch a test
>> to see whether it introduces any problems?
>
> I tested it myself, and it brought the generation time down from ~3
> minutes to about 20 seconds, so I've applied the patch.  I don't see any
> difference in the results, but then again, if failed with this both
> before and after the change:
>
> subprocess.CalledProcessError: Command 'wget -a ../download.log -O
> ../emacs-src-cache/mingw-w64-libtiff-4.3.0-6.src.tar.gz
> https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources/mingw-w64-libtiff-4.3.0-6.src.tar.gz/download'
> returned non-zero exit status 8.

The MSYS2 project has moved some stuff away from sourceforge to its own
site. The sources are now at:

  https://repo.msys2.org/mingw/sources/

For the libtiff download above, that results in:

  https://repo.msys2.org/mingw/sources/mingw-w64-libtiff-4.3.0-6.src.tar.gz

The script will need adjusting to change the URL, and drop the
"/download" suffix.

    AndyM 

bug#40628: Optimize admin/nt dependency computation

[Eli Zaretskii]

> From: Lars Ingebrigtsen <larsi@gnus.org>
> Date: Fri, 12 Nov 2021 10:23:17 +0100
> Cc: 40628@debbugs.gnu.org, phillip lord <phillip.lord@russet.org.uk>
> 
> I tested it myself, and it brought the generation time down from ~3
> minutes to about 20 seconds, so I've applied the patch.  I don't see any
> difference in the results, but then again, if failed with this both
> before and after the change:
> 
> subprocess.CalledProcessError: Command 'wget -a ../download.log -O ../emacs-src-cache/mingw-w64-libtiff-4.3.0-6.src.tar.gz https://sourceforge.net/projects/msys2/files/REPOS/MINGW/Sources/mingw-w64-libtiff-4.3.0-6.src.tar.gz/download' returned non-zero exit status 8.

According to my records, 8 means "Server issued an error response".
Whatever that means.

bug#40628: Optimize admin/nt dependency computation

[Lars Ingebrigtsen]

Andy Moreton <andrewjmoreton@gmail.com> writes:

> The MSYS2 project has moved some stuff away from sourceforge to its own
> site. The sources are now at:
>
>   https://repo.msys2.org/mingw/sources/
>
> For the libtiff download above, that results in:
>
>   https://repo.msys2.org/mingw/sources/mingw-w64-libtiff-4.3.0-6.src.tar.gz
>
> The script will need adjusting to change the URL, and drop the
> "/download" suffix.

Thanks; I've now updated the script with the new URL (and dropped the
/download).

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no