From mboxrd@z Thu Jan 1 00:00:00 1970 Path: news.gmane.io!.POSTED.ciao.gmane.io!not-for-mail From: Noam Postavsky Newsgroups: gmane.emacs.bugs Subject: bug#40628: Optimize admin/nt dependency computation Date: Tue, 14 Apr 2020 12:41:43 -0400 Message-ID: <854ktmyq7s.fsf@gmail.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Injection-Info: ciao.gmane.io; posting-host="ciao.gmane.io:159.69.161.202"; logging-data="105101"; mail-complaints-to="usenet@ciao.gmane.io" Cc: phillip lord To: 40628@debbugs.gnu.org Original-X-From: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Tue Apr 14 18:49:02 2020 Return-path: Envelope-to: geb-bug-gnu-emacs@m.gmane-mx.org Original-Received: from lists.gnu.org ([209.51.188.17]) by ciao.gmane.io with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jOOk9-000REE-D1 for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 14 Apr 2020 18:49:01 +0200 Original-Received: from localhost ([::1]:34868 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jOOk8-0003mo-Bx for geb-bug-gnu-emacs@m.gmane-mx.org; Tue, 14 Apr 2020 12:49:00 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:57482) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jOOdQ-0000cH-Ez for bug-gnu-emacs@gnu.org; Tue, 14 Apr 2020 12:42:06 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jOOdO-0006In-0q for bug-gnu-emacs@gnu.org; Tue, 14 Apr 2020 12:42:03 -0400 Original-Received: from debbugs.gnu.org ([209.51.188.43]:51299) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jOOdN-0006IH-SW for bug-gnu-emacs@gnu.org; Tue, 14 Apr 2020 12:42:01 -0400 Original-Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1jOOdN-0001Sv-Ps; Tue, 14 Apr 2020 12:42:01 -0400 X-Loop: help-debbugs@gnu.org Resent-From: Noam Postavsky Original-Sender: "Debbugs-submit" Resent-CC: phillip.lord@russet.org.uk, bug-gnu-emacs@gnu.org Resent-Date: Tue, 14 Apr 2020 16:42:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 40628 X-GNU-PR-Package: emacs X-GNU-PR-Keywords: patch X-Debbugs-Original-To: bug-gnu-emacs@gnu.org X-Debbugs-Original-Xcc: phillip lord Original-Received: via spool by submit@debbugs.gnu.org id=B.15868825135616 (code B ref -1); Tue, 14 Apr 2020 16:42:01 +0000 Original-Received: (at submit) by debbugs.gnu.org; 14 Apr 2020 16:41:53 +0000 Original-Received: from localhost ([127.0.0.1]:34611 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jOOdF-0001SW-D0 for submit@debbugs.gnu.org; Tue, 14 Apr 2020 12:41:53 -0400 Original-Received: from lists.gnu.org ([209.51.188.17]:34038) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1jOOdE-0001SP-46 for submit@debbugs.gnu.org; Tue, 14 Apr 2020 12:41:52 -0400 Original-Received: from eggs.gnu.org ([2001:470:142:3::10]:57386) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1jOOdC-0000BR-2I for bug-gnu-emacs@gnu.org; Tue, 14 Apr 2020 12:41:51 -0400 Original-Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1jOOdA-0006Ax-Kw for bug-gnu-emacs@gnu.org; Tue, 14 Apr 2020 12:41:49 -0400 Original-Received: from mail-qv1-xf2f.google.com ([2607:f8b0:4864:20::f2f]:32832) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.71) (envelope-from ) id 1jOOdA-0006AZ-Ef for bug-gnu-emacs@gnu.org; Tue, 14 Apr 2020 12:41:48 -0400 Original-Received: by mail-qv1-xf2f.google.com with SMTP id p19so169847qve.0 for ; Tue, 14 Apr 2020 09:41:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:subject:date:message-id:mime-version; bh=KpWw2VQWkuWbTHN9yy2qkLWICjE/RHWddZ9XZdDMO+E=; b=txixhTyB2NGKvulai/QixH7yn3ph+XMDDE8pAK6FAzye3/bCHgQIbUATPqcPnnno1J SU8OrUzXr7PBdHTaUacj9RagIhmemw8wBY2OHwUbn/Nf6muf8YKqM8roXj6buxaOffLs U2Sr5gQrRd/9k1ecWbyndhbLuWidDVAOhybEK7fnShRGn28bwUbNWXUvYtciEnJnGf4W 1B4UUcaPKxGAFiiYBTgGSzySnKm9/oGohOAbSWff+ymWyOAj2vtQ9NUWET95Z+IPPVL/ Da/j0OarxbtBOcYmkEYKUibCXU79nnT7LkNOumPL006fslWvJoCF/upuRCHQNB3vZjny Eg+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:mime-version; bh=KpWw2VQWkuWbTHN9yy2qkLWICjE/RHWddZ9XZdDMO+E=; b=SxGvGldukW4Hs8zk754HHFAMoCHavAYHIpzNsm/vRyzmffkKTAsRbCUBJjVcXc11A0 qE/IY16E6zJf2qsamy7nQFesXgwf9QDvVHQRVRP/0cPVjQ8966HxII7vEOevlIqfHL5i SJN+tL7vF1/9a7GkGVYY1xHkTfcJlxRr7HAZkUuxHjy6hVZDQicGc9K5IsCbwWMJbSzL xZRHZ/qlyYEravNnlB5ARgEQ2dqy2lHPyH8P3Y94DdIykGSysr1OduqhXi4pc3630urw L13zPnvf0HQV/gaRQAZ7sVJhKEAmASFy1PlzQasZfJ0w9i0m0lMRC1dFGIoqzx/Mmr58 rjzg== X-Gm-Message-State: AGi0PuYD8v4uUAX1k8SR9jIM6gz6rJb7dA1LaHpeyLToeRGhEm3xu9IR q8yEev6vMuI3mGzmR64q34Oic4Id0LE= X-Google-Smtp-Source: APiQypKe+/GHgvprOfyleCmaOMXf9YlffpVW0x4RWoAtK61xFIB3EGwOGtGrZCVNvfx3EC+FD+oAgg== X-Received: by 2002:a0c:fa08:: with SMTP id q8mr855492qvn.132.1586882504646; Tue, 14 Apr 2020 09:41:44 -0700 (PDT) Original-Received: from vhost2 (CPE001143542e1f-CMf81d0f809fa0.cpe.net.cable.rogers.com. [99.230.38.42]) by smtp.gmail.com with ESMTPSA id v62sm1773278qkb.85.2020.04.14.09.41.43 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Tue, 14 Apr 2020 09:41:43 -0700 (PDT) X-detected-operating-system: by eggs.gnu.org: Genre and OS details not recognized. X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.51.188.43 X-BeenThere: bug-gnu-emacs@gnu.org List-Id: "Bug reports for GNU Emacs, the Swiss army knife of text editors" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-gnu-emacs-bounces+geb-bug-gnu-emacs=m.gmane-mx.org@gnu.org Original-Sender: "bug-gnu-emacs" Xref: news.gmane.io gmane.emacs.bugs:178364 Archived-At: --=-=-= Content-Type: text/plain Severity: wishlist Tags: patch X-Debbugs-CC: Phillip Lord While checking the patch I suggested in https://debbugs.gnu.org/40003#11, I noticed the dependency extraction seemed pretty slow. With the patch attached below it's much faster (1m20.437s to 0m3.695s according to 'time ./build-dep-zips.py -l'). --=-=-= Content-Type: text/plain Content-Disposition: attachment; filename=0001-Optimize-admin-nt-dependency-computation.patch Content-Description: patch >From 5312f51e315cc147ca901d499aa3c3cc64aa3340 Mon Sep 17 00:00:00 2001 From: Noam Postavsky Date: Tue, 14 Apr 2020 12:27:34 -0400 Subject: [PATCH] Optimize admin/nt dependency computation admin/nt/dist-build/build-dep-zips.py (immediate_deps) (extract_deps): Gather package dependency info in batches, rather than one at a time. This reduces the number of invocations of 'pacman -Si ...' to the depth of the dependency tree, rather than the number of dependent packages. (top-level): Don't call 'extract_deps' when given the '-l' option. --- admin/nt/dist-build/build-dep-zips.py | 45 ++++++++++++++------------- 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/admin/nt/dist-build/build-dep-zips.py b/admin/nt/dist-build/build-dep-zips.py index b0345a42cf3..dd7dc5e99dc 100755 --- a/admin/nt/dist-build/build-dep-zips.py +++ b/admin/nt/dist-build/build-dep-zips.py @@ -22,6 +22,8 @@ import os import shutil import re +import functools +import operator from subprocess import check_output @@ -47,7 +49,7 @@ ## Packages to fiddle with ## Source for gcc-libs is part of gcc SKIP_SRC_PKGS=["mingw-w64-gcc-libs"] -SKIP_DEP_PKGS=["mingw-w64-x86_64-glib2"] +SKIP_DEP_PKGS=frozenset(["mingw-w64-x86_64-glib2"]) MUNGE_SRC_PKGS={"mingw-w64-libwinpthread-git":"mingw-w64-winpthreads-git"} MUNGE_DEP_PKGS={ "mingw-w64-i686-libwinpthread":"mingw-w64-i686-libwinpthread-git", @@ -68,16 +70,14 @@ def check_output_maybe(*args,**kwargs): else: return check_output(*args,**kwargs) -def immediate_deps(pkg): - package_info = check_output(["pacman", "-Si", pkg]).decode("utf-8").split("\n") +def immediate_deps(pkgs): + package_info = check_output(["pacman", "-Si"] + pkgs).decode("utf-8").splitlines() - ## Extract the "Depends On" line - depends_on = [x for x in package_info if x.startswith("Depends On")][0] - ## Remove "Depends On" prefix - dependencies = depends_on.split(":")[1] - - ## Split into dependencies - dependencies = dependencies.strip().split(" ") + ## Extract the packages listed for "Depends On:" lines. + dependencies = [line.split(":")[1].split() for line in package_info + if line.startswith("Depends On")] + ## Flatten dependency lists from multiple packages into one list. + dependencies = functools.reduce(operator.iconcat, dependencies, []) ## Remove > signs TODO can we get any other punctuation here? dependencies = [d.split(">")[0] for d in dependencies if d] @@ -92,18 +92,19 @@ def extract_deps(): print( "Extracting deps" ) # Get a list of all dependencies needed for packages mentioned above. - pkgs = PKG_REQ[:] + pkgs = set(PKG_REQ) print("Initial pkgs", pkgs) - n = 0 - while n < len(pkgs): - subdeps = immediate_deps(pkgs[n]) - for p in subdeps: - if not (p in pkgs or p in SKIP_DEP_PKGS): - print("adding", p) - pkgs.append(p) - n = n + 1 - - return sorted(pkgs) + newdeps = pkgs + print("adding...") + while True: + subdeps = frozenset(immediate_deps(list(newdeps))) + newdeps = subdeps - SKIP_DEP_PKGS - pkgs + if not newdeps: + break + print('\n'.join(newdeps)) + pkgs |= newdeps + + return list(pkgs) def gather_deps(deps, arch, directory): @@ -261,7 +262,7 @@ def clean(): if( args.l ): print("List of dependencies") - print( extract_deps() ) + print( deps ) exit(0) if args.s: -- 2.23.0.windows.1 --=-=-=--