From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id uLzxB8vNH2GwfAAAgWs5BA (envelope-from ) for ; Fri, 20 Aug 2021 17:44:11 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id kG+TA8vNH2EbDwAAbx9fmQ (envelope-from ) for ; Fri, 20 Aug 2021 15:44:11 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 9E3D719BE4 for ; Fri, 20 Aug 2021 17:44:10 +0200 (CEST) Received: from localhost ([::1]:33246 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mH6gj-0006jh-Mh for larch@yhetil.org; Fri, 20 Aug 2021 11:44:09 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:54614) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mH6gc-0006hq-CH for guix-patches@gnu.org; Fri, 20 Aug 2021 11:44:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:52075) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mH6gc-0000I1-3j for guix-patches@gnu.org; Fri, 20 Aug 2021 11:44:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mH6gb-0005hT-SB for guix-patches@gnu.org; Fri, 20 Aug 2021 11:44:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#39258] [PATCH v6 0/2] DRAFT "guix search" performances Resent-From: zimoun Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Fri, 20 Aug 2021 15:44:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 39258 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: To: Ludovic =?UTF-8?Q?Court=C3=A8s?= Cc: Arun Isaac , 39258@debbugs.gnu.org Received: via spool by 39258-submit@debbugs.gnu.org id=B39258.162947418421833 (code B ref 39258); Fri, 20 Aug 2021 15:44:01 +0000 Received: (at 39258) by debbugs.gnu.org; 20 Aug 2021 15:43:04 +0000 Received: from localhost ([127.0.0.1]:35388 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mH6ff-0005g4-Ju for submit@debbugs.gnu.org; Fri, 20 Aug 2021 11:43:04 -0400 Received: from mail-qk1-f176.google.com ([209.85.222.176]:36836) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mH6fd-0005fY-ST for 39258@debbugs.gnu.org; Fri, 20 Aug 2021 11:43:02 -0400 Received: by mail-qk1-f176.google.com with SMTP id e14so11284814qkg.3 for <39258@debbugs.gnu.org>; Fri, 20 Aug 2021 08:43:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=mf25MnZ6G5H5etVNqqabJ6qILKwrSctYOO8R1azgZ4Y=; b=Wtfn3lSG3yR7228TBMRujvAf2MUDWzzQxRbidXZfWZxKp5Bs9Maf7D4vd/3q6ysUMb t/lfh6n8WN6MZmLQfgERZmCHMqlGY1gxcWJQxrwgUxzEkgXoHUehhM2/qX3HU/5ynjhQ n0+b+9DRniJxIUigsxxpgv14K2gF1eWphxuZ5wPj9oF/b3Fb2Wi6m982ALDxgAAhE9CP peJI4gXGAm1gq6wGQj7yUY3QybmR5EEMTnKwQLPmZ6TjqiNJkyo10kEXgGomRENgS3WO 5sxBmnAgRjNFBdwef99DVkJUmdbhi2Ue9h3rOUA1p10S0zMMk4I0K54El/8UpbcNNjvy /Vyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=mf25MnZ6G5H5etVNqqabJ6qILKwrSctYOO8R1azgZ4Y=; b=hYcGbxI1gRp3LiFjaffawXemZnLRx2Sd9XubDgNOeTuQN63o+Baoe9O7VMpk3R6HoU SzaCb0lKhFra9GUbuTZ5dG6ZMXsR1PHJNGz1KzYpSqDXHxDbshoybsw6PHGfwGMOZOJe o222NYY2zOr5uMfN8+HiO+zy6wWWIs1RXQG0BmFZTcumIeTdE7KwAunRHbPF6SYgbP+n KVEnHzH1ClvYWgyFm3wsH0vTUiqeaaSx/LApC6DuMe65MaCVECFXS5vmVq5FNX9I5DOU NthHw/1QDx9rSfXlDnbgSlPy9F4VbxuWPeZqVb72ZKgB7dhyFPnsVuvfIfkz84/uv+uC Zh0A== X-Gm-Message-State: AOAM5330942DtUlJMfNQKmRB+wKYCXKJ8CeP/4aEKjrybsghosdWDzyU nmYeXBzUbkszJGIEHjpTBW9MsH0M0CqrrJW52qs= X-Google-Smtp-Source: ABdhPJzRJuwkwfXdFNLC3hrfi3f3BuWOoVNN/Fe3CUhTS6GTkW/o+QDwabxHoI1ny3+WqhEXoJCOxjh+VFojbFoqu6w= X-Received: by 2002:a05:620a:56e:: with SMTP id p14mr3078103qkp.126.1629474176118; Fri, 20 Aug 2021 08:42:56 -0700 (PDT) MIME-Version: 1.0 References: <20210715073328.212123-1-zimon.toutoune@gmail.com> <875yx1ave7.fsf@gnu.org> In-Reply-To: <875yx1ave7.fsf@gnu.org> From: zimoun Date: Fri, 20 Aug 2021 17:42:44 +0200 Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1629474250; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=mf25MnZ6G5H5etVNqqabJ6qILKwrSctYOO8R1azgZ4Y=; b=omjD8iItDRL3kGroZlBKMkO2x1iQ6LWE5qBeoOF4BdT01A8ZLL2JcxWQjCekJmaTs5oDWC /h9dMrz9OSmqAmjuLhuYrEBjg9zGFP4V+wSidLZkkvDOCSYd62wdz2EUqwxEwzaXYurJOQ TRrSnKuBaJutnkDTWKcPwD/EZeTTEnysmASjxIkakEbZE/v4uo3If1lnND0jwnb3oDkQ8U AwkY3yNwPU6T70HUWLN7gY75AxSyg/Quf0x7qrZeIgVR1k1qR6FQHZyOyWGT65xThb8niH +TY2r9hZMY/46kjBdRgl6y3eJgXr+l7jdBz0cV5Z2qlFz6d46iFqvlIXAO6kuw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1629474250; a=rsa-sha256; cv=none; b=ankJwdWOhaqjpHNWvoMt8nySqKY99vjzOEfJrQhInAgYhFf2ZDeCrB1lTKYgiLCpFEdJ4Y hwbBcnA8vya6T68TL3ErbEGk2Whorbge1iaUj6d7uDcS5l7rIbyO8VBAhaeGrkqLCmMg76 ddx4Y2HspBIaP9hErjcHlKeuNtwpXzgRHOiYt9zZof2DV/BnyUpQAup7sWCW5WBtfcEMjp +O2krZTEx20nHmQiTI1vp7LElKu+qWqTjLc/lQeEz2ui5c/48Je/uw3R/kCB/4i6ck2Qsl 7laoEmgTjM11gU7FuoJ4Qw9JqgcZr4J3E5mMMZWgWraISNW7Tgzc9lpLtzTqHg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=Wtfn3lSG; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.32 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=gmail.com header.s=20161025 header.b=Wtfn3lSG; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 9E3D719BE4 X-Spam-Score: -1.32 X-Migadu-Scanner: scn0.migadu.com X-TUID: gCWt7deYqSyL Hi Ludo, On Fri, 23 Jul 2021 at 17:43, Ludovic Court=C3=A8s wrote: > > From my understanding, the issue that 'package-relevance' accepts a 'pa= ckage' > > (and then all the chain until displaying) and 'fold-avaibale-packages' = does > > not return a package. Well, I do not know; especially where to put som= ething > > similiar to 'read-package-from'. > > Yeah that=E2=80=99s annoying. Perhaps we need or > . With some trickery we could have record type > inheritance or something, maybe. Dunno. > > It would be good if we could arrange so that =E2=80=98fold-available-pack= ages=E2=80=99 > doesn=E2=80=99t allocate anything though. It does not allocate, the allocation is done by 'find-packages-by-descripti= on'. Well, I think v4/v3 [0] is the direction to follow. Therefore, I would revisit this version and try to address two of Ludo's comments [1] and the other ones in v3. BTW, I have not investigated from where the slowness comes: - allocation - garbage collection - '(module-ref (resolve-interface module) symbol)' because I have been a bit disappointed by the performance of this v6. 0: 1: > > Let compare only for cold cache and time this cache building (Guix 7db8= fd6): > > > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > > time guix build --check $(guix gc --derivers $(readlink -f ~/.config/= guix/current/lib/guix/package.cache)) > > > > real 0m28,848s > > user 0m1,481s > > sys 0m0,252s > > > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > > time guix build --check $(guix gc --derivers $(readlink -f /tmp/new/l= ib/guix/package.cache)) > > > > real 0m40,279s > > user 0m1,582s > > sys 0m0,232s > > > > It seems longer but compared to the time of "guix pull" completion, it = seems > > acceptable. > > Both the initial timing and the target are waaay too much. :-/ Yes, but that's another story. :-) We cannot fix all in the same time, IMH= O. Here the point is to speedup "guix search" and to accept to pay a little more at "guix pull" time; then we could optimize the cache generation. Considering the overall time of "guix pull", this extra time appears to me acceptable---if "guix search" is faster. :-) > On my i7 laptop I have: > > --8<---------------cut here---------------start------------->8--- > $ time ./pre-inst-env guile -c '(use-modules (gnu packages)) (generate-p= ackage-cache "/tmp/t.cache")' > > real 0m20.738s > user 0m44.413s > sys 0m0.341s > --8<---------------cut here---------------end--------------->8--- > > It=E2=80=99s CPU-bound; we should probably start by optimizing that. > > In Guile 3.0.7 there was a change that improved this noticeably: > > https://git.savannah.gnu.org/cgit/guile.git/commit/?id=3D05614f792bfabb= c33798863edd0bb67c744e9299 > > We should prolly look for similar optimization opportunities in the > assembler=E2=80=A6 Yes, but this kind of optimization will not provide a faster "guix search" but a faster "guix pull". --8<---------------cut here---------------start------------->8--- $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' $ time guile -c '(use-modules (gnu packages)) (generate-package-cache "/tmp/t1.cache")' real 0m15,728s user 0m23,940s sys 0m0,826s $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' $ time guile -c '(load-compiled "/tmp/t1.cache/lib/guix/package.cache")' real 0m1,026s user 0m0,258s sys 0m0,051s $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' $ time ./pre-inst-env guile -c '(use-modules (gnu packages)) (generate-package-cache "/tmp/t2.cache")' real 0m35,570s user 3m12,951s sys 0m3,807s $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' $ time guile -c '(load-compiled "/tmp/t2.cache/lib/guix/package.cache")' real 0m1,055s user 0m0,283s sys 0m0,055s --8<---------------cut here---------------end--------------->8--- (And if we speak about performance, raw loading the cache takes ~1s but "guix package -A >/dev/null" takes ~8s. It is a big gap for parsing the CLI and sorting. Worse, "guix --version >/dev/null" takes ~2s on cold cache. We should probably start by reduce this overhead.) > > Let compare for some queries: > > > > sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' > > time guix search game | recsel -C -P name | wc -l > > 371 > > > > real 0m7,561s > > user 0m3,525s > > sys 0m0,391s > > I think you should run: > > time guix search game > /dev/null > > otherwise Bash=E2=80=99s built-in =E2=80=98time=E2=80=99 shows the wall-c= lock time of the whole > pipeline, and the processing time of =E2=80=98recsel=E2=80=99 is probably= not negligible > here. First, I am confused... If the formatter (recsel) is not negligible, then it should be dropped and replaced by an internal (fast) formatter. Well, I mean that as an end-user I am interested by the time of the whole pipeline. Second, on my machine the time is somehow negligible*. ;-) On cold cache, 10 runs using the pipe or using the redirection, keeping the max and the min for each: --8<---------------cut here---------------start------------->8--- real 0m9,598s user 0m3,961s sys 0m0,415s real 0m8,744s user 0m3,772s sys 0m0,431s --8<---------------cut here---------------end--------------->8--- --8<---------------cut here---------------start------------->8--- real 0m8,755s user 0m3,869s sys 0m0,540s real 0m8,390s user 0m3,767s sys 0m0,416s --8<---------------cut here---------------end--------------->8--- *negligible: better said, it does not give the bad impression. Even if it is roughly 5% of difference. Cheers, simon