From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:bcc0::]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id wAqeIy6ChGB7SgAAgWs5BA (envelope-from ) for ; Sat, 24 Apr 2021 22:40:14 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:bcc0::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id kOFVHy6ChGCgBAAA1q6Kng (envelope-from ) for ; Sat, 24 Apr 2021 20:40:14 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id D382986BF for ; Sat, 24 Apr 2021 22:40:12 +0200 (CEST) Received: from localhost ([::1]:39742 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1laP4U-0008Kv-On for larch@yhetil.org; Sat, 24 Apr 2021 16:40:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39384) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1laP4M-0008Km-Ri for bug-guix@gnu.org; Sat, 24 Apr 2021 16:40:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:58267) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1laP4M-0004zj-Jl for bug-guix@gnu.org; Sat, 24 Apr 2021 16:40:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1laP4M-0007Ki-Ha for bug-guix@gnu.org; Sat, 24 Apr 2021 16:40:02 -0400 X-Loop: help-debbugs@gnu.org Subject: bug#48005: low performance when building profile of already built packages Resent-From: Ricardo Wurmus Original-Sender: "Debbugs-submit" Resent-CC: bug-guix@gnu.org Resent-Date: Sat, 24 Apr 2021 20:40:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 48005 X-GNU-PR-Package: guix X-GNU-PR-Keywords: To: 48005@debbugs.gnu.org X-Debbugs-Original-To: bug-guix@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.161929675628125 (code B ref -1); Sat, 24 Apr 2021 20:40:02 +0000 Received: (at submit) by debbugs.gnu.org; 24 Apr 2021 20:39:16 +0000 Received: from localhost ([127.0.0.1]:41580 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1laP3b-0007JZ-Qb for submit@debbugs.gnu.org; Sat, 24 Apr 2021 16:39:16 -0400 Received: from lists.gnu.org ([209.51.188.17]:41314) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1laP3Y-0007JQ-8W for submit@debbugs.gnu.org; Sat, 24 Apr 2021 16:39:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39280) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1laP3X-0008DT-SG for bug-guix@gnu.org; Sat, 24 Apr 2021 16:39:12 -0400 Received: from sender4-of-o51.zoho.com ([136.143.188.51]:21154) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1laP3V-0004Q0-Pq for bug-guix@gnu.org; Sat, 24 Apr 2021 16:39:11 -0400 ARC-Seal: i=1; a=rsa-sha256; t=1619296743; cv=none; d=zohomail.com; s=zohoarc; b=R3t2kuMFLHPUvYA7S8aodivRH/PrhHdXntQhQEBmUC/hBcwujNUFb1/9zpuPk4BMVu8cHS/N51uo46tHS9Ta7VjICb8UltAOZYaWd9/gt7TWfXiqRjW91o/OEij1e4gMeyLpN7lqw81r5eFmVqaz1tdJai7IeOHcYqybJJO9Pi0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=zohomail.com; s=zohoarc; t=1619296743; h=Content-Type:Content-Transfer-Encoding:Date:From:MIME-Version:Message-ID:Subject:To; bh=sI0vQYrIZpHYJmB4h9CVDSgtiWmeKqAl4eoi3lHCYXo=; b=Y8mJ5OiZl+bm5sEOjIU7M+NvFnQeO3nUeGX8QDKAIH+/YFyuMG6NRmGB3Z+HreyT12dsWwI9i3hSFNtj+NqJJMttFNbUPrMTX+S9FQemB+/a+NMwbUUHZUgAoWTnJZ5EnGU++xmdv5HY1RgK7BimOOGnRWc0dh5nwePgQUDY90Q= ARC-Authentication-Results: i=1; mx.zohomail.com; dkim=pass header.i=elephly.net; spf=pass smtp.mailfrom=rekado@elephly.net; dmarc=pass header.from= header.from= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; t=1619296743; s=zoho; d=elephly.net; i=rekado@elephly.net; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type:Content-Transfer-Encoding; bh=sI0vQYrIZpHYJmB4h9CVDSgtiWmeKqAl4eoi3lHCYXo=; b=aMWInhCrxsRskEgeLrVKlPQ8WWgip4GFSurjzjh/6IV9oqYMIrau5vus240dumOe errVOq1f3YHoNh7z0ulbPpB7gHoxACV9SolEvNQAdTSnA12EmJNC0OVpr0iPbIU1mlc 6sV5GXei1PUk9/B6X5gsd2Z4RcEyICpZDjIFNRNw= Received: from localhost (p4fd5ab0c.dip0.t-ipconnect.de [79.213.171.12]) by mx.zohomail.com with SMTPS id 1619296741170139.57686629621355; Sat, 24 Apr 2021 13:39:01 -0700 (PDT) User-agent: mu4e 1.4.15; emacs 27.2 From: Ricardo Wurmus X-URL: https://elephly.net X-PGP-Key: https://elephly.net/rekado.pubkey X-PGP-Fingerprint: BCA6 89B6 3655 3801 C3C6 2150 197A 5888 235F ACAC Date: Sat, 24 Apr 2021 22:38:58 +0200 Message-ID: <87zgxnpgd9.fsf@elephly.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-ZohoMailClient: External Received-SPF: pass client-ip=136.143.188.51; envelope-from=rekado@elephly.net; helo=sender4-of-o51.zoho.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: bug-guix@gnu.org List-Id: Bug reports for GNU Guix List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: bug-guix-bounces+larch=yhetil.org@gnu.org Sender: "bug-Guix" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1619296813; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:list-id:list-help: list-unsubscribe:list-subscribe:list-post:dkim-signature; bh=sI0vQYrIZpHYJmB4h9CVDSgtiWmeKqAl4eoi3lHCYXo=; b=qBHG5M4X21MbJZUbiVRMF4Ciea/poyFDXWL41QMVbCjoy9pkzhr8W60hIW7AKJtqQBsA35 Pre5C3vX3DuMo22Cc7pOBUXgpw4dGnWaYJAhBb7uwxBQdppms3HSv1HuJ4lNNLKMTK4GB1 lKTU82OJ9Xc/n/QLD+wAxQC8+g2POdkeolFU6xfsSlquIo/6KljQNUF7OGoaq3m3urAEar FMTzJA1s9dTItswp1tzKxbvpfyprOTsIINx89bJQZWuJmF2gJ/cTnQlu7F24a/Njx1TJWy oE47av4hd/g+IDBO72//+7TNZKuxq5uSf0ddq83JX9i03UEjIJgRRMvU8fUb4g== ARC-Seal: i=2; s=key1; d=yhetil.org; t=1619296813; a=rsa-sha256; cv=fail; b=aVoncQSVffCBAM9UAO0AAbHXFPUUMRhSq8LLJM+42NiBtWp11NKEa+B/z0tFeDn7TtovzN s/5xshKxQwAOxc00ePkbnJvLItuUQp01Zd6Ebdf38rzNNX7mDNIYhSJGeItAX27XymjBJ8 2ai2jT4CQ7RS+ob2PtwdSqsnesAKD8RPzvS7DDOtQiI2tN/UDP+vfRUCrE7Ajz8CN+42d8 RFIcnbmOGuIZHj8Az+/K3AcTrv10SqSEvKLS3gItVU/4dyNp3q0Un6RA62hhmuIVnF91Ui 6L55+laPMcz6l6wDKLdYR9TEpehaiqBlCZt06sjT7UogGBplhqrkJGPaj+3nCA== ARC-Authentication-Results: i=2; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=elephly.net header.s=zoho header.b=aMWInhCr; arc=reject ("signature check failed: fail, {[1] = sig:zohomail.com:reject}"); spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Spam-Score: 0.56 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=elephly.net header.s=zoho header.b=aMWInhCr; arc=reject ("signature check failed: fail, {[1] = sig:zohomail.com:reject}"); dmarc=none; spf=pass (aspmx1.migadu.com: domain of bug-guix-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=bug-guix-bounces@gnu.org X-Migadu-Queue-Id: D382986BF X-Spam-Score: 0.56 X-Migadu-Scanner: scn0.migadu.com X-TUID: CsGT4EvdRfvJ Here is a manifest that isn=E2=80=99t unusual for a bioinformatics=20 workflow: --8<---------------cut here---------------start------------->8--- (define packages (list "bash-minimal" "fastqc" "gzip" "htseq" "multiqc" "r-corrplot" "r-crosstalk" "r-data-table" "r-deseq2" "r-dt" "r-genomicalignments" "r-genomicranges" "r-ggplot2" "r-ggrepel" "r-gprofiler" "r-knitr" "r-pheatmap" "r-plotly" "r-reshape2" "r-rmarkdown" "r-rsamtools" "r-rtracklayer" "r-s4vectors" "r-scales" "r-summarizedexperiment" "r-tximport" "samtools" "sed" "trim-galore")) (specifications->manifest packages) --8<---------------cut here---------------end--------------->8--- When all these packages already exist and nothing is to be done=20 when Guix is asked to instantiate this manifest, more than 13=20 seconds pass. --8<---------------cut here---------------start------------->8--- $ [env] export GUIX_PROFILING=3D"object-cache=20 add-data-to-store-cache rpc" $ [env] time guix package -m manifest-test.scm -p /tmp/foo The following packages will be installed: bash-minimal 5.0.16 fastqc 0.11.5 gzip 1.10 htseq 0.9.1 multiqc 1.5 r-corrplot 0.84 r-crosstalk 1.1.1 r-data-table 1.14.0 r-deseq2 1.30.1 r-dt 0.17 r-genomicalignments 1.26.0 r-genomicranges 1.42.0 r-ggplot2 3.3.3 r-ggrepel 0.9.1 r-gprofiler 0.7.0 r-knitr 1.31 r-pheatmap 1.0.12 r-plotly 4.9.3 r-reshape2 1.4.4 r-rmarkdown 2.7 r-rsamtools 2.6.0 r-rtracklayer 1.50.0 r-s4vectors 0.28.1 r-scales 1.1.1 r-summarizedexperiment 1.20.0 r-tximport 1.18.0 samtools 1.11 sed 4.8 trim-galore 0.6.6 [[most of the time passes here]] nothing to be done Store object cache: fresh caches: 2 lookups: 45493 hits: 43943 (96.6%) cache size: 1549 entries 'add-data-to-store' cache: lookups: 7755 hits: 3411 (44.0%) .drv files: 4357 (56.2%) Scheme files: 3343 (43.1%) Remote procedure call summary: 5283 RPCs built-in-builders ... 1 add-to-store/tree ... 27 add-to-store ... 213 query-references ... 698 add-text-to-store ... 4344 real 0m13.379s user 0m16.414s sys 0m0.253s --8<---------------cut here---------------end--------------->8--- Some of these R packages have a lot of transitive inputs, such as=20 r-deseq2 with 81 transitive propagated inputs. So there=E2=80=99s a lot=20 of propagation here, which I suspect contributes to things being=20 so unexpectedly slow. This report may be related to performance issues reported for=20 =E2=80=9Cguix environment=E2=80=9D in https://issues.guix.gnu.org/41702, wh= ich=20 also involves R packages (such as r-learnr with 46 transitive=20 propagated inputs). --=20 Ricardo