From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms11 with LMTPS id gIBeLcNaaV9uagAA0tVLHw (envelope-from ) for ; Tue, 22 Sep 2020 02:00:35 +0000 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id kDogKcNaaV8ZIgAAbx9fmQ (envelope-from ) for ; Tue, 22 Sep 2020 02:00:35 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 5D3669404C2 for ; Tue, 22 Sep 2020 02:00:34 +0000 (UTC) Received: from localhost ([::1]:42122 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kKXbc-0005tM-Ob for larch@yhetil.org; Mon, 21 Sep 2020 22:00:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39546) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kKXb9-0005t0-Ef for guix-patches@gnu.org; Mon, 21 Sep 2020 22:00:04 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:45242) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1kKXb8-0001gr-09 for guix-patches@gnu.org; Mon, 21 Sep 2020 22:00:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1kKXb7-0000yn-Uf for guix-patches@gnu.org; Mon, 21 Sep 2020 22:00:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#43516] Resent-From: Maxim Cournoyer Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Tue, 22 Sep 2020 02:00:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 43516 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 43516@debbugs.gnu.org Cc: Leo Famulari Received: via spool by 43516-submit@debbugs.gnu.org id=B43516.16007399723687 (code B ref 43516); Tue, 22 Sep 2020 02:00:01 +0000 Received: (at 43516) by debbugs.gnu.org; 22 Sep 2020 01:59:32 +0000 Received: from localhost ([127.0.0.1]:56785 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kKXae-0000xO-D9 for submit@debbugs.gnu.org; Mon, 21 Sep 2020 21:59:32 -0400 Received: from mail-qk1-f195.google.com ([209.85.222.195]:44642) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1kKXac-0000xC-26 for 43516@debbugs.gnu.org; Mon, 21 Sep 2020 21:59:31 -0400 Received: by mail-qk1-f195.google.com with SMTP id n133so17491894qkn.11 for <43516@debbugs.gnu.org>; Mon, 21 Sep 2020 18:59:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=50ctoBx7OLWNg+r7bPEbPojOoDEq8RmjK0qy1xiW7pM=; b=HLn8rok8kcsbD94dFz1m5oc/WPgErQpJx1qBofIHWoi9hrK7VZwZ586t4QposSO6Q1 CxTstqtxMAHogKJfy3g4gmHdmr8kDU34C4MRMWvzliVZyg3N3ZBJ6qL+/dxTSKDYxCvv DIU813131XORdy3Q5zfaPbnAEA32ZZEG2kigguo+TkLznCo0qZ6kDjpPFXr7fvOgT+EW CPop5uZEAjI0xKw0VPS2E0bRt4PRBLsbcG3jAxdMbVC5eAblDWoxotDde672WD2orR5p QRNIJBRqO/HV3qVdmhJywe+wqHYxR+GOTUu09sr1/1IOUmZgRcpEx6e11DR717YnhYlX so0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=50ctoBx7OLWNg+r7bPEbPojOoDEq8RmjK0qy1xiW7pM=; b=fbTn4cphZPFNZtlf3U0xpM7V+CUZKwJ0GYC7Kgdftxf8sk4myj5JN6c+USE2X6GIIX /rGxq7j0bkDk+Se8/ck4Lmq5Fjbb3l8/lhox4HADN7YIp7wIBpVyn3Rm4jWUYVxaSImE INd2QXzwpiJyYMu7g6jCselt6V0FjijJJsWZZYZRhlb17qVLmIzPslAnNpaS1ea5iAy+ o6NWYwqr0NaON53QkT43hkptoBRcUxO7Yt0yKj88jKehtkQG8nL+VFmmuKC4xYSrEZwa mdE+1rFTEhfm7ku4xflpTuIs64pfPP/1xEnz/kxabEkZsguh/l9GjlAJrAdWkkDnIthz N+mw== X-Gm-Message-State: AOAM533Yfg/8ZJ6NAWWCdDptXSoDhzhJRkNgGOMQHMt/h9W8P7Mt4eHm mQcrbSsIgha8MtosRFsroOfy18oEbyA= X-Google-Smtp-Source: ABdhPJwH2dRbMCX+MQ8DaFzBL7WdOk5duZQHph5wiNCjnoagkRLSWXKtez9MDr3NQLGbJoW9YKwTWQ== X-Received: by 2002:a05:620a:1485:: with SMTP id w5mr2712403qkj.124.1600739964197; Mon, 21 Sep 2020 18:59:24 -0700 (PDT) Received: from localhost.localdomain (dsl-10-131-6.b2b2c.ca. [72.10.131.6]) by smtp.gmail.com with ESMTPSA id x26sm11113945qtr.78.2020.09.21.18.59.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 21 Sep 2020 18:59:23 -0700 (PDT) From: Maxim Cournoyer Date: Mon, 21 Sep 2020 22:00:02 -0400 Message-Id: <20200922020003.6954-1-maxim.cournoyer@gmail.com> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20200919193805.GA31344@jasmine.lan> References: <20200919193805.GA31344@jasmine.lan> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Spam-Score: 2.0 (++) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-Spam-Score: 1.0 (+) X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Scanner: scn0 Authentication-Results: aspmx1.migadu.com; dkim=fail (rsa verify failed) header.d=gmail.com header.s=20161025 header.b=HLn8rok8; dmarc=fail reason="SPF not aligned (relaxed)" header.from=gmail.com (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Spam-Score: 4.59 X-TUID: zjtnpJy+6x81 Hi Leo! > On Sat, Sep 19, 2020 at 01:03:57PM -0400, Maxim Cournoyer wrote: > > The xz compression is slow; using multiple threads/cores yields a linear > > performance improvement. > > > > * guix/packages.scm (patch-and-repack): Ensure xz is invoked with --threads=N > > by setting the XZ_DEFAULTS environment variable. > We tried this previous but reverted it because the archives were not > bit-reproducible: > https://git.savannah.gnu.org/cgit/guix.git/commit/?id=3e95125e9bd0676d4a9add9105217ad3eaef3ff0 Thanks for bringing this to my attention! I've studied what others have done about it, and found a solution that seems to work well on the OpenEmbedded mailing list [0]. Debian uses something similar in their dpkg. The important point is that xz will produce reproducible results as long as it operates in either the single thread mode OR the multi-thread mode (we can't go from one mode to another reproducibly). So the following v2 patch ensures we always use --threads=2 at a minimum, forcing the xz code path into multi-thread operation. The --memlimit=50% argument limits the RAM use of xz to at most half of the available memory, which allows xz to reduce the number of threads used to meet this requirement. I've rebuilt the world or core-updates to test this and got impressive results, such as when building the linux-libre source with 24 cores instead of 1: $ time guix build --source linux-libre --check With this change, on a 24 cores/32 GiB system: 24 cores used, 2.9 GiB max memory used, 36.76 s. On master (same machine): 1 core used, 95 MiB max memory used, 4 m 10 s. [0] https://patchwork.openembedded.org/patch/170475/ [1] https://sources.debian.org/src/dpkg/1.19.7/lib/dpkg/compress.c/#L566-L574 > It's really a shame... it would be nice to reduce the time used for XZ > compression. Seems we can have our cake and eat it, too! Maxim