From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp2 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id cBqRB5PnxmDtyAAAgWs5BA (envelope-from ) for ; Mon, 14 Jun 2021 07:22:27 +0200 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp2 with LMTPS id eJERA5PnxmBpagAAB5/wlQ (envelope-from ) for ; Mon, 14 Jun 2021 05:22:27 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 6FAEF274C7 for ; Mon, 14 Jun 2021 07:22:26 +0200 (CEST) Received: from localhost ([::1]:58402 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lsf3J-0006Eu-Dq for larch@yhetil.org; Mon, 14 Jun 2021 01:22:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:45914) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lsf39-0006Em-Tk for guix-devel@gnu.org; Mon, 14 Jun 2021 01:22:15 -0400 Received: from mail-qt1-x836.google.com ([2607:f8b0:4864:20::836]:37534) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lsf37-0000VZ-F4 for guix-devel@gnu.org; Mon, 14 Jun 2021 01:22:15 -0400 Received: by mail-qt1-x836.google.com with SMTP id z4so7677985qts.4 for ; Sun, 13 Jun 2021 22:22:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:references:date:in-reply-to:message-id :user-agent:mime-version; bh=9ZtkEofv4ynUjmu2rspp+sGWxApzA6HzVyz5YsPt3qE=; b=p4lnn9aBpix9YONmm+rf1AYuOvFLRjUyn0Dq33ZwBZDMAdIsxuodZ6LJUdSAllK5rl TnJCcGTDJTT0pPe0kZZf8GZiundIdshNOREiVjeu9qkR2Ap69Fta84m0u0gGyoOU13Tg xejuzqhQGk8Ag2y6uKvWbSY4lwIXTk0EUfZHV1Oz5fAZAZgDMRaEfPbsFusvYepJmJ2H gCMtSLY6gA7yGN5QZ+aCgfoj8Sklq0KSAE6sYaCGRnoNq+QKd8QjrV23Jdlm6PrRsFh6 Swq3BTeh/uy+k+ingwxcin2LF1wpDZZlIDvNvXYRTIeBUjDjuR+pL3dKeGQ301aOUXps jwCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:references:date:in-reply-to :message-id:user-agent:mime-version; bh=9ZtkEofv4ynUjmu2rspp+sGWxApzA6HzVyz5YsPt3qE=; b=AMpVrfOGF8JHLESQ870d33CzkpHL1MXQzpMmoAimrdNYHBxIavY/jNNqWaLtbQcS9u 1aqxIQPdY0mbtWHGRnwxqbBzaEw4j9oVFRCvwkd51/B1pr/DO0v3xU+AqwMfi4byhfcX WQcp8a011YpzHb0ElIbL0Xre6MNlegyftsgDBrN9hrsrK07iBArt7qVRcy1YuaVPOswL oryUFRVULqYNdix0phf6Ay8oVVT+LbrBiB6mZCuZZhkYOwUNmv52STDTH2hZHOyeQseH xG+B3Qkqk/iZ/HjIdDLEGNw8qlfx7yF2BKG9fX4/I+ltYSeHb6SQ7N6ntnbkyxLC9zZU MQpA== X-Gm-Message-State: AOAM532dgeOEOXpgLI8wZBIqSEE4UXE3G2vaOdBWzPfR5RZ84MVIq4/w dB61BdzExqX2LpJrlJchxR45HpmpHg0= X-Google-Smtp-Source: ABdhPJyQkpPbieExH7sA1aB0uu1tLXxQ9nWufaxILLReAlDNgt+EZTZ7W0IVSETrM5rDrUDR/t/MzQ== X-Received: by 2002:ac8:6b5a:: with SMTP id x26mr11806508qts.137.1623648132382; Sun, 13 Jun 2021 22:22:12 -0700 (PDT) Received: from hurd (dsl-141-167.b2b2c.ca. [66.158.141.167]) by smtp.gmail.com with ESMTPSA id i1sm9598935qtg.81.2021.06.13.22.22.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 13 Jun 2021 22:22:11 -0700 (PDT) From: Maxim Cournoyer To: Hartmut Goebel Subject: Re: Some more rust/cargo insights References: <7d180f4a-3e5f-374b-0fcd-6ae5b438a1c8@crazy-compilers.com> Date: Mon, 14 Jun 2021 01:22:10 -0400 In-Reply-To: <7d180f4a-3e5f-374b-0fcd-6ae5b438a1c8@crazy-compilers.com> (Hartmut Goebel's message of "Sun, 6 Jun 2021 16:15:02 +0200") Message-ID: <871r95t3zx.fsf@gmail.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.2 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" Received-SPF: pass client-ip=2607:f8b0:4864:20::836; envelope-from=maxim.cournoyer@gmail.com; helo=mail-qt1-x836.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: guix-devel Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: "Guix-devel" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1623648146; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=9ZtkEofv4ynUjmu2rspp+sGWxApzA6HzVyz5YsPt3qE=; b=fZWRUn/1648aSkkTdoDoIK53fk+Fmn7sut7+Q+36DlIfKcVLvpM7GsPwVWsSyacl8fmW7D Urbj1FoVHb62UQKpXMuDmfxcaBYu2ybJdqEUsheYFi2qsmT907QLLjbse5+PlXPQpbSvA/ hb7jqHrQT4sgJLf9ZbY+MZZES+lwXhqV4sBXp0/XWTj2AQUNnowGFgP4ej2YsnZeI/EGQS aYQojVhAgdCe98ppFpryyj7xGkBORweqSdJhihbIzFL9odbcmFjRhFCuoAcjxftYBh6/ru FzAFpGAITvNRfbOT+LaCrEJDqjsqOTiNaWWMk2Qo//oRkfTnPtHzy1RUtW3FJg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1623648146; a=rsa-sha256; cv=none; b=j49BEgAuIMfLv3TgatDDqB/4rKIk8DnotEHAP6NB+mN7z+UNG1wWU/CXtjvBh79BR8rTnj lJJ5XOZfUyxLvacO+St17jl0sanISVgbu9plcmk1jLAOioCKrs4FYNZ49nsFWaEvMo4aoL UMK110WPw1ywoJYw/WWP72nn6TwE16y/G9TitxSVmPvdnrQXykO9VMtcdVQ005GV7s3T+c F8QnuaDErBflPO0gyyyWnmm2H9zyWZjbSvFjrM6DbzgqaLT+hrTsIOhZck4nCp3ogLTKJL JjjxBI77+/u4I7N0WkGu4VC0/y8MVdomFffCq8NdDZSXTAZr6K+Ruvg++HieKg== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=p4lnn9aB; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Spam-Score: -3.12 Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=p4lnn9aB; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (aspmx1.migadu.com: domain of guix-devel-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-devel-bounces@gnu.org X-Migadu-Queue-Id: 6FAEF274C7 X-Spam-Score: -3.12 X-Migadu-Scanner: scn0.migadu.com X-TUID: wGUfIhobcw5o --=-=-= Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi Hartmut, Hartmut Goebel writes: > Hi., > > these day I had spent some more hours struggling with rust ans cargo, > trying to get "pre-built" crates. > > Summery; Cargo is cruft, no solution found yet. > > I tried reusing a crate from the very same place it was built (see > enclosed script). Anyhow, this does not work since cargo uses a=20 > different "metadata" value, even if noting changed. Based in the > verbose output (cargo build -v =E2=80=A6) I assume that some parameters o= f the=20 > "destination" build get included into this value. > > This meets another observation; when building the sequoia suite, > several crates are build several times - even if all builds are > performed in the same environment. > > Rust's build system is such a cruft - I really would like to throw it > where it belongs: into the trash. I've also tried to naively copy prebuilt rlibs at their expected cached location to see if rust would reuse them, but they didn't seem to; see the patch below if you are curious. There was a 'build-plan' feature in cargo that made it output Makefiles but it's not been under active development and will probably be removed. [0] There's an issue demanding support in Cargo to allow reusing pre-built libraries (last comment is mine) [1]. [0] https://github.com/rust-lang/cargo/issues/7614 [1] https://github.com/rust-lang/cargo/issues/1139 --=-=-= Content-Type: text/x-patch; charset=utf-8 Content-Disposition: attachment; filename=0001-tentatively-reuse-rlib-for-cargo-build-system.patch Content-Transfer-Encoding: quoted-printable >From 0807a912db3faaa1ac2a652c7570ecf63ebed8a5 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Thu, 3 Jun 2021 12:58:15 -0400 Subject: [PATCH] tentatively reuse rlib for cargo-build-system --- guix/build-system/cargo.scm | 3 +- guix/build/cargo-build-system.scm | 78 ++++++++++++++++++++++++++----- 2 files changed, 68 insertions(+), 13 deletions(-) diff --git a/guix/build-system/cargo.scm b/guix/build-system/cargo.scm index e53d2a7523..9ef9f6b149 100644 --- a/guix/build-system/cargo.scm +++ b/guix/build-system/cargo.scm @@ -271,7 +271,8 @@ any dependent crates. This can be a benefits: (build-inputs `(("cargo" ,rust "cargo") ("rustc" ,rust) ,@(expand-crate-sources cargo-inputs cargo-develo= pment-inputs) - ,@native-inputs)) + ,@native-inputs + ,@(if target '() inputs))) (outputs outputs) (build cargo-build) (arguments (strip-keyword-arguments private-keywords arguments)))= )) diff --git a/guix/build/cargo-build-system.scm b/guix/build/cargo-build-sys= tem.scm index 0a95672b00..e68f20e463 100644 --- a/guix/build/cargo-build-system.scm +++ b/guix/build/cargo-build-system.scm @@ -5,6 +5,7 @@ ;;; Copyright =C2=A9 2019, 2020, 2021 Efraim Flashner ;;; Copyright =C2=A9 2020 Jakub K=C4=85dzio=C5=82ka ;;; Copyright =C2=A9 2020 Marius Bakke +;;; Copyright =C2=A9 2021 Maxim Cournoyer ;;; ;;; This file is part of GNU Guix. ;;; @@ -24,7 +25,7 @@ (define-module (guix build cargo-build-system) #:use-module ((guix build gnu-build-system) #:prefix gnu:) #:use-module (guix build json) - #:use-module (guix build utils) + #:use-module ((guix build utils) #:hide (delete)) #:use-module (guix build cargo-utils) #:use-module (ice-9 popen) #:use-module (ice-9 rdelim) @@ -34,7 +35,10 @@ #:use-module (srfi srfi-1) #:use-module (srfi srfi-26) #:export (%standard-phases - cargo-build)) + cargo-build + + rust-version + rust-library-prefix)) =20 ;; Commentary: ;; @@ -42,6 +46,25 @@ ;; ;; Code: =20 +(define (rust-version rust) + "Return the version triplet (major.minor.patch) as a string, given RUST,= a +store file name." + (let* ((version (last (string-split rust #\-))) + (components (string-split version #\.)) + (major+minor+patch (take components 3))) + (string-join major+minor+patch "."))) + +(define (rust-library-prefix/relative inputs) + "Return the relative versioned Rust library prefix where Rust libraries = are +to be installed." + (string-append "lib/rust/" (rust-version (assoc-ref inputs "rustc")))) + +(define (rust-library-prefix inputs outputs) + "Return the absolute versioned Rust library prefix where Rust libraries = are +to be installed." + (string-append (assoc-ref outputs "out") "/" + (rust-library-prefix/relative inputs))) + (define (manifest-targets) "Extract all targets from the Cargo.toml manifest" (let* ((port (open-input-pipe "cargo read-manifest")) @@ -73,6 +96,16 @@ Cargo.toml file present at its root." " | cut -d/ -f2" " | grep -q '^Cargo.toml$'")))= )) =20 +(define (rlib? file) + "Check if FILE has the .rlib extension." + (string-suffix? ".rlib" file)) + +(define (inputs->directories inputs) + "Extract the directory part from INPUTS." + (match inputs + (((names . directories) ...) + directories))) + (define* (unpack-rust-crates #:key inputs vendor-dir #:allow-other-keys) (define (inputs->rust-inputs inputs) "Filter using the label part from INPUTS." @@ -80,11 +113,6 @@ Cargo.toml file present at its root." (match input ((name . _) (rust-package? name)))) inputs)) - (define (inputs->directories inputs) - "Extract the directory part from INPUTS." - (match inputs - (((names . directories) ...) - directories))) =20 (let ((rust-inputs (inputs->directories (inputs->rust-inputs inputs)))) (unless (null? rust-inputs) @@ -185,6 +213,22 @@ directory =3D '" port) (generate-all-checksums vendor-dir) #t) =20 +(define* (populate-cargo-cache #:key inputs outputs #:allow-other-keys) + "Populate the 'target/release' directory with any pre-built Rust librari= es, +to avoid rebuilding them from sources when possible." + (let* ((rust-lib-prefix (rust-library-prefix/relative inputs)) + (input-dirs (inputs->directories inputs)) + (rust-lib-dirs (filter (lambda (f) + (file-exists? (string-append + f "/" rust-lib-prefix))) + input-dirs)) + (rlibs (delete-duplicates (append-map (cut find-files <> "\\.rlib= $") + rust-lib-dirs)))) + (pk 'rust-lib-dirs rust-lib-dirs) + (pk 'rlibs rlibs) + (for-each (cut install-file <> "target/release") rlibs) + (invoke "find" "target"))) + (define* (build #:key skip-build? (features '()) @@ -228,7 +272,9 @@ directory =3D '" port) "Install a given Cargo package." (let* ((out (assoc-ref outputs "out")) (registry (string-append out "/share/cargo/registry")) - (sources (string-append out "/share/cargo/src"))) + (sources (string-append out "/share/cargo/src")) + (libdir (rust-library-prefix inputs outputs)) + (release-dir "target/release")) (mkdir-p out) =20 ;; Make cargo reuse all the artifacts we just built instead @@ -237,10 +283,17 @@ directory =3D '" port) =20 ;; Only install crates which include binary targets, ;; otherwise cargo will raise an error. - (or skip-build? - (not (has-executable-target?)) - (invoke "cargo" "install" "--no-track" "--path" "." "--root" out - "--features" (string-join features))) + (unless skip-build? + ;; Install binaries. + (when (has-executable-target?) + (apply invoke "cargo" "install" "--no-track" "--path" "." "--root"= out + (if (not (null? features)) + (list "--features" (string-join features)) + '()))) + ;; Install static libraries. + (for-each (lambda (file) + (install-file (string-append release-dir "/" file) libdi= r)) + (scandir release-dir (cut string-suffix? ".rlib" <>)))) =20 (when install-source? ;; Install crate tarballs and unpacked sources for later use. @@ -260,6 +313,7 @@ directory =3D '" port) (modify-phases gnu:%standard-phases (delete 'bootstrap) (replace 'configure configure) + (add-before 'build 'populate-cargo-cache populate-cargo-cache) (replace 'build build) (replace 'check check) (replace 'install install) --=20 2.31.1 --=-=-=--