From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms1.migadu.com with LMTPS id COTPEXz/OGZaiAAA62LTzQ:P1 (envelope-from ) for ; Mon, 06 May 2024 18:04:12 +0200 Received: from aspmx1.migadu.com ([2001:41d0:303:e16b::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1.migadu.com with LMTPS id COTPEXz/OGZaiAAA62LTzQ (envelope-from ) for ; Mon, 06 May 2024 18:04:12 +0200 X-Envelope-To: larch@yhetil.org Authentication-Results: aspmx1.migadu.com; dkim=pass header.d=tscripta.net header.s=fm2 header.b=hOPUaqXn; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=FDtZF2CR; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=tscripta.net ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1715011452; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=LliNRJo3upnBBmZEA19CMGpppmlOTbVTorpdV9v/FTM=; b=lbzYz/QlMfky5inQntcs2GfcE5TJ/OTkrX4F6tjJJg+fB3NuwvLO6Y2ZQYH9v8zM2FZiLL szPXJX/dI9q5M6vr3fYaLQDQIR2rNhvHocaOq7HPGoiRnKFuvEzHT9dFrfLYOaEDw0UM4N aVBzGXSu1Flo0bb29ACshYUYZVYUGhqCJoGGgsrxKQ6kBDUx3v5pW+gdc8t9k8GfPxy/nA tmi4kk1/4AUXUepAb8BMU8znBboXM8yS9XwQ+GP+m9HKrlSd4b70xQ54di/AfFVQjaBYvU 9w+weFuH2WSuvGp9X0z35j6APUZXzg88vp36tN5MGgwFg5WMZbNpaw16W8dqgw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=pass header.d=tscripta.net header.s=fm2 header.b=hOPUaqXn; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=FDtZF2CR; spf=pass (aspmx1.migadu.com: domain of "guix-devel-bounces+larch=yhetil.org@gnu.org" designates 209.51.188.17 as permitted sender) smtp.mailfrom="guix-devel-bounces+larch=yhetil.org@gnu.org"; dmarc=pass (policy=none) header.from=tscripta.net ARC-Seal: i=1; s=key1; d=yhetil.org; t=1715011452; a=rsa-sha256; cv=none; b=KVWsv6Q1kxVx4keqLUJMDv+YBMd4ysSnmIsfon0hWOlesGZJJR2e3r54yI9ttpEMvbLwjQ bExKA68UlKHG5tLTpi0sC9gR9VjN55zL58VOsfc14OSHuXe3s9TNvFNO3mgikyj02n+Wa1 zJgwAmJgKoLkldmEfOJfmXoaotRA0Cz/bFpeVsJhkeqq+lTCnlme8b1v42scvMhxq1ZzXe Cky7K2zv7oef1qev1an5qPtT5DUPm3LwxoPCfaDd92scb6ctAYu7Xvr0g+d1L3YL//wjBK aIY1c04395J/CKxB+tHutz/4exHKmC+cnnvzodqYVcdXO1ow2SZw0yMmQaNteQ== Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id A359F111AB for ; Mon, 6 May 2024 18:04:09 +0200 (CEST) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1s40nI-0005RI-CK; Mon, 06 May 2024 12:02:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s40mb-0005I7-Ju for guix-devel@gnu.org; Mon, 06 May 2024 12:01:54 -0400 Received: from fout6-smtp.messagingengine.com ([103.168.172.149]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1s40mZ-0006L2-76 for guix-devel@gnu.org; Mon, 06 May 2024 12:01:41 -0400 Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailfout.nyi.internal (Postfix) with ESMTP id D49E11380376; Mon, 6 May 2024 12:01:32 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Mon, 06 May 2024 12:01:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tscripta.net; h= cc:cc:content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:subject :subject:to:to; s=fm2; t=1715011292; x=1715097692; bh=LliNRJo3up nBBmZEA19CMGpppmlOTbVTorpdV9v/FTM=; b=hOPUaqXnPfRmbbG7Y9FD/e+NmU DD1pY9ij6+YptjNyEReFp7XiUCTdsU4lJ1C6q2CANcWJYVfQIEZpEyAWh1hG9Umh dWuOTBihjHTtEEYfcU5zHDMkJl9SFa46v6o01xwgcA6YIhId577Hp5m+QqPRJVt/ wOhW+d/uaJA7oR4HQe+lzcDpCaYwBVlFIv6Tv1SPs2JPdtVG2ILYaNLZEsDWqS/k pEJf6pIqVIkAIbHY4a8I9XckOa+V4G2n1OXX++Vza+ZlG1/kOnerpJV/Yl7ncMye +QjEnh0+gP7AOVcu8HCeDSMgFVjTIty/woVe+eC275YkWO3qEc2Q7HKN+5SA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1715011292; x=1715097692; bh=LliNRJo3upnBBmZEA19CMGpppmlO TbVTorpdV9v/FTM=; b=FDtZF2CRSTWVjZlhLBWBe1YKZ8qaHnAtMvC59AfxVWlp cTqIwveoSIP4iMvWawhMvIt49Q8GQfA5z/gJhh0AUrV+xosFtRKqzTCTAGpsQfz3 G5ft9axSx5H8QQcpP+eBgt1B196ZbGo3+w0K7IQeQJY5n99VYBuFasa2j9QzVoCt COM9Ta1Wo+lHfDK7LlneKRfz+f34mACXIV2Rq9jbOsINp6F7+0FhyQwI1aiKm8Sj SL30Inyzl23JdJ6OawbfuwNwbErNGeC0J3hhZXMw1WzLK8uL7i8AZdwrH0WkwxNv scIYGzqIXI3u1CXrR74gNprCjUsz8m/cdIhRCN2Yqw== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddviedgleejucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfhfhvfevufffjgfkgggtsehttdggtddttddtnecuhfhrohhmpeflrghsohhn ucevohhnrhhohicuoehjtghonhhrohihsehtshgtrhhiphhtrgdrnhgvtheqnecuggftrf grthhtvghrnhepleduteeugfehteeuvdeuheefffdtvdegtdehvddvhefhudfgvdekffeg vdduhedunecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomh epjhgtohhnrhhohiesthhstghrihhpthgrrdhnvght X-ME-Proxy: Feedback-ID: i0c91496c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 6 May 2024 12:01:32 -0400 (EDT) References: <21GRCQ0Z0K9NXCO1IEDLT7V9RGQLRCVDMN1TT5B106HNS6E4S@tscripta.net> From: Jason Conroy To: Efraim Flashner Cc: guix-devel@gnu.org Subject: Re: rust-team branch merged Date: Mon, 06 May 2024 10:00:26 -0400 In-reply-to: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; format=flowed Received-SPF: pass client-ip=103.168.172.149; envelope-from=jconroy@tscripta.net; helo=fout6-smtp.messagingengine.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: guix-devel@gnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Development of GNU Guix and the GNU System distribution." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-devel-bounces+larch=yhetil.org@gnu.org Sender: guix-devel-bounces+larch=yhetil.org@gnu.org X-Migadu-Flow: FLOW_IN X-Migadu-Country: US X-Migadu-Spam-Score: -10.00 X-Spam-Score: -10.00 X-Migadu-Queue-Id: A359F111AB X-Migadu-Scanner: mx13.migadu.com X-TUID: Hz/YbzPwuLdV Efraim Flashner writes: > [[PGP Signed Part:Undecided]] > On Wed, Apr 24, 2024 at 11:58:22AM -0400, Jason Conroy wrote: >> >> Efraim Flashner writes: >> > On the other hand, by generating it during the build of each >> > package we >> > make sure to pull in all the crates which exist in the build, >> > so we >> > could add into a profile/manifest just the crates listed in a >> > Cargo.toml >> > and then each crate would pull in its own dependencies, and >> > then the >> > profile hook could combine them all together. >> >> Thanks. Just to make sure I understand: it sounds like you're >> saying that by >> creating the JSON index files up front, we'd be preserving some >> knowledge >> about a package's dependency graph that isn't easily recovered >> later by >> recursively walking through inputs and cargo-inputs for the >> package specs in >> a manifest/profile? > > If we create it upfront it may be easier to keep track of the > cargo-dependencies, but I suppose it would mostly depend on the > implementation. I'm not sure that'd actually be the case, so > don't > worry about it too much. Whatever we end up with will be better > than > what we have now, and we can always make it better later if it > needs to > change. Hi Efraim, I spent some time on this and have a few thoughts below. In short, I made progress but also hit some roadblocks that give me second thoughts about the approach we discussed up-thread. As you suggested, I kept the metadata generation code in cargo-build-system and used the cargo-registry profile hook to merge all of those package artifacts into index files. I also modified the hook to lift the full closure of source dependencies into the profile for any rust packages in the manifest, borrowing logic from `(guix build-system cargo)`. However, testing revealed some issues with this approach: 1) Formerly, for any cargo package with `#:skip-build? #t` the crate was treated as an opaque file: it was copied to a location where cargo could access it, but otherwise unprocessed. But to generate the index file we need to unpack the crate and call `cargo metadata`, which performs some sanity checks on the package structure. The implication is that if #:skip-build? is used for reasons other than performance - e.g., for a package that works fine as a dependency but somehow fails during a standalone build - then we might have trouble including such a package in Cargo's local registry. I don't know how big an issue this is in practice, but I think I encountered one package that fits this scenario: custom_derive-0.1.7. (We can discuss the details in a separate thread if you're interested.) 2) When building against a local-registry, it seems like Cargo expects to find crates that it wouldn't for a vendor tree. As mentioned above, my current patch pulls source crates into the profile using the same logic as cargo-build-system currently does: for each manifest package and its cargo-development-inputs, recursively walk through all of the cargo-inputs. But after building a profile this way, I encountered the following: $ cargo build error: no matching package named `automod` found location searched: registry `crates-io` required by package `serde_json v1.0.111` ... which satisfies dependency `serde_json = "^1.0"` of package `rand_chacha v0.3.1` ... which satisfies dependency `rand_chacha = "^0.3.0"` of package `rand v0.8.5` ... which satisfies dependency `rand = "^0.8.5"` of package `mytest v0.1.0 (/home/guix/mytest)` Note that serde_json is actually a dev-dependency of rand_chacha, and automod is a dev-dependency of serde_json, but our algorithm above assumes that dev-dependencies of dependencies can be omitted. So in practice, it seems like building a local-registry that cargo will accept is even more expensive than building a vendor tree. 3) The build error above is interesting for another reason: cargo simply gives up rather than fetching the missing dependencies. In our earlier discussion, the motivation for using a local-registry over a vendor tree in a profile was to support a sort of hybrid workflow: the profile seeds the registry with some packages, but cargo can still download others on its own during `cargo build`. But in this test and several others, I haven't observed a case where cargo falls back to crates.io for missing packages. Efraim, could you share details on how you accomplished this in the past? I wonder if our discussion might have conflated cargo's registry cache ($CARGO_HOME/registry) with a true local-registry; I haven't found any good documentation on the former, but it has a different directory layout and index file format from the local-registry's, so it wouldn't surprise me if it had different semantics as well. Cheers, Jason