From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp0 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id e55zHRTDGmFrPAAAgWs5BA (envelope-from ) for ; Mon, 16 Aug 2021 21:57:08 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp0 with LMTPS id WMaMGBTDGmE+IgAA1q6Kng (envelope-from ) for ; Mon, 16 Aug 2021 19:57:08 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id 78C05FEFA for ; Mon, 16 Aug 2021 21:57:07 +0200 (CEST) Received: from localhost ([::1]:58996 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFijK-0001w7-I5 for larch@yhetil.org; Mon, 16 Aug 2021 15:57:06 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57724) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFijG-0001vh-A5 for guix-patches@gnu.org; Mon, 16 Aug 2021 15:57:02 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:39533) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mFijG-0007l7-3y for guix-patches@gnu.org; Mon, 16 Aug 2021 15:57:02 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mFijF-0005WX-Se for guix-patches@gnu.org; Mon, 16 Aug 2021 15:57:01 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#50072] [PATCH WIP 0/4] Add upstream updater for git-fetch origins. Resent-From: Sarah Morgensen Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 16 Aug 2021 19:57:01 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 50072 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Maxime Devos Cc: Xinglu Chen , 50072@debbugs.gnu.org Received: via spool by 50072-submit@debbugs.gnu.org id=B50072.162914381621221 (code B ref 50072); Mon, 16 Aug 2021 19:57:01 +0000 Received: (at 50072) by debbugs.gnu.org; 16 Aug 2021 19:56:56 +0000 Received: from localhost ([127.0.0.1]:51079 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mFij9-0005WC-JI for submit@debbugs.gnu.org; Mon, 16 Aug 2021 15:56:56 -0400 Received: from out1.migadu.com ([91.121.223.63]:17779) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mFij6-0005W2-HF for 50072@debbugs.gnu.org; Mon, 16 Aug 2021 15:56:54 -0400 X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mgsn.dev; s=key1; t=1629143810; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=XcPzAOGC3MLrGTlnAIv+DLhlnggw2fMpY0o03x7a+Hw=; b=DNeb0tYuw5dCQCHQlCGNjXV5JCA2fchfmNg56SltBRorJQCYse6Egz/2dN3wChuDq8BTeb PSFcpfY6kUb8ouuoTKnpCTPEWQmiqNaB24EzGZGOVypPDT513jVJW1dE1R04T6Y98oYWZS FW8MKJmBL3p0vdmC+MhjQawuC88dysY= From: Sarah Morgensen References: <8d1ae518b23fac5b15812a30b11df1c360ab3fbf.1629068119.git.iskarian@mgsn.dev> Date: Mon, 16 Aug 2021 12:56:46 -0700 In-Reply-To: (Maxime Devos's message of "Mon, 16 Aug 2021 12:46:25 +0200") Message-ID: <86fsv9jh8h.fsf_-_@mgsn.dev> MIME-Version: 1.0 Content-Type: text/plain X-Migadu-Auth-User: iskarian@mgsn.dev X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1629143827; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post: dkim-signature; bh=XcPzAOGC3MLrGTlnAIv+DLhlnggw2fMpY0o03x7a+Hw=; b=eew376kT9WzBM/wDcFE1TnfE2DW4y5O1xqhcteRTLIbmx75PKx8Rwp9IncuOxtnJxwrH9K Z/QLc8p77Qre9aJ0fCc+uuPy12KNGs87GoVDHaF5VFnPruqHmVat0nHgesGxlE+NVeLKio uHPocA2BzJnZVLmdRss9Fqg+4aSfH7XFV3srYAS4ZN9t/sZwjwgwLQYTIkIfr7J0Zw39qw Fqg5P07OD2xkHtePUtT7edGqeypg+CqBrYe6Wp0hRWjg3ckzkhw6dLXrXIa3fds2kCWR73 raq04G9kZ5xtSdgh/JubwceUkopkdmYr/H4557Bd+O8LRhE+AuIai6NaNrftwQ== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1629143827; a=rsa-sha256; cv=none; b=gPojd7Q6mSe3hnVtw6B0w34dUBi9qk3XT5fzq8G+dUF4369qlVbqJKyI9ckZxJH2oIAlpO Zc/dtZKEM9EnibwkfW2ydYvCwyWJSCoEvFvPaiwioyCIXmF/1zma0kJUJdicd3UM0JMsb5 HUAp3BCNzDoaqEoTdCx92c3wKlSFbJl2/R3Vuinxmhmsh9fwQhwhzaS/C47da+HWO58V+W hgqs4I0i2qgyHv8iesAOusTyoJah5RVwGxAGhCzMdVCjIQ6vvKrSAyuDUGy3bGAF+fIlje TLspCeDslBJb/vWHlyKbUlPXU3yoE4+CbNfUoXg/aJzG0+sLtte/mjz//eHAtQ== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=mgsn.dev header.s=key1 header.b=DNeb0tYu; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.31 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=mgsn.dev header.s=key1 header.b=DNeb0tYu; dmarc=fail reason="SPF not aligned (relaxed)" header.from=mgsn.dev (policy=none); spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: 78C05FEFA X-Spam-Score: -1.31 X-Migadu-Scanner: scn0.migadu.com X-TUID: 3ibvcfUAlglN Hi Maxime, Thanks for taking a look at this. :) Maxime Devos writes: > Sarah Morgensen schreef op zo 15-08-2021 om 16:25 [-0700]: >> * guix/git-download.scm (checkout-to-store): New procedure. >> * guix/upstream.scm (guess-version-transform) >> (package-update/git-fetch): New procedures. >> (%method-updates): Add GIT-FETCH mapping. > > Does it support packages defined like (a) > > (define-public gnash > (let ((commit "583ccbc1275c7701dc4843ec12142ff86bb305b4") > (revision "0")) > (package > (name "gnash") > (version (git-version "0.8.11" revision commit)) > (source (git-reference > (url "https://example.org") > (commit commit))) > [...]))) No, it doesn't. Since the commit definition isn't part of the actual package definition, the current code has no way of updating it. It would require a rewrite of the edit-in-place logic with probably a lot of special-casing. There are currently ~1250 package which use this format, though, so it could be worth it... Perhaps what we actually need is a better idiom to express this situation. Package properties ('git-commit)? A 'git-version*'? --8<---------------cut here---------------start------------->8--- (define (git-version* version revision) (let* ((source (package-source this-package)) (commit (git-reference-commit (origin-uri source)))) (git-version version revision commit))) --8<---------------cut here---------------end--------------->8--- I'm not sure if binding order would be an issue with that. > and (b) > > (define-public gnash > (package > (name "gnash") > (version "0.8.11") > (source (git-reference > (url "https://example.org") > (commit commit)) > [...])) > ? Is this missing a definition for commit? If it's like above, the same applies. Or if you mean --8<---------------cut here---------------start------------->8--- (source (git-reference (url "https://example.org") (commit "583ccbc1275c7701dc4843ec12142ff86bb305b")) --8<---------------cut here---------------end--------------->8--- Then that wouldn't be too hard to support. There seem to be ~136 packages with this idiom. > (Maybe (a) and (b) can be used as test cases.) > > FWIW, I had a try at supporting git-fetch origins in "--with-latest" and > "guix refresh -e" myself, and had to modify 'package-update' to replace > commit strings. IIRC, it supports (b), but not (a). The patch is > attached, hopefully it will be useful. > > Greetings, > Maxime. > > diff --git a/guix/import/minetest.scm b/guix/import/minetest.scm > index 4264341d6a..2904c3f94a 100644 > --- a/guix/import/minetest.scm > +++ b/guix/import/minetest.scm > @@ -297,7 +297,7 @@ results. The return value is a list of records." > (define (make-minetest-sexp author/name version repository commit > inputs home-page synopsis > description media-license license) > - "Return a S-expression for the minetest package with the given author/NAME, > + "Return a S-expression for the minetest package with the given AUTHOR/NAME, > VERSION, REPOSITORY, COMMIT, INPUTS, HOME-PAGE, SYNOPSIS, DESCRIPTION, > MEDIA-LICENSE and LICENSE." > `(package > @@ -452,3 +452,37 @@ list of AUTHOR/NAME strings." > #:repo->guix-package minetest->guix-package* > #:guix-name > (compose contentdb->package-name author/name->name))) > + > +#| > +(define (minetest-package? pkg) > + (and (string-prefix? "minetest-" (package:package-name pkg)) > + (assq-ref (package:package-properties pkg) 'upstream-name))) > + > +(define (latest-minetest-release pkg) > + "Return an for the latest release of the package PKG." > + (define upstream-name > + (assoc-ref (package:package-properties pkg) 'upstream-name)) > + (define contentdb-package (contentdb-fetch upstream-name)) > + (define release (latest-release upstream-name)) > + (and contentdb-package release > + (and-let* ((old-origin (package:package-source pkg)) > + (old-reference (package:origin-uri old-origin)) > + (is-git? (download:git-reference? old-reference)) > + (commit (release-commit release))) > + (upstream-source > + (package (package:package-name pkg)) > + (version (release-title release)) > + (urls (download:git-reference > + (url (package-repository contentdb-package)) > + (commit commit))))))) Aha! This is actually what should be done, having the updater put the git-reference into upstream-source, since the updater is going to know better how to manipulate the uri. > + > +(define %minetest-updater > + (upstream-updater > + (name 'minetest) > + (description "Updater for Minetest packages on ContentDB") > + (pred minetest-package?) > + (latest latest-minetest-release))) > +|# > +;; #:use-module (guix upstream) > +;; #:use-module ((guix git-download) #:prefix download:) > +;; #:use-module ((guix packages) #:prefix package:) > diff --git a/guix/scripts/refresh.scm b/guix/scripts/refresh.scm > index fb6c52a567..4f3bbbcb94 100644 > --- a/guix/scripts/refresh.scm > +++ b/guix/scripts/refresh.scm > @@ -28,8 +28,10 @@ > #:use-module (guix ui) > #:use-module (gcrypt hash) > #:use-module (guix scripts) > + #:use-module (guix serialization) > #:use-module ((guix scripts build) #:select (%standard-build-options)) > #:use-module (guix store) > + #:use-module (guix build utils) > #:use-module (guix utils) > #:use-module (guix packages) > #:use-module (guix profiles) > @@ -307,6 +309,17 @@ update would trigger a complete rebuild." > (G_ "no updater for ~a~%") > (package-name package))) > > + > +;; XXX adapted from (guix scripts hash) > +(define (file-hash file select? recursive?) > + ;; Compute the hash of FILE. > + (if recursive? > + (let-values (((port get-hash) (open-sha256-port))) > + (write-file file port #:select? select?) > + (force-output port) > + (get-hash)) > + (call-with-input-file file port-sha256))) > + > (define* (update-package store package updaters > #:key (key-download 'interactive) warn?) > "Update the source file that defines PACKAGE with the new version. > @@ -347,8 +360,8 @@ warn about packages that have no matching updater." > (package-name package) > (upstream-input-change-name change))) > (upstream-source-input-changes source)) > - (let ((hash (call-with-input-file tarball > - port-sha256))) > + (let ((hash (file-hash tarball (const #t) > + (directory-exists? tarball)))) > (update-package-source package source hash))) > (warning (G_ "~a: version ~a could not be \ > downloaded and authenticated; not updating~%") > diff --git a/guix/upstream.scm b/guix/upstream.scm > index 632e9ebc4f..61f67b57c1 100644 > --- a/guix/upstream.scm > +++ b/guix/upstream.scm > @@ -24,6 +24,11 @@ > #:use-module (guix discovery) > #:use-module ((guix download) > #:select (download-to-store url-fetch)) > + #:use-module ((guix git-download) > + #:select (git-fetch git-reference? > + git-reference-url > + git-reference-commit > + git-reference-recursive?)) > #:use-module (guix gnupg) > #:use-module (guix packages) > #:use-module (guix diagnostics) > @@ -33,6 +38,7 @@ > #:use-module (guix store) > #:use-module ((guix derivations) #:select (built-derivations derivation->output-path)) > #:autoload (gcrypt hash) (port-sha256) > + #:autoload (guix git) (latest-repository-commit) > #:use-module (guix monads) > #:use-module (srfi srfi-1) > #:use-module (srfi srfi-9) > @@ -93,7 +99,8 @@ > upstream-source? > (package upstream-source-package) ;string > (version upstream-source-version) ;string > - (urls upstream-source-urls) ;list of strings > + ; list of strings or a > + (urls upstream-source-urls) Is it possible for an updater to want to return a list of ? I'm still not sure what the purpose of multiple urls is, since nearly everthing seems to just take (first urls)... > (signature-urls upstream-source-signature-urls ;#f | list of strings > (default #f)) > (input-changes upstream-source-input-changes > @@ -361,6 +368,11 @@ values: 'interactive' (default), 'always', and 'never'." > system target) > "Download SOURCE from its first URL and lower it as a fixed-output > derivation that would fetch it." > + (define url > + (match (upstream-source-urls source) > + ((first . _) first) > + (_ (raise (formatted-message > + (G_ "git origins are unsupported by --with-latest")))))) > (mlet* %store-monad ((url -> (first (upstream-source-urls source))) > (signature > -> (and=> (upstream-source-signature-urls source) > @@ -430,9 +442,23 @@ SOURCE, an ." > #:key-download key-download))) > (values version tarball source)))))) What is this 'upstream-source-compiler' actually used for? I couldn't figure that out, so I just left it untouched. > > +(define* (package-update/git-fetch store package source #:key key-download) > + "Return the version, source code directory, and SOURCE, to update PACKAGE to > +SOURCE, an ." > + (match source > + (($ _ version ref _) > + (values version > + (latest-repository-commit > + store > + (git-reference-url ref) > + #:ref `(commit . ,(git-reference-commit ref)) > + #:recursive? (git-reference-recursive? ref)) > + source)))) > + > (define %method-updates > ;; Mapping of origin methods to source update procedures. > - `((,url-fetch . ,package-update/url-fetch))) > + `((,url-fetch . ,package-update/url-fetch) > + (,git-fetch . ,package-update/git-fetch))) > > (define* (package-update store package > #:optional (updaters (force %updaters)) > @@ -492,9 +518,22 @@ new version string if an update was made, and #f otherwise." > (origin-hash (package-source package)))) > (old-url (match (origin-uri (package-source package)) > ((? string? url) url) > + ((? git-reference? ref) > + (git-reference-url ref)) > (_ #f))) > (new-url (match (upstream-source-urls source) > - ((first _ ...) first))) > + ((first _ ...) first) > + ((? git-reference? ref) > + (git-reference-url ref)) > + (_ #f))) > + (old-commit (match (origin-uri (package-source package)) > + ((? git-reference? ref) > + (git-reference-commit ref)) > + (_ #f))) > + (new-commit (match (upstream-source-urls source) > + ((? git-reference? ref) > + (git-reference-commit ref)) > + (_ #f))) > (file (and=> (location-file loc) > (cut search-path %load-path <>)))) > (if file > @@ -508,6 +547,9 @@ new version string if an update was made, and #f otherwise." > 'filename file)) > (replacements `((,old-version . ,version) > (,old-hash . ,hash) > + ,@(if (and old-commit new-commit) > + `((,old-commit . ,new-commit)) > + '()) > ,@(if (and old-url new-url) > `((,(dirname old-url) . > ,(dirname new-url))) Thanks for sharing your work; it was very helpful! -- Sarah