From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id qJu5JYDSiGHcDgAAgWs5BA (envelope-from ) for ; Mon, 08 Nov 2021 08:32:16 +0100 Received: from aspmx1.migadu.com ([2001:41d0:8:6d80::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id 2ARXIYDSiGHyEwAAbx9fmQ (envelope-from ) for ; Mon, 08 Nov 2021 07:32:16 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id E25C6F88B for ; Mon, 8 Nov 2021 08:32:15 +0100 (CET) Received: from localhost ([::1]:35882 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mjz8Y-0007NK-7z for larch@yhetil.org; Mon, 08 Nov 2021 02:32:15 -0500 Received: from eggs.gnu.org ([209.51.188.92]:58578) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mjz8M-0007Ls-S1 for guix-patches@gnu.org; Mon, 08 Nov 2021 02:32:02 -0500 Received: from debbugs.gnu.org ([209.51.188.43]:44408) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mjz8M-0004yi-EN for guix-patches@gnu.org; Mon, 08 Nov 2021 02:32:02 -0500 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mjz8M-0001Gm-BH for guix-patches@gnu.org; Mon, 08 Nov 2021 02:32:02 -0500 X-Loop: help-debbugs@gnu.org Subject: [bug#51319] [PATCH 1/2] build-system/julia: Enable Julia Pkg to find installed packages Resent-From: Efraim Flashner Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Mon, 08 Nov 2021 07:32:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 51319 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: Jean-Baptiste Volatier Cc: 51319@debbugs.gnu.org Received: via spool by 51319-submit@debbugs.gnu.org id=B51319.16363566724776 (code B ref 51319); Mon, 08 Nov 2021 07:32:02 +0000 Received: (at 51319) by debbugs.gnu.org; 8 Nov 2021 07:31:12 +0000 Received: from localhost ([127.0.0.1]:55943 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mjz7X-0001Es-Q6 for submit@debbugs.gnu.org; Mon, 08 Nov 2021 02:31:12 -0500 Received: from flashner.co.il ([178.62.234.194]:33500) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mjz7W-0001Ef-3z for 51319@debbugs.gnu.org; Mon, 08 Nov 2021 02:31:11 -0500 Received: from localhost (unknown [141.226.169.107]) by flashner.co.il (Postfix) with ESMTPSA id 2B66D40349; Mon, 8 Nov 2021 07:31:04 +0000 (UTC) Date: Mon, 8 Nov 2021 09:30:31 +0200 From: Efraim Flashner Message-ID: Mail-Followup-To: Efraim Flashner , Jean-Baptiste Volatier , 51319@debbugs.gnu.org References: <20211021130952.14696-1-jbv@pm.me> <20211021130952.14696-2-jbv@pm.me> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="+rZxTGDfJO+wrlYC" Content-Disposition: inline In-Reply-To: <20211021130952.14696-2-jbv@pm.me> X-PGP-Key-ID: 0x41AAE7DCCA3D8351 X-PGP-Key: https://flashner.co.il/~efraim/efraim_flashner.asc X-PGP-Fingerprint: A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1636356736; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:resent-cc:resent-from:resent-sender: resent-message-id:in-reply-to:in-reply-to:references:references: list-id:list-help:list-unsubscribe:list-subscribe:list-post; bh=70awNea+oylcqh94GzkUb7W22zWrtRcAp7NfZIkLKq4=; b=AMQq5QkB8CKdRi5tJp2J4b7XGtFo4OfzW5XunBnDL+keIvkkzVl5Rom9UQFCxk/ieXmepk 2KxdlMJ4fYDFyic02VkGX+m8gAgZPAck4ioGgVEysltoSBecaqka4qNgQXT5uRJQnWDUxm ltz3K+EURWQpRzQ0IhCVPomP17ylUdG5VuVaIy9GJMtTsg952nH71MJkZSv+WRs75u73F3 I1Ikk2gVvzT/B3FNgUmxSE2SUjPvbYyp1sHnEDMh1Q10B7qIKpttyiG/qTLE2cmvy3SedS gW1wzDV44AptX0JO1UJsuBqyx+UVrxKdH77jjT4qeu3+0+GXSsJ5ij1WCWlOlg== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1636356736; a=rsa-sha256; cv=none; b=mVZUQv395kvnqMEIGpP46ctd5XClWLEOWilyIwBmGuv19e5WRKn5hPQPmRT+sXsYT6vrCC IrrluJEQcf+tFppXPsywbEQHA5eEHTnt05OPVxWqJcB53rOQkrjSLaiEcx6dSDoh9gFq50 i1MMmpKGnr1N2VbDW8fNHoRPuEnSB3mtAJLXwljtVT6Ks434GSD4xaKt4lgVghyGiKKWEj fKjcqi4tgdZgbrcTmWltdymAeos96pPvDIX2nIqCNsn90Gb+rsQTLp03UkCKTpuIONLSOb BJhOusKIYfE6tpNdR0GJ7cIGUOSjU2gos0iRuNj9yiX1D97sDJBqV4fG1fmCag== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -2.52 Authentication-Results: aspmx1.migadu.com; dkim=none; dmarc=none; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: E25C6F88B X-Spam-Score: -2.52 X-Migadu-Scanner: scn0.migadu.com X-TUID: 9hgitMS2LOvz --+rZxTGDfJO+wrlYC Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Thanks for the patch series! I'm really looking forward to this working out in Guix, being able to use Julia to download as-yet unpackaged julia packages will make using Julia much easier. A couple of comments inline in the patch: On Thu, Oct 21, 2021 at 01:09:58PM +0000, Jean-Baptiste Volatier via Guix-p= atches via wrote: > * guix/build/julia-build-system.scm (link-depot): New phase. > Julia built-in package manager (Pkg) looks for packages in > JULIA_DEPOT_PATH/packages/PACKAGENAME/XXXX where XXXX is > a string encoding package UUID and SHA1 of files. > The link-depot phase creates a link at the correct location > to allow Pkg to find packages that were already installed by > Guix. > (%package-path): modified package path from packages/ to > loadpath/ > (project.toml->uuid): retrive package uuid from TOML file >=20 > * gnu/packages/julia-xyz.scm: correct paths to reflect > change of %package-path >=20 > * gnu/packages/julia.scm: correct paths to reflect change of > %package-path > --- > gnu/packages/julia-xyz.scm | 2 +- > gnu/packages/julia.scm | 2 +- > guix/build/julia-build-system.scm | 46 ++++++++++++++++++++++++++++--- > 3 files changed, 44 insertions(+), 6 deletions(-) >=20 > diff --git a/gnu/packages/julia-xyz.scm b/gnu/packages/julia-xyz.scm > index c1ebc0fba1..1e407a2277 100644 > --- a/gnu/packages/julia-xyz.scm > +++ b/gnu/packages/julia-xyz.scm > @@ -4610,7 +4610,7 @@ (define-public julia-uris > (lambda* (#:key source outputs #:allow-other-keys) > (let ((out (assoc-ref outputs "out"))) > (chdir > - (string-append out "/share/julia/packages/URIs/test"))) > + (string-append out "/share/julia/loadpath/URIs/test"))) > #t))))) > ;; required for tests > (inputs `(("julia-json" ,julia-json))) > diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm > index 411f2e2e10..5b9bf91ed6 100644 > --- a/gnu/packages/julia.scm > +++ b/gnu/packages/julia.scm > @@ -673,7 +673,7 @@ (define-public julia > (native-search-paths > (list (search-path-specification > (variable "JULIA_LOAD_PATH") > - (files (list "share/julia/packages/"))) > + (files (list "share/julia/loadpath/"))) > (search-path-specification > (variable "JULIA_DEPOT_PATH") > (files (list "share/julia/"))))) > diff --git a/guix/build/julia-build-system.scm b/guix/build/julia-build-s= ystem.scm > index d74acf2a05..c7f2119974 100644 > --- a/guix/build/julia-build-system.scm > +++ b/guix/build/julia-build-system.scm > @@ -1,5 +1,6 @@ > ;;; GNU Guix --- Functional package management for GNU > ;;; Copyright =C2=A9 2019, 2020 Nicol=C3=B2 Balzarotti > +;;; Copyright =C2=A9 2021 Jean-Baptiste Volatier > ;;; > ;;; This file is part of GNU Guix. > ;;; > @@ -20,10 +21,13 @@ > (define-module (guix build julia-build-system) > #:use-module ((guix build gnu-build-system) #:prefix gnu:) > #:use-module (guix build utils) > + #:use-module (rnrs io ports) > #:use-module (ice-9 match) > #:use-module (ice-9 regex) > #:use-module (ice-9 rdelim) > + #:use-module (ice-9 popen) > #:export (%standard-phases > + %package-path > julia-create-package-toml > julia-build)) >=20 > @@ -37,7 +41,7 @@ (define (invoke-julia code) > (invoke "julia" "-e" code)) >=20 > ;; subpath where we store the package content > -(define %package-path "/share/julia/packages/") > +(define %package-path "/share/julia/loadpath/") >=20 > (define (project.toml->name file) > "Look for Julia package name in the TOML file FILE (usually named > @@ -51,6 +55,18 @@ (define (project.toml->name file) > (if m (match:substring m 1) > (loop (read-line in 'concat))))))))) >=20 > +(define (project.toml->uuid file) > + "Look for Julia package uuid in the TOML file FILE (usually named > +Project.toml)." > + (call-with-input-file file > + (lambda (in) > + (let loop ((line (read-line in 'concat))) > + (if (eof-object? line) > + #f > + (let ((m (string-match "uuid\\s*=3D\\s*\"(.*)\"" line))) > + (if m (match:substring m 1) > + (loop (read-line in 'concat))))))))) > + > (define* (install #:key source inputs outputs julia-package-name > #:allow-other-keys) > (let* ((out (assoc-ref outputs "out")) > @@ -73,7 +89,7 @@ (define* (precompile #:key source inputs outputs julia-= package-name > (setenv "JULIA_DEPOT_PATH" builddir) > ;; Add new package dir to the load path. > (setenv "JULIA_LOAD_PATH" > - (string-append builddir "packages/" ":" > + (string-append builddir "loadpath/" ":" > (or (getenv "JULIA_LOAD_PATH") > ""))) > ;; Actual precompilation: > @@ -97,15 +113,36 @@ (define* (check #:key tests? source inputs outputs j= ulia-package-name > (setenv "SOURCE_DATE_EPOCH" "1") > (setenv "JULIA_DEPOT_PATH" builddir) > (setenv "JULIA_LOAD_PATH" > - (string-append builddir "packages/" ":" > + (string-append builddir "loadpath/" ":" > (or (getenv "JULIA_LOAD_PATH") > ""))) > (setenv "HOME" "/tmp") > (invoke "julia" "--depwarn=3Dyes" > - (string-append builddir "packages/" > + (string-append builddir "loadpath/" > package "/test/runtests.jl")))) > #t) >=20 > +(define* (link-depot #:key source inputs outputs julia-package-name > + #:allow-other-keys) I had to wrap this with '(when (file-exists? "Project.toml")' since we do have some packages that pre-date the use of Project.toml. Building julia-bufferedstreams will test both a package with and without a Project.toml. > + (let* ((out (assoc-ref outputs "out")) > + (package-name (or > + julia-package-name > + (project.toml->name "Project.toml"))) > + (package-dir (string-append out %package-path package-name)) > + (uuid (project.toml->uuid "Project.toml")) > + (pipe (open-pipe* OPEN_READ "julia" "-e" > + (format #f "using Pkg; > +println(Base.version_slug(Base.UUID(\"~a\"), > + Base.SHA1(Pkg.GitTools.tree_hash(\".\"))))" uu= id package-dir))) Here you have a '~a' to use uuid but package-dir isn't used anywhere. Is package-dir not needed or is there a missing substitution? > + (slug (string-trim-right (get-string-all pipe)))) > + ;; When installing a package, julia looks first at in the JULIA_DEPO= T_PATH > + ;; for a path like packages/PACKAGE/XXXX > + ;; Where XXXX is a slug encoding the package UUID and SHA1 of the fi= les > + ;; Here we create a link with the correct path to enable julia to fi= nd the package > + (mkdir-p (string-append out "/share/julia/packages/" package-name)) > + (symlink package-dir (string-append out "/share/julia/packages/" pac= kage-name "/" slug))) > + #t) > + > (define (julia-create-package-toml outputs source > name uuid version > deps) > @@ -138,6 +175,7 @@ (define %standard-phases > (delete 'check) ; tests must be run after installation > (replace 'install install) > (add-after 'install 'precompile precompile) > + (add-after 'unpack 'link-depot link-depot) > (add-after 'install 'check check) > ;; TODO: In the future we could add a "system-image-generation" phase > ;; where we use PackageCompiler.jl to speed up package loading times > -- > 2.33.1 >=20 --=20 Efraim Flashner =D7=A8=D7=A0=D7=A9=D7=9C=D7=A4 = =D7=9D=D7=99=D7=A8=D7=A4=D7=90 GPG key =3D A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 Confidentiality cannot be guaranteed on emails sent or received unencrypted --+rZxTGDfJO+wrlYC Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEoov0DD5VE3JmLRT3Qarn3Mo9g1EFAmGI0hcACgkQQarn3Mo9 g1FvIxAAh4UeF65a0A6KTwS+RNL6qlEfYbOAfd55vzqYQo+wW2hjIGK7+H6+N9SB mlsrw2nwMnjJmwnEcLrISYk5GaL2JT4oIA4ajzVOhcw3bOgbZcaTLry1loLUPawN jbHQZf6V3U6yd/nbNcMj/vkP1nwPLV42vqBuZMxhRh2ri5r1iJ/PB2xcZRO/U6NW m3h36TLqVKoa6F9xp41rYntrmqZc3Sp+MF2tUKg8/5rwQmcoCArLw/egYm2wI/Bh uQmRq9TOY09doTYCeK6LEw6L3ApE3IcIQmDE7jtvLtYkl8DornJrkDcnkt8pfZ7u CJWtYLKiwT+Z915sWhkJLQwpOo2MUQvpZhZB9gU8Ix7isNhlZz0aWc7EHnbYgiyb qRQsUzF1SO5vR0+i1O1DlGHvy2fv3bmwjFgB/iLg/jTDE1BvjIuTlFLl1XHTXoBq Jxu0Nck+VKTXzQ0c559rKzZ1URXHDJWlWhdKMq87qOsxpm75npnOgq5mdm++bqLH CO1/3FySy1U9Cp9rrxR5AWZDUKU6G9c2WMYzyJU7YrtU21C9MKlQ9wilPP5LpGk9 GjPWk/OqNabtHNnXxetJoXW1OYAKdqPyonVf/0Ztp+/lMbZdwbBeFAvMpdXLaT81 fb2BryQIdqpXDiGOxG6vlwepotByDWm1Kh8oGf7c4k/e+8qTFo8= =KRz2 -----END PGP SIGNATURE----- --+rZxTGDfJO+wrlYC--