From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mp1 ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by ms0.migadu.com with LMTPS id KCU/LahncWGWagAAgWs5BA (envelope-from ) for ; Thu, 21 Oct 2021 15:14:16 +0200 Received: from aspmx1.migadu.com ([2001:41d0:2:4a6f::]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) by mp1 with LMTPS id GM7hKKhncWGLaQAAbx9fmQ (envelope-from ) for ; Thu, 21 Oct 2021 13:14:16 +0000 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by aspmx1.migadu.com (Postfix) with ESMTPS id DAC4E346B8 for ; Thu, 21 Oct 2021 15:14:15 +0200 (CEST) Received: from localhost ([::1]:55830 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mdXtf-0000KS-3U for larch@yhetil.org; Thu, 21 Oct 2021 09:14:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:39640) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mdXqZ-000580-Mt for guix-patches@gnu.org; Thu, 21 Oct 2021 09:11:03 -0400 Received: from debbugs.gnu.org ([209.51.188.43]:44963) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1mdXqZ-0007s2-C9 for guix-patches@gnu.org; Thu, 21 Oct 2021 09:11:03 -0400 Received: from Debian-debbugs by debbugs.gnu.org with local (Exim 4.84_2) (envelope-from ) id 1mdXqY-0008Oa-Vq for guix-patches@gnu.org; Thu, 21 Oct 2021 09:11:02 -0400 X-Loop: help-debbugs@gnu.org Subject: [bug#51319] [PATCH 1/2] build-system/julia: Enable Julia Pkg to find installed packages Resent-From: Jean-Baptiste Volatier Original-Sender: "Debbugs-submit" Resent-CC: guix-patches@gnu.org Resent-Date: Thu, 21 Oct 2021 13:11:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 51319 X-GNU-PR-Package: guix-patches X-GNU-PR-Keywords: patch To: 51319@debbugs.gnu.org Cc: Jean-Baptiste Volatier Received: via spool by 51319-submit@debbugs.gnu.org id=B51319.163482182032193 (code B ref 51319); Thu, 21 Oct 2021 13:11:02 +0000 Received: (at 51319) by debbugs.gnu.org; 21 Oct 2021 13:10:20 +0000 Received: from localhost ([127.0.0.1]:56507 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mdXpr-0008NB-Qv for submit@debbugs.gnu.org; Thu, 21 Oct 2021 09:10:20 -0400 Received: from mail-40133.protonmail.ch ([185.70.40.133]:18355) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1mdXpn-0008MV-3U for 51319@debbugs.gnu.org; Thu, 21 Oct 2021 09:10:16 -0400 Date: Thu, 21 Oct 2021 13:09:58 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pm.me; s=protonmail; t=1634821804; bh=cEdR2dCvWdAvcCyYVXz5VXAYrVBsAQxdqJl8j64rjac=; h=Date:To:From:Cc:Reply-To:Subject:In-Reply-To:References:From; b=pbz02UYIMykj7eEOIQzUbz+DbjbnkIAGXmXD3yNq3DoFxmPFw4OJ0ju4RWWenUxtL eNOz+UdADI+2imGB3jMG6Kwjo3RARkiibRjBck0/924MN/uK90S1WlFOzkp69i/+Ga ztfxbef9t8TsjFKiTfv4ybv6CO0SAiOoCMzlCNvW0o1EAIPT+7D42wTICmnnCbm5LV l8gB117HAyhxZnjLXQFdfMpevO7CLBppgwLM7oXcmDcb2/EdSIxFwz/Dwy7MHVEXT7 BVhqeqvXQUHj34JHle/DQukzO8FChI2YMF7vMp24mOasoLeE8YSP6SnHk1GGkNlRKZ QCfdsDGaDjLgQ== Message-ID: <20211021130952.14696-2-jbv@pm.me> In-Reply-To: <20211021130952.14696-1-jbv@pm.me> References: <20211021130952.14696-1-jbv@pm.me> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list X-BeenThere: guix-patches@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: guix-patches-bounces+larch=yhetil.org@gnu.org Sender: "Guix-patches" Reply-to: Jean-Baptiste Volatier X-ACL-Warn: , Jean-Baptiste Volatier via Guix-patches From: Jean-Baptiste Volatier via Guix-patches via X-Migadu-Flow: FLOW_IN ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=yhetil.org; s=key1; t=1634822056; h=from:from:sender:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:resent-cc: resent-from:resent-sender:resent-message-id:in-reply-to:in-reply-to: references:references:list-id:list-help:list-unsubscribe: list-subscribe:list-post:dkim-signature; bh=cEdR2dCvWdAvcCyYVXz5VXAYrVBsAQxdqJl8j64rjac=; b=CIql0HtVexMQPjNwXkSlWYkcysYnNlAzr0si69tB/QOmrGEPN9keAsY7Zy/JoKO14eHOqF 9D68zWimUPYd6/aFaAHNq+1q3I7It+ZDCkZSmt8WRfYJ8zVm04KVTy222Qzf0VI4CFJLpk KSH5wzctnGymRZbUhRSJLKZv18KOewxZ5V7R8i39U66NzRFLDbc/S/H4WO+Q+ikN0fGcHJ nrG1eXlo7oFzq0BqofZu5XNR+tFK+8jPxABI2g+LRAnaV2RBTOnONmoYOk62+JZ8MjmFkz xmgAkUD2dzCBDZgGCx0OrFoHDaAftMopVtBSZemJwquAvcXOn45RKkG2lGerFw== ARC-Seal: i=1; s=key1; d=yhetil.org; t=1634822056; a=rsa-sha256; cv=none; b=sCQPtnyKTYGqhG7mIlo9m4iALVRiGi6W+e7tV0z0We+VFsai1y6oA+H7G98SGkBBOACy0b gChJiZVlmkGZrEU9jK4446zZA2nWZYNqbDgAamqj3m0BtIonkVNtmGxZkKSy3Mvcn+3tHt 3K71U+wrEKkPBly+sM+WnvcPRmBkIf6nSVkwPuicNXaenNjJbaSLBvY1tpPrYgx08j4cHw ZaN/nwZwSQnCdq/KpO4dYLJJ9NtItqDETu6oGLdxBJtKFRMD5H48zefMau9O5M6BEf9mn2 vhXaLiZETALS/DQABEOf4a/NMH0kszQzVqnh/JMCm3FeZY2egltxgJ5Fzj6Opw== ARC-Authentication-Results: i=1; aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=pm.me header.s=protonmail header.b=pbz02UYI; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Spam-Score: -1.43 Authentication-Results: aspmx1.migadu.com; dkim=fail ("headers rsa verify failed") header.d=pm.me header.s=protonmail header.b=pbz02UYI; dmarc=pass (policy=none) header.from=gnu.org; spf=pass (aspmx1.migadu.com: domain of guix-patches-bounces@gnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom=guix-patches-bounces@gnu.org X-Migadu-Queue-Id: DAC4E346B8 X-Spam-Score: -1.43 X-Migadu-Scanner: scn0.migadu.com X-TUID: 9EUT+SRFDuRn * guix/build/julia-build-system.scm (link-depot): New phase. Julia built-in package manager (Pkg) looks for packages in JULIA_DEPOT_PATH/packages/PACKAGENAME/XXXX where XXXX is a string encoding package UUID and SHA1 of files. The link-depot phase creates a link at the correct location to allow Pkg to find packages that were already installed by Guix. (%package-path): modified package path from packages/ to loadpath/ (project.toml->uuid): retrive package uuid from TOML file * gnu/packages/julia-xyz.scm: correct paths to reflect change of %package-path * gnu/packages/julia.scm: correct paths to reflect change of %package-path --- gnu/packages/julia-xyz.scm | 2 +- gnu/packages/julia.scm | 2 +- guix/build/julia-build-system.scm | 46 ++++++++++++++++++++++++++++--- 3 files changed, 44 insertions(+), 6 deletions(-) diff --git a/gnu/packages/julia-xyz.scm b/gnu/packages/julia-xyz.scm index c1ebc0fba1..1e407a2277 100644 --- a/gnu/packages/julia-xyz.scm +++ b/gnu/packages/julia-xyz.scm @@ -4610,7 +4610,7 @@ (define-public julia-uris (lambda* (#:key source outputs #:allow-other-keys) (let ((out (assoc-ref outputs "out"))) (chdir - (string-append out "/share/julia/packages/URIs/test"))) + (string-append out "/share/julia/loadpath/URIs/test"))) #t))))) ;; required for tests (inputs `(("julia-json" ,julia-json))) diff --git a/gnu/packages/julia.scm b/gnu/packages/julia.scm index 411f2e2e10..5b9bf91ed6 100644 --- a/gnu/packages/julia.scm +++ b/gnu/packages/julia.scm @@ -673,7 +673,7 @@ (define-public julia (native-search-paths (list (search-path-specification (variable "JULIA_LOAD_PATH") - (files (list "share/julia/packages/"))) + (files (list "share/julia/loadpath/"))) (search-path-specification (variable "JULIA_DEPOT_PATH") (files (list "share/julia/"))))) diff --git a/guix/build/julia-build-system.scm b/guix/build/julia-build-sys= tem.scm index d74acf2a05..c7f2119974 100644 --- a/guix/build/julia-build-system.scm +++ b/guix/build/julia-build-system.scm @@ -1,5 +1,6 @@ ;;; GNU Guix --- Functional package management for GNU ;;; Copyright =C2=A9 2019, 2020 Nicol=C3=B2 Balzarotti +;;; Copyright =C2=A9 2021 Jean-Baptiste Volatier ;;; ;;; This file is part of GNU Guix. ;;; @@ -20,10 +21,13 @@ (define-module (guix build julia-build-system) #:use-module ((guix build gnu-build-system) #:prefix gnu:) #:use-module (guix build utils) + #:use-module (rnrs io ports) #:use-module (ice-9 match) #:use-module (ice-9 regex) #:use-module (ice-9 rdelim) + #:use-module (ice-9 popen) #:export (%standard-phases + %package-path julia-create-package-toml julia-build)) @@ -37,7 +41,7 @@ (define (invoke-julia code) (invoke "julia" "-e" code)) ;; subpath where we store the package content -(define %package-path "/share/julia/packages/") +(define %package-path "/share/julia/loadpath/") (define (project.toml->name file) "Look for Julia package name in the TOML file FILE (usually named @@ -51,6 +55,18 @@ (define (project.toml->name file) (if m (match:substring m 1) (loop (read-line in 'concat))))))))) +(define (project.toml->uuid file) + "Look for Julia package uuid in the TOML file FILE (usually named +Project.toml)." + (call-with-input-file file + (lambda (in) + (let loop ((line (read-line in 'concat))) + (if (eof-object? line) + #f + (let ((m (string-match "uuid\\s*=3D\\s*\"(.*)\"" line))) + (if m (match:substring m 1) + (loop (read-line in 'concat))))))))) + (define* (install #:key source inputs outputs julia-package-name #:allow-other-keys) (let* ((out (assoc-ref outputs "out")) @@ -73,7 +89,7 @@ (define* (precompile #:key source inputs outputs julia-pa= ckage-name (setenv "JULIA_DEPOT_PATH" builddir) ;; Add new package dir to the load path. (setenv "JULIA_LOAD_PATH" - (string-append builddir "packages/" ":" + (string-append builddir "loadpath/" ":" (or (getenv "JULIA_LOAD_PATH") ""))) ;; Actual precompilation: @@ -97,15 +113,36 @@ (define* (check #:key tests? source inputs outputs jul= ia-package-name (setenv "SOURCE_DATE_EPOCH" "1") (setenv "JULIA_DEPOT_PATH" builddir) (setenv "JULIA_LOAD_PATH" - (string-append builddir "packages/" ":" + (string-append builddir "loadpath/" ":" (or (getenv "JULIA_LOAD_PATH") ""))) (setenv "HOME" "/tmp") (invoke "julia" "--depwarn=3Dyes" - (string-append builddir "packages/" + (string-append builddir "loadpath/" package "/test/runtests.jl")))) #t) +(define* (link-depot #:key source inputs outputs julia-package-name + #:allow-other-keys) + (let* ((out (assoc-ref outputs "out")) + (package-name (or + julia-package-name + (project.toml->name "Project.toml"))) + (package-dir (string-append out %package-path package-name)) + (uuid (project.toml->uuid "Project.toml")) + (pipe (open-pipe* OPEN_READ "julia" "-e" + (format #f "using Pkg; +println(Base.version_slug(Base.UUID(\"~a\"), + Base.SHA1(Pkg.GitTools.tree_hash(\".\"))))" uuid= package-dir))) + (slug (string-trim-right (get-string-all pipe)))) + ;; When installing a package, julia looks first at in the JULIA_DEPOT_= PATH + ;; for a path like packages/PACKAGE/XXXX + ;; Where XXXX is a slug encoding the package UUID and SHA1 of the file= s + ;; Here we create a link with the correct path to enable julia to find= the package + (mkdir-p (string-append out "/share/julia/packages/" package-name)) + (symlink package-dir (string-append out "/share/julia/packages/" packa= ge-name "/" slug))) + #t) + (define (julia-create-package-toml outputs source name uuid version deps) @@ -138,6 +175,7 @@ (define %standard-phases (delete 'check) ; tests must be run after installation (replace 'install install) (add-after 'install 'precompile precompile) + (add-after 'unpack 'link-depot link-depot) (add-after 'install 'check check) ;; TODO: In the future we could add a "system-image-generation" phase ;; where we use PackageCompiler.jl to speed up package loading times -- 2.33.1