Subdirectories in GUIX_PACKAGE_PATH

Subdirectories in GUIX_PACKAGE_PATH

[Christopher Baines]

[-- Attachment #1.1: Type: text/plain, Size: 1861 bytes --]

Hey,

Recently I had problems with the way GUIX_PACKAGE_PATH was working with
govuk-guix [1]. Currently, I'm using a separate directory for the
GUIX_PACKAGE_PATH that contains symlinks to a subset of the Guile
modules necessary for the packages in the repository.

I think support (whether intentional or otherwise) for this approach was
removed in [2]. While I could get it working again by just symlinking
all the files individually, it reminded me that there was the
possibility of improving GUIX_PACKAGE_PATH.

The reason I'm separating the package search path from the complete
set of modules is linked to the current way I've approached making
govuk-guix work reproducibly in terms of the version of Guix itself.

The govuk-guix repository includes a Guix package definition that
points at a specific revision of Guix. The guix environment command is
used to setup this Guix package, at which point the behaviour should
be independent of the Guix on the system that was used at first. The
script to do this is here [3].

Guix itself does something similar for the package definitions, as it
only looks in (gnu packages). I'm unsure what the motivation is for
this, maybe its faster just to check modules that are known to contain
packages.

Anyway, given the above, I think it would be useful to support
specifying subdirectories when using GUIX_PACKAGE_PATH. Any thoughts?

I've attached a rough patch that sets this up, such that you can do
something like:

  export GUIX_PACKAGE_PATH="/tmp/foo^bar/baz:/tmp/cats"

Where ^ acts as the separator, and bar/baz is the subdirectory.

Thanks,

Chris

1: https://github.com/alphagov/govuk-guix
2:
https://git.savannah.gnu.org/cgit/guix.git/commit/?id=d27cc3bfaafe6b5b0831e88afb1c46311d382a0b
3: https://github.com/alphagov/govuk-guix/blob/master/guix-pre-inst-env

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1.2: 0001-gnu-packages-Support-subdirectories-in-GUIX_PACKAGE_.patch --]
[-- Type: text/x-patch, Size: 2595 bytes --]

From eb859e7866e38b8fb19b1aa14ccd85c5375feff0 Mon Sep 17 00:00:00 2001
From: Christopher Baines <mail@cbaines.net>
Date: Fri, 23 Jun 2017 20:53:10 +0100
Subject: [PATCH] gnu: packages: Support subdirectories in GUIX_PACKAGE_PATH

* gnu/packages.scm (%package-module-path): Treat anything after ^ within each
  element of the GUIX_PACKAGE_PATH as a subdirectory.
---
 gnu/packages.scm | 27 ++++++++++++++++++++++-----
 1 file changed, 22 insertions(+), 5 deletions(-)

diff --git a/gnu/packages.scm b/gnu/packages.scm
index 562906178..25be89c87 100644
--- a/gnu/packages.scm
+++ b/gnu/packages.scm
@@ -117,18 +117,35 @@ for system '~a'")
   ;; Search path for package modules.  Each item must be either a directory
   ;; name or a pair whose car is a directory and whose cdr is a sub-directory
   ;; to narrow the search.
-  (let* ((not-colon   (char-set-complement (char-set #\:)))
-         (environment (string-tokenize (or (getenv "GUIX_PACKAGE_PATH") "")
-                                       not-colon)))
+  (let* ((not-colon (char-set-complement (char-set #\:)))
+         (not-caret (char-set-complement (char-set #\^)))
+         (path-elements-from-environment
+          (map
+           (lambda (path-element)
+             (match (string-tokenize path-element not-caret)
+                    ((dir) dir)
+                    ((dir sub) (cons dir sub))
+                    ((dir sub rest ...)
+                     (leave
+                      (G_ "%package-module-path: GUIX_PACKAGE_PATH element \"~A\"
+ does not match form \"directory[^sub-directory]\".")
+                      path-element))))
+           (string-tokenize (or (getenv "GUIX_PACKAGE_PATH") "")
+                            not-colon))))
+
     ;; Automatically add items from $GUIX_PACKAGE_PATH to Guile's search path.
     (for-each (lambda (directory)
                 (set! %load-path (cons directory %load-path))
                 (set! %load-compiled-path
                       (cons directory %load-compiled-path)))
-              environment)
+              (map (match-lambda
+                    (directory directory)
+                    ((directory sub-directory) directory))
+                   path-elements-from-environment))
 
     (make-parameter
-     (append environment `((,%distro-root-directory . "gnu/packages"))))))
+     `(,@path-elements-from-environment
+       (,%distro-root-directory . "gnu/packages")))))
 
 (define %patch-path
   ;; Define it after '%package-module-path' so that '%load-path' contains user
-- 
2.13.1


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 963 bytes --]

Re: Subdirectories in GUIX_PACKAGE_PATH

[Alex Kost]

Christopher Baines (2017-06-23 21:20 +0100) wrote:

> Hey,
>
> Recently I had problems with the way GUIX_PACKAGE_PATH was working with
> govuk-guix [1]. Currently, I'm using a separate directory for the
> GUIX_PACKAGE_PATH that contains symlinks to a subset of the Guile
> modules necessary for the packages in the repository.

Ouch, I use the same workflow (I mean I symlink a directory with my
packages in GUIX_PACKAGE_PATH), and...

> I think support (whether intentional or otherwise) for this approach was
> removed in [2].

... I have the same problem: guix does not find my packages anymore.
Thanks for finding it!  I would spend a lot of time trying to
investigate what's wrong.

> 2: https://git.savannah.gnu.org/cgit/guix.git/commit/?id=d27cc3bfaafe6b5b0831e88afb1c46311d382a0b

-- 
Alex

Re: Subdirectories in GUIX_PACKAGE_PATH

[Ricardo Wurmus]

Christopher Baines <mail@cbaines.net> writes:

> Recently I had problems with the way GUIX_PACKAGE_PATH was working with
> govuk-guix [1]. Currently, I'm using a separate directory for the
> GUIX_PACKAGE_PATH that contains symlinks to a subset of the Guile
> modules necessary for the packages in the repository.
>
> I think support (whether intentional or otherwise) for this approach was
> removed in [2].

Looks like this was removed in an attempt to improve performance over
NFS.  The “scheme-files” procedure now includes a comment:

  ;; XXX: We don't recurse if we find a symlink.

Would it not be better to fix this instead of adding support for special
syntax in GUIX_PACKAGE_PATH?

> I've attached a rough patch that sets this up, such that you can do
> something like:
>
>   export GUIX_PACKAGE_PATH="/tmp/foo^bar/baz:/tmp/cats"
>
> Where ^ acts as the separator, and bar/baz is the subdirectory.

I sympathize with the desire to work around this problem, but for some
reason I really don’t like adding special notation to path variables.
(“texmf.cnf” of TeX Live has a similar feature where e.g. a trailing
“//” indicates recursion.)

--
Ricardo

GPG: BCA6 89B6 3655 3801 C3C6  2150 197A 5888 235F ACAC
https://elephly.net

Re: Subdirectories in GUIX_PACKAGE_PATH

[Christopher Baines]

[-- Attachment #1.1: Type: text/plain, Size: 2260 bytes --]

On 26/06/17 12:22, Ricardo Wurmus wrote:
> 
> Christopher Baines <mail@cbaines.net> writes:
> 
>> Recently I had problems with the way GUIX_PACKAGE_PATH was working with
>> govuk-guix [1]. Currently, I'm using a separate directory for the
>> GUIX_PACKAGE_PATH that contains symlinks to a subset of the Guile
>> modules necessary for the packages in the repository.
>>
>> I think support (whether intentional or otherwise) for this approach was
>> removed in [2].
> 
> Looks like this was removed in an attempt to improve performance over
> NFS.  The “scheme-files” procedure now includes a comment:
> 
>   ;; XXX: We don't recurse if we find a symlink.
> 
> Would it not be better to fix this instead of adding support for special
> syntax in GUIX_PACKAGE_PATH?

I won't comment on the general case, but in the specific case of trying
to limit the modules which Guix will scan for packages, using symlinks
seems like a unsatisfying approach to me. So, I'm more interested in
better approaches for providing packages to Guix, but specifying
subdirectories to search.

>> I've attached a rough patch that sets this up, such that you can do
>> something like:
>>
>>   export GUIX_PACKAGE_PATH="/tmp/foo^bar/baz:/tmp/cats"
>>
>> Where ^ acts as the separator, and bar/baz is the subdirectory.
> 
> I sympathize with the desire to work around this problem, but for some
> reason I really don’t like adding special notation to path variables.
> (“texmf.cnf” of TeX Live has a similar feature where e.g. a trailing
> “//” indicates recursion.)

I only suggested this, as it was the simplest solution that came to mind
in terms of implementation.

Other ideas include:

A second search path environment variable, e.g.
GUIX_PACKAGE_PATH_SUB_DIRECTORIES . For a GUIX_PACKAGE_PATH like
"/tmp/repo1", this could then be set to "/tmp/repo1/mod1/mod2".


Files in the module directories to either have guix only use or ignore
directories, e.g.

/tmp/
  - repo1/
    - services/
      - .no-packages-here
    - packages/


Exported values in the modules themselves to have Guix ignore the
contents, e.g.

(define-public no-packages-here #t)


Plus probably more that other people can think of?


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 858 bytes --]

Re: Subdirectories in GUIX_PACKAGE_PATH

[Ludovic Courtès]

[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]

Hello,

Ricardo Wurmus <rekado@elephly.net> skribis:

> Christopher Baines <mail@cbaines.net> writes:
>
>> Recently I had problems with the way GUIX_PACKAGE_PATH was working with
>> govuk-guix [1]. Currently, I'm using a separate directory for the
>> GUIX_PACKAGE_PATH that contains symlinks to a subset of the Guile
>> modules necessary for the packages in the repository.
>>
>> I think support (whether intentional or otherwise) for this approach was
>> removed in [2].
>
> Looks like this was removed in an attempt to improve performance over
> NFS.  The “scheme-files” procedure now includes a comment:
>
>   ;; XXX: We don't recurse if we find a symlink.
>
> Would it not be better to fix this instead of adding support for special
> syntax in GUIX_PACKAGE_PATH?

Indeed, apologies for the breakage.

I think the patch below fixes it.  It incurs overhead only when a
symlink is encountered, which is reasonable I think.

Chris & Alex: could you give it a try and report back?

Thanks in advance,
Ludo’.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Type: text/x-patch, Size: 1249 bytes --]

diff --git a/guix/discovery.scm b/guix/discovery.scm
index 292df2bd9..b1731de93 100644
--- a/guix/discovery.scm
+++ b/guix/discovery.scm
@@ -60,11 +60,20 @@ DIRECTORY is not accessible."
                      (case (entry-type absolute properties)
                        ((directory)
                         (append (scheme-files absolute) result))
-                       ((regular symlink)
-                        ;; XXX: We don't recurse if we find a symlink.
+                       ((regular)
                         (if (string-suffix? ".scm" name)
                             (cons absolute result)
                             result))
+                       ((symlink)
+                        (cond ((string-suffix? ".scm" name)
+                               (cons absolute result))
+                              ((stat absolute #f)
+                               =>
+                               (match-lambda
+                                 (#f result)
+                                 ((= stat:type 'directory)
+                                  (append (scheme-files absolute)
+                                          result))))))
                        (else
                         result))))))
               '()

Re: Subdirectories in GUIX_PACKAGE_PATH

[Alex Kost]

Ludovic Courtès (2017-06-30 11:35 +0200) wrote:

> Hello,
>
> Ricardo Wurmus <rekado@elephly.net> skribis:
>
>> Christopher Baines <mail@cbaines.net> writes:
>>
>>> Recently I had problems with the way GUIX_PACKAGE_PATH was working with
>>> govuk-guix [1]. Currently, I'm using a separate directory for the
>>> GUIX_PACKAGE_PATH that contains symlinks to a subset of the Guile
>>> modules necessary for the packages in the repository.
>>>
>>> I think support (whether intentional or otherwise) for this approach was
>>> removed in [2].
>>
>> Looks like this was removed in an attempt to improve performance over
>> NFS.  The “scheme-files” procedure now includes a comment:
>>
>>   ;; XXX: We don't recurse if we find a symlink.
>>
>> Would it not be better to fix this instead of adding support for special
>> syntax in GUIX_PACKAGE_PATH?
>
> Indeed, apologies for the breakage.
>
> I think the patch below fixes it.  It incurs overhead only when a
> symlink is encountered, which is reasonable I think.
>
> Chris & Alex: could you give it a try and report back?

I confirm that the patch you attached fixes this problem, thanks!

I have a (probably not related) question though: in the past, package
'location' contained an *absolute* file name for the packages from
GUIX_PACKAGE_PATH.  Now these locations are *relative* file names.  Is
this intended?

If so, this is a problem for Emacs-Guix: you see, when you try to open a
package location (for example, by pressing an according button in *Guix
Package Info* buffer), it is expanded against 'guix-directory' variable.
So if you open "gnu/packages/guile.scm" location, it works, but now if
it is your location like "my-guix-packages/foo.scm", the wrong
(non-existent) file will be opened.  Previously it worked, because that
location was absolute, like "/home/me/my-guix-packages/foo.scm".

I hope it was clear, let me know, if it's not :-)

-- 
Alex

Re: Subdirectories in GUIX_PACKAGE_PATH

[Ludovic Courtès]

Hi Alex,

Alex Kost <alezost@gmail.com> skribis:

> Ludovic Courtès (2017-06-30 11:35 +0200) wrote:

[...]

>>> Looks like this was removed in an attempt to improve performance over
>>> NFS.  The “scheme-files” procedure now includes a comment:
>>>
>>>   ;; XXX: We don't recurse if we find a symlink.
>>>
>>> Would it not be better to fix this instead of adding support for special
>>> syntax in GUIX_PACKAGE_PATH?
>>
>> Indeed, apologies for the breakage.
>>
>> I think the patch below fixes it.  It incurs overhead only when a
>> symlink is encountered, which is reasonable I think.
>>
>> Chris & Alex: could you give it a try and report back?
>
> I confirm that the patch you attached fixes this problem, thanks!

Great, committed with a test as
960c6ce96d746cf19829ad26e092ec5dad2a5c62.

> I have a (probably not related) question though: in the past, package
> 'location' contained an *absolute* file name for the packages from
> GUIX_PACKAGE_PATH.  Now these locations are *relative* file names.  Is
> this intended?
>
> If so, this is a problem for Emacs-Guix: you see, when you try to open a
> package location (for example, by pressing an according button in *Guix
> Package Info* buffer), it is expanded against 'guix-directory' variable.
> So if you open "gnu/packages/guile.scm" location, it works, but now if
> it is your location like "my-guix-packages/foo.scm", the wrong
> (non-existent) file will be opened.  Previously it worked, because that
> location was absolute, like "/home/me/my-guix-packages/foo.scm".

I’ve noticed this but I didn’t think it was a regression.

Basically Guile supports two “file name canonicalization” modes: one
that returns a file name relative to %load-path, and one that returns an
absolute file name.  By default, when loading modules, we’re in
‘relative’ mode; this is so that modules can be moved on disk and don’t
record their initial location.

I think the last time this was changed was a year ago in
14d5ca2e2e57643b6b4acfb980b18b7474c27e7b.

Thanks,
Ludo’.

Re: Subdirectories in GUIX_PACKAGE_PATH

[Alex Kost]

Ludovic Courtès (2017-07-03 23:53 +0200) wrote:

> Hi Alex,
>
> Alex Kost <alezost@gmail.com> skribis:
>
>> Ludovic Courtès (2017-06-30 11:35 +0200) wrote:
>
> [...]

>> I have a (probably not related) question though: in the past, package
>> 'location' contained an *absolute* file name for the packages from
>> GUIX_PACKAGE_PATH.  Now these locations are *relative* file names.  Is
>> this intended?
>>
>> If so, this is a problem for Emacs-Guix: you see, when you try to open a
>> package location (for example, by pressing an according button in *Guix
>> Package Info* buffer), it is expanded against 'guix-directory' variable.
>> So if you open "gnu/packages/guile.scm" location, it works, but now if
>> it is your location like "my-guix-packages/foo.scm", the wrong
>> (non-existent) file will be opened.  Previously it worked, because that
>> location was absolute, like "/home/me/my-guix-packages/foo.scm".
>
> I’ve noticed this but I didn’t think it was a regression.
>
> Basically Guile supports two “file name canonicalization” modes: one
> that returns a file name relative to %load-path, and one that returns an
> absolute file name.  By default, when loading modules, we’re in
> ‘relative’ mode; this is so that modules can be moved on disk and don’t
> record their initial location.

Thanks for the explanation!

> I think the last time this was changed was a year ago in
> 14d5ca2e2e57643b6b4acfb980b18b7474c27e7b.

-- 
Alex