[bug#36563] [PATCH] guix: Add directory to channel.

[bug#36563] [PATCH] guix: Add directory to channel.

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 447 bytes --]

Hi!

I commonly use a "guix/" subdirectory in upstream that contains updated
or modified packages.  The toplevel `guix.scm' then uses those package
definitions.

It would be nice if I could use that "guix/" subdirectory directly in a
channel.  This means a channel definition should look in a subdirectory
only: often an upstream source archive contains .scm files that do not
compile (guix.scm, build-aux/*.scm, etc.)

WDYT?

Greetings,
janneke


[-- Attachment #2: 0001-guix-Add-directory-to-channel.patch --]
[-- Type: text/x-patch, Size: 7360 bytes --]

From cf85280211ff0060b5283dc5a53cb15ee09a7998 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Tue, 9 Jul 2019 21:01:11 +0200
Subject: [PATCH] guix: Add directory to channel.

Typical use

    (cons* (channel
            (name 'mes)
            (url "https://git.savannah.gnu.org/git/mes.git")
            (directory "guix")
            (branch "wip"))
           %default-channels)

* guix/channels.scm (<channel>): Add directory.
(read-channel-metadata): Fill directory slot.
(checkout->channel-instance): Add #:directory parameter.  Update callers.
(standard-module-derivation): Add directory parameter.  Update callers.
(build-channel-instance): Provide directory argument.
---
 guix/channels.scm | 58 ++++++++++++++++++++++++++++++-----------------
 1 file changed, 37 insertions(+), 21 deletions(-)

diff --git a/guix/channels.scm b/guix/channels.scm
index e6bb9b891b..bd64906832 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2018, 2019 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2019 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -50,6 +51,7 @@
             channel-branch
             channel-commit
             channel-location
+            channel-directory
 
             %default-channels
             guix-channel?
@@ -86,7 +88,8 @@
   (branch    channel-branch (default "master"))
   (commit    channel-commit (default #f))
   (location  channel-location
-             (default (current-source-location)) (innate)))
+             (default (current-source-location)) (innate))
+  (directory channel-directory (default #f)))
 
 (define %default-channels
   ;; Default list of channels.
@@ -141,7 +144,8 @@ file."
                         (name name)
                         (branch branch)
                         (url url)
-                        (commit (get 'commit))))))
+                        (commit (get 'commit))
+                        (directory (get 'directory))))))
                  dependencies))))))
 
 (define (channel-instance-dependencies instance)
@@ -205,13 +209,16 @@ of previously processed channels."
 
 (define* (checkout->channel-instance checkout
                                      #:key commit
-                                     (url checkout) (name 'guix))
+                                     (url checkout)
+                                     (name 'guix)
+                                     directory)
   "Return a channel instance for CHECKOUT, which is assumed to be a checkout
 of COMMIT at URL.  Use NAME as the channel name."
   (let* ((commit  (or commit (make-string 40 #\0)))
          (channel (channel (name name)
                            (commit commit)
-                           (url url))))
+                           (url url)
+                           (directory directory))))
     (channel-instance channel commit checkout)))
 
 (define %self-build-file
@@ -225,11 +232,12 @@ of COMMIT at URL.  Use NAME as the channel name."
   ;; place a set of compiled Guile modules in ~/.config/guix/latest.
   1)
 
-(define (standard-module-derivation name source core dependencies)
+(define (standard-module-derivation name source directory core dependencies)
   "Return a derivation that builds with CORE, a Guix instance, the Scheme
-modules in SOURCE and that depend on DEPENDENCIES, a list of lowerable
-objects.  The assumption is that SOURCE contains package modules to be added
-to '%package-module-path'."
+modules in SOURCE or if DIRECTORY in SOURCE/DIRECTORY and that depend on
+DEPENDENCIES, a list of lowerable objects.  The assumption is that SOURCE or
+SOURCE/DIRECTORY contains package modules to be added to
+'%package-module-path'."
   ;; FIXME: We should load, say SOURCE/.guix-channel.scm, which would allow
   ;; channel publishers to specify things such as the sub-directory where .scm
   ;; files live, files to exclude from the channel, preferred substitute URLs,
@@ -253,20 +261,27 @@ to '%package-module-path'."
             (string-append #$output "/share/guile/site/"
                            (effective-version)))
 
-          (compile-files #$source go
-                         (find-files #$source "\\.scm$"))
-          (mkdir-p (dirname scm))
-          (symlink #$source scm)
+          (let* ((subdir (if #$directory
+                             (string-append "/" #$directory)
+                             ""))
+                 (dir (string-append #$source subdir)))
+            (compile-files dir go
+                           (warn 'files (find-files dir "\\.scm$")))
+            (mkdir-p (dirname scm))
+            (symlink (string-append #$source subdir) scm))
+
           scm)))
 
   (gexp->derivation-in-inferior name build core))
 
 (define* (build-from-source name source
                             #:key core verbose? commit
-                            (dependencies '()))
+                            (dependencies '())
+                            directory)
   "Return a derivation to build Guix from SOURCE, using the self-build script
 contained therein; use COMMIT as the version string.  When CORE is true, build
-package modules under SOURCE using CORE, an instance of Guix."
+package modules under SOURCE or if DIRECTORY under SOURCE/DIRECTORY using
+CORE, an instance of Guix."
   ;; Running the self-build script makes it easier to update the build
   ;; procedure: the self-build script of the Guix-to-be-installed contains the
   ;; right dependencies, build procedure, etc., which the Guix-in-use may not
@@ -293,19 +308,20 @@ package modules under SOURCE using CORE, an instance of Guix."
                #:pull-version %pull-version))
 
       ;; Build a set of modules that extend Guix using the standard method.
-      (standard-module-derivation name source core dependencies)))
+      (standard-module-derivation name source directory core dependencies)))
 
 (define* (build-channel-instance instance
                                  #:optional core (dependencies '()))
   "Return, as a monadic value, the derivation for INSTANCE, a channel
 instance.  DEPENDENCIES is a list of extensions providing Guile modules that
 INSTANCE depends on."
-  (build-from-source (symbol->string
-                      (channel-name (channel-instance-channel instance)))
-                     (channel-instance-checkout instance)
-                     #:commit (channel-instance-commit instance)
-                     #:core core
-                     #:dependencies dependencies))
+  (let ((channel (channel-instance-channel instance)))
+    (build-from-source (symbol->string (channel-name channel))
+                       (channel-instance-checkout instance)
+                       #:commit (channel-instance-commit instance)
+                       #:core core
+                       #:dependencies dependencies
+                       #:directory (channel-directory channel))))
 
 (define (resolve-dependencies instances)
   "Return a procedure that, given one of the elements of INSTANCES, returns
-- 
2.21.0


[-- Attachment #3: Type: text/plain, Size: 152 bytes --]


-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#36563] [PATCH] guix: Add directory to channel.

[Ludovic Courtès]

Hello!

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> I commonly use a "guix/" subdirectory in upstream that contains updated
> or modified packages.  The toplevel `guix.scm' then uses those package
> definitions.
>
> It would be nice if I could use that "guix/" subdirectory directly in a
> channel.  This means a channel definition should look in a subdirectory
> only: often an upstream source archive contains .scm files that do not
> compile (guix.scm, build-aux/*.scm, etc.)

That makes a lot of sense to me.

I think that, instead of letting the user specify the right directory in
~/.config/guix/channels.scm, it should be the channel author that
specifies the sub-directory.

There’s already support for a ‘.guix-channel’ file so this is where this
‘directory’ thing could be added (actually there’s a FIXME in
channels.scm about this particular use case :-)).

WDYT?

Ludo’.

[bug#36563] [PATCH] guix: Add directory to channel.

[Jan Nieuwenhuizen]

[-- Attachment #1: Type: text/plain, Size: 837 bytes --]

Ludovic Courtès writes:

Hi!

> That makes a lot of sense to me.
>
> I think that, instead of letting the user specify the right directory in
> ~/.config/guix/channels.scm, it should be the channel author that
> specifies the sub-directory.
>
> There’s already support for a ‘.guix-channel’ file so this is where this
> ‘directory’ thing could be added (actually there’s a FIXME in
> channels.scm about this particular use case :-)).
>
> WDYT?

Yes, I agree that this should be the first option to support.  Attached
is a second attempt: I removed the FIXME and moved `directory' to
channel-metadata.

A downside could be that this could make a channel user dependent on
upstream to provide this `.guix-channel' file; probably best to worry
about that when the need arises :)

Greetings,
janneke


[-- Attachment #2: 0001-channels-Add-optional-directory-to-channel-metadata.patch --]
[-- Type: text/x-patch, Size: 8016 bytes --]

From 5d78a7eab391ea93c044d85736706b333c2dfee8 Mon Sep 17 00:00:00 2001
From: Jan Nieuwenhuizen <janneke@gnu.org>
Date: Sat, 13 Jul 2019 16:31:50 +0200
Subject: [PATCH] channels: Add optional directory to channel metadata.

* guix/channels.scm (<channel-metadata>): Add directory slot.  Update users.
(read-channel-metadata-from-source): New function.
(standard-module-derivation): Use it.
* doc/guix.texi (Package Modules in a Subdirectory): New subsection.
---
 doc/guix.texi     | 23 ++++++++++++
 guix/channels.scm | 93 ++++++++++++++++++++++++++---------------------
 2 files changed, 75 insertions(+), 41 deletions(-)

diff --git a/doc/guix.texi b/doc/guix.texi
index 5a8ad7ebda..8bc00a5350 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -3921,6 +3921,29 @@ For the sake of reliability and maintainability, you should avoid dependencies
 on channels that you don't control, and you should aim to keep the number of
 dependencies to a minimum.
 
+@cindex subdirectory, channels
+@subsection Package Modules in a Subdirectory
+
+A developer may include a @file{guix} subdirectory in their upstream source
+that contains additional or modified Guix package descriptions.  Typically,
+their toplevel @file{guix.scm} then uses those package definitions:
+
+@lisp
+(define %source-dir (dirname (current-filename)))
+(add-to-load-path (string-append %source-dir "/guix"))
+(use-modules @dots{})
+@dots{}
+@end lisp
+
+By adding a meta-data file @file{.guix-channel} that specifies the
+subdirectory that contains the Guix package descriptions it can be used
+directly as a channel:
+
+@lisp
+(channel
+ (directory "guix"))
+@end lisp
+
 @subsection Replicating Guix
 
 @cindex pinning, channels
diff --git a/guix/channels.scm b/guix/channels.scm
index e6bb9b891b..614639f091 100644
--- a/guix/channels.scm
+++ b/guix/channels.scm
@@ -1,6 +1,7 @@
 ;;; GNU Guix --- Functional package management for GNU
 ;;; Copyright © 2018, 2019 Ludovic Courtès <ludo@gnu.org>
 ;;; Copyright © 2018 Ricardo Wurmus <rekado@elephly.net>
+;;; Copyright © 2019 Jan (janneke) Nieuwenhuizen <janneke@gnu.org>
 ;;;
 ;;; This file is part of GNU Guix.
 ;;;
@@ -107,9 +108,10 @@
   (checkout  channel-instance-checkout))
 
 (define-record-type <channel-metadata>
-  (channel-metadata version dependencies)
+  (channel-metadata version directory dependencies)
   channel-metadata?
   (version       channel-metadata-version)
+  (directory     channel-metadata-directory)
   (dependencies  channel-metadata-dependencies))
 
 (define (channel-reference channel)
@@ -119,18 +121,18 @@
     (#f      `(branch . ,(channel-branch channel)))
     (commit  `(commit . ,(channel-commit channel)))))
 
-(define (read-channel-metadata instance)
-  "Return a channel-metadata record read from the channel INSTANCE's
-description file, or return #F if the channel instance does not include the
-file."
-  (let* ((source (channel-instance-checkout instance))
-         (meta-file (string-append source "/.guix-channel")))
+(define (read-channel-metadata-from-source source)
+  "Return a channel-metadata record read from channel's SOURCE/.guix-channel
+description file, or return #F if SOURCE/.guix-channel does not exist."
+  (let ((meta-file (string-append source "/.guix-channel")))
     (and (file-exists? meta-file)
-         (and-let* ((raw (call-with-input-file meta-file read))
-                    (version (and=> (assoc-ref raw 'version) first))
-                    (dependencies (or (assoc-ref raw 'dependencies) '())))
+         (let* ((raw (call-with-input-file meta-file read))
+                (version (and=> (assoc-ref raw 'version) first))
+                (directory (and=> (assoc-ref raw 'directory) first))
+                (dependencies (or (assoc-ref raw 'dependencies) '())))
            (channel-metadata
             version
+            directory
             (map (lambda (item)
                    (let ((get (lambda* (key #:optional default)
                                 (or (and=> (assoc-ref item key) first) default))))
@@ -144,12 +146,18 @@ file."
                         (commit (get 'commit))))))
                  dependencies))))))
 
+(define (read-channel-metadata instance)
+  "Return a channel-metadata record read from the channel INSTANCE's
+description file, or return #F if the channel instance does not include the
+file."
+  (read-channel-metadata-from-source (channel-instance-checkout instance)))
+
 (define (channel-instance-dependencies instance)
   "Return the list of channels that are declared as dependencies for the given
 channel INSTANCE."
   (match (read-channel-metadata instance)
     (#f '())
-    (($ <channel-metadata> version dependencies)
+    (($ <channel-metadata> version directory dependencies)
      dependencies)))
 
 (define* (latest-channel-instances store channels #:optional (previous-channels '()))
@@ -230,36 +238,39 @@ of COMMIT at URL.  Use NAME as the channel name."
 modules in SOURCE and that depend on DEPENDENCIES, a list of lowerable
 objects.  The assumption is that SOURCE contains package modules to be added
 to '%package-module-path'."
-  ;; FIXME: We should load, say SOURCE/.guix-channel.scm, which would allow
-  ;; channel publishers to specify things such as the sub-directory where .scm
-  ;; files live, files to exclude from the channel, preferred substitute URLs,
-  ;; etc.
-
-  (define build
-    ;; This is code that we'll run in CORE, a Guix instance, with its own
-    ;; modules and so on.  That way, we make sure these modules are built for
-    ;; the right Guile version, with the right dependencies, and that they get
-    ;; to see the right (gnu packages …) modules.
-    (with-extensions dependencies
-      #~(begin
-          (use-modules (guix build compile)
-                       (guix build utils)
-                       (srfi srfi-26))
-
-          (define go
-            (string-append #$output "/lib/guile/" (effective-version)
-                           "/site-ccache"))
-          (define scm
-            (string-append #$output "/share/guile/site/"
-                           (effective-version)))
-
-          (compile-files #$source go
-                         (find-files #$source "\\.scm$"))
-          (mkdir-p (dirname scm))
-          (symlink #$source scm)
-          scm)))
-
-  (gexp->derivation-in-inferior name build core))
+
+  (let* ((metadata (read-channel-metadata-from-source source))
+         (directory (and=> metadata channel-metadata-directory)))
+
+    (define build
+      ;; This is code that we'll run in CORE, a Guix instance, with its own
+      ;; modules and so on.  That way, we make sure these modules are built for
+      ;; the right Guile version, with the right dependencies, and that they get
+      ;; to see the right (gnu packages …) modules.
+      (with-extensions dependencies
+        #~(begin
+            (use-modules (guix build compile)
+                         (guix build utils)
+                         (srfi srfi-26))
+
+            (define go
+              (string-append #$output "/lib/guile/" (effective-version)
+                             "/site-ccache"))
+            (define scm
+              (string-append #$output "/share/guile/site/"
+                             (effective-version)))
+
+            (let* ((subdir (if #$directory
+                               (string-append "/" #$directory)
+                               ""))
+                   (dir (string-append #$source subdir)))
+              (compile-files dir go (find-files dir "\\.scm$"))
+              (mkdir-p (dirname scm))
+              (symlink (string-append #$source subdir) scm))
+
+            scm)))
+
+    (gexp->derivation-in-inferior name build core)))
 
 (define* (build-from-source name source
                             #:key core verbose? commit
-- 
2.21.0


[-- Attachment #3: Type: text/plain, Size: 154 bytes --]



-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#36563] [PATCH] guix: Add directory to channel.

[Ludovic Courtès]

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Yes, I agree that this should be the first option to support.  Attached
> is a second attempt: I removed the FIXME and moved `directory' to
> channel-metadata.

That was fast!

> A downside could be that this could make a channel user dependent on
> upstream to provide this `.guix-channel' file; probably best to worry
> about that when the need arises :)

If upstream doesn’t provide this file, then the channel is effectively
unusable because of a file name/module name mismatch, no?

> From 5d78a7eab391ea93c044d85736706b333c2dfee8 Mon Sep 17 00:00:00 2001
> From: Jan Nieuwenhuizen <janneke@gnu.org>
> Date: Sat, 13 Jul 2019 16:31:50 +0200
> Subject: [PATCH] channels: Add optional directory to channel metadata.
>
> * guix/channels.scm (<channel-metadata>): Add directory slot.  Update users.
> (read-channel-metadata-from-source): New function.
> (standard-module-derivation): Use it.
> * doc/guix.texi (Package Modules in a Subdirectory): New subsection.

[...]

> +@cindex subdirectory, channels
> +@subsection Package Modules in a Subdirectory
> +
> +A developer may include a @file{guix} subdirectory in their upstream source
> +that contains additional or modified Guix package descriptions.  Typically,
> +their toplevel @file{guix.scm} then uses those package definitions:

What about framing it along these lines:

  As a channel author, you may want to keep your channel modules in a
  sub-directory.  In that case, blah…

?

> +By adding a meta-data file @file{.guix-channel} that specifies the
> +subdirectory that contains the Guix package descriptions it can be used
> +directly as a channel:
> +
> +@lisp
> +(channel
> + (directory "guix"))

It should be:

  (channel
    (version 0)
    (directory "something"))

(‘read-channel-metadata’ currently ignores ‘version’, but that’s a bug
that we should fix afterwards, IMO.)

> +            (let* ((subdir (if #$directory
> +                               (string-append "/" #$directory)
> +                               ""))
> +                   (dir (string-append #$source subdir)))

Maybe s/dir/source/?

Otherwise LGTM!

Thanks,
Ludo’.

bug#36563: [PATCH] guix: Add directory to channel.

[Jan Nieuwenhuizen]

Ludovic Courtès writes:

Hi!

> What about framing it along these lines:
>
>   As a channel author, you may want to keep your channel modules in a
>   sub-directory.  In that case, blah…
>
> ?

Better, changed it to

--8<---------------cut here---------------start------------->8---
@cindex subdirectory, channels
@subsection Package Modules in a Sub-directory

As a channel author, you may want to keep your channel modules in a
sub-directory.  If your modules are in the sub-directory @file{guix}, you must
add a meta-data file @file{.guix-channel} that contains:

@lisp
(channel
  (version 0)
  (directory "guix"))
@end lisp
--8<---------------cut here---------------end--------------->8---

> (‘read-channel-metadata’ currently ignores ‘version’, but that’s a bug
> that we should fix afterwards, IMO.)

OK.

>> +            (let* ((subdir (if #$directory
>> +                               (string-append "/" #$directory)
>> +                               ""))
>> +                   (dir (string-append #$source subdir)))
>
> Maybe s/dir/source/?

Sure, done.

> Otherwise LGTM!

Great, thanks; pushed to master as 53f21642729e4786141c072dd835b04cb85dfe28

Greetings,
janneke

-- 
Jan Nieuwenhuizen <janneke@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.com

[bug#36563] [PATCH] guix: Add directory to channel.

[Ludovic Courtès]

Hi,

Jan Nieuwenhuizen <janneke@gnu.org> skribis:

> Great, thanks; pushed to master as 53f21642729e4786141c072dd835b04cb85dfe28

Awesome, thank you!

Ludo’.