GUIX_EXTENSIONS_PATH error?

GUIX_EXTENSIONS_PATH error?

[zimoun]

Hi,

In the mood for documenting the recent GUIX_EXTENSIONS_PATH, I am
missing something because I hit an error.  Before diving, maybe my
daily coffee dose is not enough…


--8<---------------cut here---------------start------------->8---
$ cat /tmp/foo/bar.scm
(define-module (guix extensions bar)
  #:use-module (guix scripts))


(define-command (guix-bar . args)
  (category extension)
  (synopsis "a bar with beers")

  (pk "dumb"))


$ export GUIX_EXTENSIONS_PATH=/tmp/foo

$ guix --help
Usage: guix COMMAND ARGS...
Run COMMAND with ARGS.

COMMAND must be one of the sub-commands listed below:
Backtrace:
           4 (primitive-load "/home/simon/.config/guix/current/bin/guix")
In ice-9/boot-9.scm:
  1731:15  3 (with-exception-handler #<procedure 7f13f2064b70 at ice-9/boot-9.scm:1815:7 (exn)> _ # _ …)
In guix/ui.scm:
  2105:20  2 (show-guix-help)
In srfi/srfi-1.scm:
   691:23  1 (filter-map #<procedure source-file-command (file)> _ . _)
In guix/ui.scm:
   2023:2  0 (source-file-command _)

guix/ui.scm:2023:2: In procedure source-file-command:
Throw to key `match-error' with args `("match" "no matching pattern" ("tmp" "foo" "bar.scm"))'.
--8<---------------cut here---------------end--------------->8---

What do I miss?


All the best,
simon

Re: GUIX_EXTENSIONS_PATH error?

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> In the mood for documenting the recent GUIX_EXTENSIONS_PATH, I am
> missing something because I hit an error.

It’s not your fault.  It’s hard to do this without documentation.

> --8<---------------cut here---------------start------------->8---
> $ cat /tmp/foo/bar.scm
> (define-module (guix extensions bar)
>   #:use-module (guix scripts))

This right here is the problem.  The module name and the file name do
not match.  The file must be
$GUIX_EXTENSIONS_PATH/guix/extensions/bar.scm and provide the module
(guix extensions bar).

> $ export GUIX_EXTENSIONS_PATH=/tmp/foo
>
> $ guix --help
> Usage: guix COMMAND ARGS...
> Run COMMAND with ARGS.
>
> COMMAND must be one of the sub-commands listed below:
> Backtrace:
>            4 (primitive-load "/home/simon/.config/guix/current/bin/guix")
> In ice-9/boot-9.scm:
>   1731:15  3 (with-exception-handler #<procedure 7f13f2064b70 at ice-9/boot-9.scm:1815:7 (exn)> _ # _ …)
> In guix/ui.scm:
>   2105:20  2 (show-guix-help)
> In srfi/srfi-1.scm:
>    691:23  1 (filter-map #<procedure source-file-command (file)> _ . _)
> In guix/ui.scm:
>    2023:2  0 (source-file-command _)
>
> guix/ui.scm:2023:2: In procedure source-file-command:
> Throw to key `match-error' with args `("match" "no matching pattern" ("tmp" "foo" "bar.scm"))'.

But it still shouldn’t throw an error like that.  I’ll fix it.  Sorry!

-- 
Ricardo

Re: GUIX_EXTENSIONS_PATH error?

[zimoun]

Hi,

On Fri, 15 Jan 2021 at 21:45, Ricardo Wurmus <rekado@elephly.net> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:
>
>> In the mood for documenting the recent GUIX_EXTENSIONS_PATH, I am
>> missing something because I hit an error.
>
> It’s not your fault.  It’s hard to do this without documentation.

It’s so cool that I would like to document it. ;-)  Sorry to not dive in
the code first.

>> --8<---------------cut here---------------start------------->8---
>> $ cat /tmp/foo/bar.scm
>> (define-module (guix extensions bar)
>>   #:use-module (guix scripts))
>
> This right here is the problem.  The module name and the file name do
> not match.  The file must be
> $GUIX_EXTENSIONS_PATH/guix/extensions/bar.scm and provide the module
> (guix extensions bar).

Ah yeah, that makes sense.  Now, no error but nothing showed:

--8<---------------cut here---------------start------------->8---
$ cat $GUIX_EXTENSIONS_PATH/guix/extensions/bar.scm
(define-module (guix extensions bar)
  #:use-module (guix scripts)
  #:export (guix-bar)
  )


(define-command (guix-bar . args)
  (category extension)
  (synopsis "a bar with beer")

  (pk "dumb"))

$ guix --help | grep bar
--8<---------------cut here---------------end--------------->8---


Cheers,
simon

Re: GUIX_EXTENSIONS_PATH error?

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

>>> --8<---------------cut here---------------start------------->8---
>>> $ cat /tmp/foo/bar.scm
>>> (define-module (guix extensions bar)
>>>   #:use-module (guix scripts))
>>
>> This right here is the problem.  The module name and the file name do
>> not match.  The file must be
>> $GUIX_EXTENSIONS_PATH/guix/extensions/bar.scm and provide the module
>> (guix extensions bar).
>
> Ah yeah, that makes sense.  Now, no error but nothing showed:
>
> --8<---------------cut here---------------start------------->8---
> $ cat $GUIX_EXTENSIONS_PATH/guix/extensions/bar.scm
> (define-module (guix extensions bar)
>   #:use-module (guix scripts)
>   #:export (guix-bar)
>   )
>
>
> (define-command (guix-bar . args)
>   (category extension)
>   (synopsis "a bar with beer")
>
>   (pk "dumb"))
>
> $ guix --help | grep bar
> --8<---------------cut here---------------end--------------->8---

Oh, my bad:  $GUIX_EXTENSIONS_PATH must actually include the
“/guix/extensions” sub-directory.

So if the file is at /tmp/foo/guix/extensions/bar.scm then
$GUIX_EXTENSIONS_PATH should be /tmp/foo/guix/extensions.

Sorry for the confusion!

-- 
Ricardo

Re: GUIX_EXTENSIONS_PATH error?

[zimoun]

Hi,

On Fri, 15 Jan 2021 at 23:25, Ricardo Wurmus <rekado@elephly.net> wrote:

> Oh, my bad:  $GUIX_EXTENSIONS_PATH must actually include the
> “/guix/extensions” sub-directory.

Thanks.  It works! \o/
Awesome :-)


> So if the file is at /tmp/foo/guix/extensions/bar.scm then
> $GUIX_EXTENSIONS_PATH should be /tmp/foo/guix/extensions.

Well, I will propose to have the thing as you said before.  Because
fixing (guix extensions bar) and the path $GUIX_EXTENSIONS_PATH/foo
seems reasonable but $GUIX_EXTENSIONS_PATH/foo/guix/extensions is too
redundant.
Moreover, it could nice to have GUIX_EXTENSIONS_PATH look by default
in ~/.config/guix/extensions, i.e., by default
GUIX_EXTENSIONS_PATH=~/.config.


Cheers,
simon

Re: GUIX_EXTENSIONS_PATH error?

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

>> So if the file is at /tmp/foo/guix/extensions/bar.scm then
>> $GUIX_EXTENSIONS_PATH should be /tmp/foo/guix/extensions.
>
> Well, I will propose to have the thing as you said before.  Because
> fixing (guix extensions bar) and the path $GUIX_EXTENSIONS_PATH/foo
> seems reasonable but $GUIX_EXTENSIONS_PATH/foo/guix/extensions is too
> redundant.

It does seem so, but the idea was to reduce the number of files that
Guix will have to parse before finding an extension.

Extensions would be installed to the same prefix /share/guix/extensions,
so Guix can set GUIX_EXTENSIONS_PATH to $profile/share/guix/extensions
and not worry about having to parse all the Scheme files in
$profile/share, which may be a lot — for example Guix’s own commands are
defined in files under $profile/share, so they would end up as
extensions…

We can change this, but we’d need to agree on an as yet unused directory
as the root for extensions.

> Moreover, it could nice to have GUIX_EXTENSIONS_PATH look by default
> in ~/.config/guix/extensions, i.e., by default
> GUIX_EXTENSIONS_PATH=~/.config.

The last part of this sentence is what I meant above: we need to avoid
that, because that would cause
~/.config/guix/current/share/guile/site/3.0/guix/scripts/ to be included
in the search for extensions.

I have added a search path specification to the “guix” package itself,
so that it will set GUIX_EXTENSIONS_PATH to
$profile/share/guix/extensions automatically.  The idea is to update the
format of the “channels.scm” file to allow for the installation of extra
packages into the “guix pull” profile (by default that’s
~/.config/guix/current).

This way I could say that I want the freshly pulled Guix to also install
the “gwl” package in the same profile, and that new Guix would
automatically have the “gwl” extension’s “workflow” sub-command.

-- 
Ricardo

Re: GUIX_EXTENSIONS_PATH error?

[zimoun]

Re,

On Fri, 15 Jan 2021 at 23:42, Ricardo Wurmus <rekado@elephly.net> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:
>
> >> So if the file is at /tmp/foo/guix/extensions/bar.scm then
> >> $GUIX_EXTENSIONS_PATH should be /tmp/foo/guix/extensions.
> >
> > Well, I will propose to have the thing as you said before.  Because
> > fixing (guix extensions bar) and the path $GUIX_EXTENSIONS_PATH/foo
> > seems reasonable but $GUIX_EXTENSIONS_PATH/foo/guix/extensions is too
> > redundant.
>
> It does seem so, but the idea was to reduce the number of files that
> Guix will have to parse before finding an extension.
>
> Extensions would be installed to the same prefix /share/guix/extensions,
> so Guix can set GUIX_EXTENSIONS_PATH to $profile/share/guix/extensions
> and not worry about having to parse all the Scheme files in
> $profile/share, which may be a lot — for example Guix’s own commands are
> defined in files under $profile/share, so they would end up as
> extensions…

Ah I see.

> We can change this, but we’d need to agree on an as yet unused directory
> as the root for extensions.

Well, I have to rehash a bit and then see what could be "user
friendly" without typing redundancy.

> > Moreover, it could nice to have GUIX_EXTENSIONS_PATH look by default
> > in ~/.config/guix/extensions, i.e., by default
> > GUIX_EXTENSIONS_PATH=~/.config.
>
> The last part of this sentence is what I meant above: we need to avoid
> that, because that would cause
> ~/.config/guix/current/share/guile/site/3.0/guix/scripts/ to be included
> in the search for extensions.

Ah I see.  But we could filter out this "current" from the path.  In
the case of default to "~/.config/guix/extensions/".

Well, need some thoughts...

> I have added a search path specification to the “guix” package itself,
> so that it will set GUIX_EXTENSIONS_PATH to
> $profile/share/guix/extensions automatically.  The idea is to update the
> format of the “channels.scm” file to allow for the installation of extra
> packages into the “guix pull” profile (by default that’s
> ~/.config/guix/current).

...because yeah that's the aim. :-)  Provide extensions as channels.


Cheers,
simon

Re: GUIX_EXTENSIONS_PATH error?

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

>> I have added a search path specification to the “guix” package itself,
>> so that it will set GUIX_EXTENSIONS_PATH to
>> $profile/share/guix/extensions automatically.  The idea is to update the
>> format of the “channels.scm” file to allow for the installation of extra
>> packages into the “guix pull” profile (by default that’s
>> ~/.config/guix/current).
>
> ...because yeah that's the aim. :-)  Provide extensions as channels.

It’s very similar to providing extensions as channels, but without
actually using the channel mechanism.   Channels are a little different
from regular packages.  They consist solely of Guile source files, they
are not compiled ahead of time but built during “guix pull”, and they
cannot have inputs other than channels.

Extensions like the GWL have non-trivial dependencies, which cannot be
included when the GWL is treated as a channel.

Maybe there’s a way to express channels themselves as regular packages,
but I’m not seeing it.  So until we can unify this, extensions should be
installed as packages once “guix pull” has finished building the new
Guix (with all its channels).

Does this make sense?

-- 
Ricardo

Re: GUIX_EXTENSIONS_PATH error?

[zimoun]

[-- Attachment #1: Type: text/plain, Size: 2252 bytes --]

Hi Ricardo,

About the precedence, i.e., extension that overrides default command,
see patch#45910.

<http://issues.guix.gnu.org/45910>


On Fri, 15 Jan 2021 at 23:42, Ricardo Wurmus <rekado@elephly.net> wrote:
> zimoun <zimon.toutoune@gmail.com> writes:

> We can change this, but we’d need to agree on an as yet unused directory
> as the root for extensions.

I do not know, maybe I will change my mind after a good ol’ black
coffee. :-)

We could say that:

 1. the prototype of GUIX_EXTENSIONS_PATH is path/to/guix
 2. the folder /extensions is implicitly appended 
 3. ~/.config/guix is implicitly appended

The patch attached does that.  But, the definition of the package ’guix’
needs to be tweaked (not done) in agreement, especially:

--8<---------------cut here---------------start------------->8---
      (native-search-paths
       (list (search-path-specification
              (variable "GUIX_EXTENSIONS_PATH")
              (files '("share/guix/extensions")))))
--8<---------------cut here---------------end--------------->8---


>> Moreover, it could nice to have GUIX_EXTENSIONS_PATH look by default
>> in ~/.config/guix/extensions, i.e., by default
>> GUIX_EXTENSIONS_PATH=~/.config.
>
> The last part of this sentence is what I meant above: we need to avoid
> that, because that would cause
> ~/.config/guix/current/share/guile/site/3.0/guix/scripts/ to be included
> in the search for extensions.

It is easy to filter out by adding rules in ’extensions-directories’. :-)

> I have added a search path specification to the “guix” package itself,
> so that it will set GUIX_EXTENSIONS_PATH to
> $profile/share/guix/extensions automatically.  The idea is to update the
> format of the “channels.scm” file to allow for the installation of extra
> packages into the “guix pull” profile (by default that’s
> ~/.config/guix/current).

I see.  Well, I need to fail myself to really understand…

> This way I could say that I want the freshly pulled Guix to also install
> the “gwl” package in the same profile, and that new Guix would
> automatically have the “gwl” extension’s “workflow” sub-command.

I agree.


Cheers,
simon
--


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: trivial.patch --]
[-- Type: text/x-diff, Size: 629 bytes --]

diff --git a/guix/ui.scm b/guix/ui.scm
index ad78d5cedd..e918b5b64f 100644
--- a/guix/ui.scm
+++ b/guix/ui.scm
@@ -2066,8 +2066,10 @@ contain a 'define-command' form."
 (define (extension-directories)
   "Return the list of directories containing Guix extensions."
   (filter file-exists?
-          (parse-path
-           (getenv "GUIX_EXTENSIONS_PATH"))))
+          (map (cut string-append <> "/extensions")
+               (parse-path
+                (string-append (config-directory) ":"
+                 (getenv "GUIX_EXTENSIONS_PATH"))))))
 
 (define (commands)
   "Return the list of commands, alphabetically sorted."

GUIX_EXTENSIONS_PATH pattern

[zimoun]

Hi Ricardo,

>> We can change this, but we’d need to agree on an as yet unused directory
>> as the root for extensions.

> We could say that:
>
>  1. the prototype of GUIX_EXTENSIONS_PATH is path/to/guix
>  2. the folder /extensions is implicitly appended 
>  3. ~/.config/guix is implicitly appended

Do we agree on this default?  Which implies…

> The patch attached does that.  But, the definition of the package ’guix’
> needs to be tweaked (not done) in agreement, especially:
>
> --8<---------------cut here---------------start------------->8---
>       (native-search-paths
>        (list (search-path-specification
>               (variable "GUIX_EXTENSIONS_PATH")
>               (files '("share/guix/extensions")))))
> --8<---------------cut here---------------end--------------->8---

…and…

>>> Moreover, it could nice to have GUIX_EXTENSIONS_PATH look by default
>>> in ~/.config/guix/extensions, i.e., by default
>>> GUIX_EXTENSIONS_PATH=~/.config.
>>
>> The last part of this sentence is what I meant above: we need to avoid
>> that, because that would cause
>> ~/.config/guix/current/share/guile/site/3.0/guix/scripts/ to be included
>> in the search for extensions.
>
> It is easy to filter out by adding rules in ’extensions-directories’. :-)

…that.

To me, the easiest is to fix the module name as ’(guix extensions foo)’
and since that is fixed, GUIX_EXTENSIONS_PATH should not declare
explicitly ’/extensions’.

I get your other points about channels.  From my understanding, the
pattern should fixed and then some other parts tweaked in agreement.

WDYT?


BTW, I have not tried to adapt and makes it work for extension as
subcommand.  I have in mind: “guix git log” or “guix system foo”.  Well,
these subcommands are not defined as ’define-command’ but we could take
this road.  WDYT?


All the best,
simon

Re: GUIX_EXTENSIONS_PATH pattern

[Ricardo Wurmus]

Hi,

sorry for the delay!

zimoun <zimon.toutoune@gmail.com> writes:

>>> We can change this, but we’d need to agree on an as yet unused directory
>>> as the root for extensions.
>
>> We could say that:
>>
>>  1. the prototype of GUIX_EXTENSIONS_PATH is path/to/guix
>>  2. the folder /extensions is implicitly appended 
>>  3. ~/.config/guix is implicitly appended
>
> Do we agree on this default?  Which implies…

I’m really not sure.

1. do you mean “share/guix” here?

2. I don’t know if this is a good idea, because the extensions really
are in the “extensions” sub-directory.  Why would the variable refer to
a more generic parent directory?

3. My opinion is that ~/.config/guix should not get special treatment.
People can use different profiles for “guix pull”.

>> The patch attached does that.  But, the definition of the package ’guix’
>> needs to be tweaked (not done) in agreement, especially:
>>
>> --8<---------------cut here---------------start------------->8---
>>       (native-search-paths
>>        (list (search-path-specification
>>               (variable "GUIX_EXTENSIONS_PATH")
>>               (files '("share/guix/extensions")))))
>> --8<---------------cut here---------------end--------------->8---
>
> …and…
>
>>>> Moreover, it could nice to have GUIX_EXTENSIONS_PATH look by default
>>>> in ~/.config/guix/extensions, i.e., by default
>>>> GUIX_EXTENSIONS_PATH=~/.config.
>>>
>>> The last part of this sentence is what I meant above: we need to avoid
>>> that, because that would cause
>>> ~/.config/guix/current/share/guile/site/3.0/guix/scripts/ to be included
>>> in the search for extensions.
>>
>> It is easy to filter out by adding rules in ’extensions-directories’. :-)
>
> …that.
>
> To me, the easiest is to fix the module name as ’(guix extensions foo)’
> and since that is fixed, GUIX_EXTENSIONS_PATH should not declare
> explicitly ’/extensions’.

Perhaps we can invite some other people to comment on this.  I have
become confused about what is the best way forward (which is one of the
reasons why it took me so long to reply).

> BTW, I have not tried to adapt and makes it work for extension as
> subcommand.  I have in mind: “guix git log” or “guix system foo”.  Well,
> these subcommands are not defined as ’define-command’ but we could take
> this road.  WDYT?

I haven’t really thought about this, but extensions for sub-commands of
sub-commands currently only work by replacing the parent command, so
they cannot be composed.

We probably could add a mechanism to that (guix extensions system foo)
would extend the “guix system” command, but this would require more work
as the “system” command is responsible for dispatching to the
sub-command.  I don’t know if there’s a good generic way to implement
this.  “guix system foo” could bypass the “guix system” entry point if
(guix extensions system foo) exists, but this may have unwanted
consequences.

-- 
Ricardo

Re: GUIX_EXTENSIONS_PATH pattern

[zimoun]

Hi Ricardo,

On Mon, 08 Feb 2021 at 23:37, Ricardo Wurmus <rekado@elephly.net> wrote:

> 1. do you mean “share/guix” here?
>
> 2. I don’t know if this is a good idea, because the extensions really
> are in the “extensions” sub-directory.  Why would the variable refer to
> a more generic parent directory?

Let take an example, the recent Ludo’s explorer.

  <https://notabug.org/civodul/guix-explorer/>

It is easy to transform it as an extension and then use “guix explore”.


But 2 things are strict:

 - user side: GUIX_EXTENSIONS_PATH
 - extensioner side: define-module

Maybe I am wrong and I miss something.  From my understanding,

 - the file must be located at $GUIX_EXTENSIONS_PATH
 - the module must be defined as (guix extension explore)

because of:

--8<---------------cut here---------------start------------->8---
    (match (search-path (extension-directories)
                        (format #f "~a.scm" command))
[...]
       (resolve-interface `(guix extensions ,command)))))
--8<---------------cut here---------------end--------------->8---


So, taking the previous example, it would be easy to say: hey, clone
this, export GUIX_EXTENSIONS_PATH as /path/to/checkout and run “guix
explore”.  But, to work, the repo must have the structure:

 - guix-explorer
   - guix
     - extensions
       - explore.scm

because the module and in the same time $GUIX_EXTENSIONS_PATH must be
“guix-explore/guix/extensions“.  I feel something is wrong.  And I
initially proposed to append ’/guix/extensions’ by default to
GUIX_EXTENSIONS_PATH in ’extension-directories’.

I do not understand why it should be a problem.


BTW, for packages as GWL, it does not change.  It is just an agreement
between the two sides, right?


> 3. My opinion is that ~/.config/guix should not get special treatment.
> People can use different profiles for “guix pull”.

Ok, fine with me.  It is reasonable to have GUIX_EXTENSIONS_PATH defined
by user preference in their ~/.bash_profile or whatever.


> Perhaps we can invite some other people to comment on this.  I have
> become confused about what is the best way forward (which is one of the
> reasons why it took me so long to reply).

For sure!  Even if we are already on guix-devel. :-)


>> BTW, I have not tried to adapt and makes it work for extension as
>> subcommand.  I have in mind: “guix git log” or “guix system foo”.  Well,
>> these subcommands are not defined as ’define-command’ but we could take
>> this road.  WDYT?
>
> I haven’t really thought about this, but extensions for sub-commands of
> sub-commands currently only work by replacing the parent command, so
> they cannot be composed.

Indeed, it is a separated topic. :-) First, the subcommand should use
something like ’define-command’.


> We probably could add a mechanism to that (guix extensions system foo)
> would extend the “guix system” command, but this would require more work
> as the “system” command is responsible for dispatching to the
> sub-command.  I don’t know if there’s a good generic way to implement
> this.  “guix system foo” could bypass the “guix system” entry point if
> (guix extensions system foo) exists, but this may have unwanted
> consequences.

Yeah, you have right.  It needs more thoughts and discussions.


Cheers,
simon