Questions regarding substitutes with debug output

Questions regarding substitutes with debug output

[Olivier Dion via]

Hi,

I require debugging symbols of multiple packages.  Some of them already
have a debug output from the main guix channel, but others don't.  So
I had in mind to make a channel and publish the substitutes with the
debug outputs.

Regarding this specific case, I have a question:

  Let's say I need firefox:out + firefox:debug.  My channel does provide
  these outputs, guix's main channel only provides the firefox:out.

  I need to specify the package programmatically as a string in Guile.
  More specifically in the process packages field of Guix Workflow Language.

  How could I make sure that Guix use my version of Firefox over the one
  defined by the main channel?  My guess is to change the name of my
  package to something like "my-firefox".  But it would be nicer if I
  don't have to do so.  Perhaps there's a way to force channel
  preference?

  I know that we can specify version like "firefox@99" but what about
  outputs?  Is it possible to do (list "firefox:out" "firefox:debug").?

Regards,
old

-- 
Olivier Dion
oldiob.dev

Re: Questions regarding substitutes with debug output

[Maxim Cournoyer]

Hi Olivier,

Olivier Dion via <help-guix@gnu.org> writes:

> Hi,
>
> I require debugging symbols of multiple packages.  Some of them already
> have a debug output from the main guix channel, but others don't.  So
> I had in mind to make a channel and publish the substitutes with the
> debug outputs.

In general, if you need debug symbols for something, I think it's OK to
contribute adding them to Guix proper (in a debug output), unless they
make like >= 1 GiB (as I can imagine may be the case for IceCat).

[...]

>   How could I make sure that Guix use my version of Firefox over the one
>   defined by the main channel?  My guess is to change the name of my
>   package to something like "my-firefox".  But it would be nicer if I
>   don't have to do so.  Perhaps there's a way to force channel
>   preference?

Channels can only extend, not override the default Guix channel (the
world would be a bit of a mess if it did).  So the easiest path is to
use a different name; alternatively for graph rewriting you could use
the various APIs to effect package transformations.

>   I know that we can specify version like "firefox@99" but what about
>   outputs?  Is it possible to do (list "firefox:out" "firefox:debug").?

I know nothing about GWL, but if it uses the Guix-provided
specification->package procedure, it would work that way yes.n

I hope that helps,

Cheers!

Maxim

Re: Questions regarding substitutes with debug output

[Olivier Dion via]

On Fri, 22 Apr 2022, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>> I require debugging symbols of multiple packages.  Some of them
>> already have a debug output from the main guix channel, but others
>> don't.  So I had in mind to make a channel and publish the
>> substitutes with the debug outputs.
>
> In general, if you need debug symbols for something, I think it's OK
> to contribute adding them to Guix proper (in a debug output), unless
> they make like >= 1 GiB (as I can imagine may be the case for IceCat).

Okay I will try.  I'm not sure how easy it is so add debug symbols to some
packages e.g. git, nginx.  In any case I need them, so I will contribute
back to the main channel if it works.

>>   How could I make sure that Guix use my version of Firefox over the one
>>   defined by the main channel?  My guess is to change the name of my
>>   package to something like "my-firefox".  But it would be nicer if I
>>   don't have to do so.  Perhaps there's a way to force channel
>>   preference?
>
> Channels can only extend, not override the default Guix channel (the
> world would be a bit of a mess if it did).  So the easiest path is to
> use a different name; alternatively for graph rewriting you could use
> the various APIs to effect package transformations.

Would be nice to have some way to specify channel in a package
specification.  I don't think that it would break things if we
considerer channels as namespaces, i.e. different graph.  A
specification like:

  {channel}package@version:output

would be useful.  For now I will just rename them to "my/package".

Regards,
old

-- 
Olivier Dion
oldiob.dev

Re: Questions regarding substitutes with debug output

[Maxim Cournoyer]

Hello,

Olivier Dion <olivier.dion@polymtl.ca> writes:

> On Fri, 22 Apr 2022, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>>> I require debugging symbols of multiple packages.  Some of them
>>> already have a debug output from the main guix channel, but others
>>> don't.  So I had in mind to make a channel and publish the
>>> substitutes with the debug outputs.
>>
>> In general, if you need debug symbols for something, I think it's OK
>> to contribute adding them to Guix proper (in a debug output), unless
>> they make like >= 1 GiB (as I can imagine may be the case for IceCat).
>
> Okay I will try.  I'm not sure how easy it is so add debug symbols to some
> packages e.g. git, nginx.  In any case I need them, so I will contribute
> back to the main channel if it works.

When the package uses the gnu-build-system, it is usually just adding a
"debug" outputs and the copying of the stripped symbols to this output
is taken care of automatically.  Some packages use their own flags or a
release build without symbols in which case just adding "debug" alone
won't be enough.

>>>   How could I make sure that Guix use my version of Firefox over the one
>>>   defined by the main channel?  My guess is to change the name of my
>>>   package to something like "my-firefox".  But it would be nicer if I
>>>   don't have to do so.  Perhaps there's a way to force channel
>>>   preference?
>>
>> Channels can only extend, not override the default Guix channel (the
>> world would be a bit of a mess if it did).  So the easiest path is to
>> use a different name; alternatively for graph rewriting you could use
>> the various APIs to effect package transformations.
>
> Would be nice to have some way to specify channel in a package
> specification.  I don't think that it would break things if we
> considerer channels as namespaces, i.e. different graph.  A
> specification like:
>
>   {channel}package@version:output
>
> would be useful.  For now I will just rename them to "my/package".

That could be neat, yes.  I wonder how easy/difficult its implementation
would be.

Thanks,

Maxim

Re: Questions regarding substitutes with debug output

[Olivier Dion via]

On Sat, 23 Apr 2022, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

> When the package uses the gnu-build-system, it is usually just adding a
> "debug" outputs and the copying of the stripped symbols to this output
> is taken care of automatically.  Some packages use their own flags or a
> release build without symbols in which case just adding "debug" alone
> won't be enough.

Right.  My guess is that project using autotools are handled
automagically.  Although, I don't know of any configuration option for
splitting debug symbols from binaries in autotools.  For other build
systems, it's a case by case study.

-- 
Olivier Dion
oldiob.dev

Re: Questions regarding substitutes with debug output

[Maxim Cournoyer]

Hi,

Olivier Dion <olivier.dion@polymtl.ca> writes:

> On Sat, 23 Apr 2022, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>
>> When the package uses the gnu-build-system, it is usually just adding a
>> "debug" outputs and the copying of the stripped symbols to this output
>> is taken care of automatically.  Some packages use their own flags or a
>> release build without symbols in which case just adding "debug" alone
>> won't be enough.
>
> Right.  My guess is that project using autotools are handled
> automagically.  Although, I don't know of any configuration option for
> splitting debug symbols from binaries in autotools.

If you are curious, it's defined in the 'strip' phase in (guix build
gnu-build-system).  By default Autoconf causes the builds to include the
debug symbols (-g).  In the usual case (no "debug" output defined on a
Guix package), these debug symbols are simply stripped (using 'strip',
part of GNU binutils) and discarded.  When the "debug" output exists,
the debug symbols are instead copied to the debug output.

It should also work with the other C/C++ build systems such as
meson-build-system or cmake-build-system so long as they generate debug
symbols by default (which they should).

I hope that helps,

Maxim

Re: Questions regarding substitutes with debug output

[zimoun]

Hi,

On Fri, 22 Apr 2022 at 10:29, Olivier Dion via <help-guix@gnu.org> wrote:
> On Fri, 22 Apr 2022, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:

>> Channels can only extend, not override the default Guix channel (the
>> world would be a bit of a mess if it did).  So the easiest path is to
>> use a different name; alternatively for graph rewriting you could use
>> the various APIs to effect package transformations.
>
> Would be nice to have some way to specify channel in a package
> specification.  I don't think that it would break things if we
> considerer channels as namespaces, i.e. different graph.  A
> specification like:
>
>   {channel}package@version:output
>
> would be useful.  For now I will just rename them to "my/package".

What do you mean by «different graph»?  From my understanding, the
additions of channels makes just the graph bigger: extending the initial
(upstream) graph with more nodes (channel). :-)

IIUC, the question is how to refer to these nodes, and from my
understanding, we use basically two ways:

 1. by symbol; and thanks to Guile modules, this way provides
 namespaces, somehow.

 2. by metadata (name, version, output); and here I am not convinced it
 is doable to have a namespace but maybe mimic it.


Therefore, since your question is rooted from GWL:

        I need to specify the package programmatically as a string in
        Guile.  More specifically in the process packages field of Guix
        Workflow Language.

maybe GWL could also accept a symbol instead of a name string.  Well, I
have not used GWL since many months and I do not remember but I think it
is doable.  Ricardo?


Last, back to the feature you would like – be able to write:

    (specifications->manifest (list "foo"))

and select "foo" from your channel instead from upstream, right?  Or a
way to specify the channel using the symbol from the name field in
<channel>, right?

Aside the syntax of the string – why not the one your are proposing –
and so adapt ’package-name->name+version’ (or similar), the function
’find-best-packages-by-name’ requires some improvements.

What I am missing is the mapping from channel name to package.  Well,
Guix does not track this information at pull time.  And I miss how to
implement such mapping.

Therefore, assuming this mapping, the package cache (%package-cache-file)
could be updated to also track such map.  Such feature would also
simplify when searching; especially for variants (same as upstream but
other arguments).


WDYT?


Cheers,
simon

Re: Questions regarding substitutes with debug output

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> Therefore, since your question is rooted from GWL:
>
>         I need to specify the package programmatically as a string in
>         Guile.  More specifically in the process packages field of Guix
>         Workflow Language.
>
> maybe GWL could also accept a symbol instead of a name string.  Well, I
> have not used GWL since many months and I do not remember but I think it
> is doable.  Ricardo?

GWL accepts specifications or package values, so you could use (@ (my
packages foo) gcc) to prefer that over some other package with the same
name.

-- 
Ricardo

Re: Questions regarding substitutes with debug output

[Olivier Dion via]

On Thu, 28 Apr 2022, zimoun <zimon.toutoune@gmail.com> wrote:
> Hi,
>
> On Fri, 22 Apr 2022 at 10:29, Olivier Dion via <help-guix@gnu.org> wrote:
>> On Fri, 22 Apr 2022, Maxim Cournoyer <maxim.cournoyer@gmail.com> wrote:
>
>>> Channels can only extend, not override the default Guix channel (the
>>> world would be a bit of a mess if it did).  So the easiest path is to
>>> use a different name; alternatively for graph rewriting you could use
>>> the various APIs to effect package transformations.
>>
>> Would be nice to have some way to specify channel in a package
>> specification.  I don't think that it would break things if we
>> considerer channels as namespaces, i.e. different graph.  A
>> specification like:
>>
>>   {channel}package@version:output
>>
>> would be useful.  For now I will just rename them to "my/package".
>
> What do you mean by «different graph»?  From my understanding, the
> additions of channels makes just the graph bigger: extending the initial
> (upstream) graph with more nodes (channel). :-)

Right it's not a new graph per say since you're not introducing the full
dependencies themself.  You're only pulling more nodes into the graph.
I guess the correct term would be a "view" of the graph.

My point being it would be cool if we can specify a package with a
preference if two packages have the same name.  It could be the channel
name, but it could also be some metadata provided by Guix.  So really,
one could build a specific query (specification) until only a single
package is left from the setfff of matches.

Could be something like:
"guile@3.0.8:debug;channel=my-channel;some-metadata=foo"

I don't know, I'm making syntax up.  But I think it would be useful in
cases outside of mine.  I'm thinking about root filesystems for embedded
systems where you might want to use a variant of some package.

> IIUC, the question is how to refer to these nodes, and from my
> understanding, we use basically two ways:
>
>  1. by symbol; and thanks to Guile modules, this way provides
>  namespaces, somehow.
>  
>  2. by metadata (name, version, output); and here I am not convinced it
>  is doable to have a namespace but maybe mimic it.
>
> Therefore, since your question is rooted from GWL:
>
>         I need to specify the package programmatically as a string in
>         Guile.  More specifically in the process packages field of Guix
>         Workflow Language.
>
> maybe GWL could also accept a symbol instead of a name string.  Well, I
> have not used GWL since many months and I do not remember but I think it
> is doable.  Ricardo?

In my case, I prefer to avoid using package object directly.  As
mentioned in GWL' manual, the version of Guix running GWL and the
version of Guix used by GWL (the inferior) might not be the same.  Thus,
it is not okay for reproducibility in the long term.  In my case, I use
`guix time-machine --channels` as the inferior.

> Last, back to the feature you would like – be able to write:
>
>     (specifications->manifest (list "foo"))
>
> and select "foo" from your channel instead from upstream, right?  Or a
> way to specify the channel using the symbol from the name field in
> <channel>, right?

Correct.  In fact as I mentioned above, I think having a way to select a
package variant -- e.g. if you've introduced an existing package in your
channel -- is a more generic way of describing it.  Selection of channel
being only a type of metadata that can be used for selection of the
variant.  More metadata could be added in the future as well.

> Aside the syntax of the string – why not the one your are proposing –
> and so adapt ’package-name->name+version’ (or similar), the function
> ’find-best-packages-by-name’ requires some improvements.
>
> What I am missing is the mapping from channel name to package.  Well,
> Guix does not track this information at pull time.  And I miss how to
> implement such mapping.

I could not say.  I don't know much about the internal of Guix :-/.

> Therefore, assuming this mapping, the package cache
> (%package-cache-file) could be updated to also track such map.  Such
> feature would also simplify when searching; especially for variants
> (same as upstream but other arguments).

Yes that would be the idea!

Regards,
old

-- 
Olivier Dion
oldiob.dev

Re: Questions regarding substitutes with debug output

[Ricardo Wurmus]

Olivier Dion <olivier.dion@polymtl.ca> writes:

>> Therefore, since your question is rooted from GWL:
>>
>>         I need to specify the package programmatically as a string in
>>         Guile.  More specifically in the process packages field of Guix
>>         Workflow Language.
>>
>> maybe GWL could also accept a symbol instead of a name string.  Well, I
>> have not used GWL since many months and I do not remember but I think it
>> is doable.  Ricardo?
>
> In my case, I prefer to avoid using package object directly.  As
> mentioned in GWL' manual, the version of Guix running GWL and the
> version of Guix used by GWL (the inferior) might not be the same.  Thus,
> it is not okay for reproducibility in the long term.  In my case, I use
> `guix time-machine --channels` as the inferior.

The process itself could specify channels to use for its environment.
Then it would be self-contained and reproducible without having to rely
on time-machine.

-- 
Ricardo

Re: Questions regarding substitutes with debug output

[Olivier Dion via]

On Thu, 28 Apr 2022, Olivier Dion <olivier.dion@polymtl.ca> wrote:
> setfff

Oops :-O.  Tell me you're using emacs without telling me you're using
emacs.

-- 
Olivier Dion
oldiob.dev

Re: Questions regarding substitutes with debug output

[Olivier Dion via]

On Thu, 28 Apr 2022, Ricardo Wurmus <rekado@elephly.net> wrote:
> Olivier Dion <olivier.dion@polymtl.ca> writes:
>
>> In my case, I prefer to avoid using package object directly.  As
>> mentioned in GWL' manual, the version of Guix running GWL and the
>> version of Guix used by GWL (the inferior) might not be the same.  Thus,
>> it is not okay for reproducibility in the long term.  In my case, I use
>> `guix time-machine --channels` as the inferior.
>
> The process itself could specify channels to use for its environment.
> Then it would be self-contained and reproducible without having to rely
> on time-machine.

Is this already supported in GWL?  I'm not sure if I understand how you
would do this.  How would it be different from using time-machine?

-- 
Olivier Dion
oldiob.dev

Re: Questions regarding substitutes with debug output

[Ricardo Wurmus]

Olivier Dion <olivier.dion@polymtl.ca> writes:

> On Thu, 28 Apr 2022, Ricardo Wurmus <rekado@elephly.net> wrote:
>> Olivier Dion <olivier.dion@polymtl.ca> writes:
>>
>>> In my case, I prefer to avoid using package object directly.  As
>>> mentioned in GWL' manual, the version of Guix running GWL and the
>>> version of Guix used by GWL (the inferior) might not be the same.  Thus,
>>> it is not okay for reproducibility in the long term.  In my case, I use
>>> `guix time-machine --channels` as the inferior.
>>
>> The process itself could specify channels to use for its environment.
>> Then it would be self-contained and reproducible without having to rely
>> on time-machine.
>
> Is this already supported in GWL?  I'm not sure if I understand how you
> would do this.  How would it be different from using time-machine?

It is not yet supported in GWL, but it would only be a small change.
The GWL has support for inferiors but currently only uses the “invoking”
Guix for packages.

We could use a different set of channels per process definition.  Each
process runs in its own environment; it doesn’t have to be all done with
the same version of Guix.

Whether or not to support this is merely a question of how easily
workflows and processes can be reused or integrated in other workflows.

-- 
Ricardo

Re: Questions regarding substitutes with debug output

[zimoun]

Hi Olivier,

On Thu, 28 Apr 2022 at 10:11, Olivier Dion via <help-guix@gnu.org> wrote:
> On Thu, 28 Apr 2022, zimoun <zimon.toutoune@gmail.com> wrote:

> In my case, I prefer to avoid using package object directly.  As
> mentioned in GWL' manual, the version of Guix running GWL and the
> version of Guix used by GWL (the inferior) might not be the same.  Thus,
> it is not okay for reproducibility in the long term.  In my case, I use
> `guix time-machine --channels` as the inferior.

You wrote:

         I need to specify the package programmatically as a string in
         Guile.  More specifically in the process packages field of Guix
         Workflow Language.

so I do not understand why it would be an issue to deal with the package
instead of the specification string.  Because your need seems «to
specify the package programmatically». :-)


>> Last, back to the feature you would like – be able to write:
>>
>>     (specifications->manifest (list "foo"))
>>
>> and select "foo" from your channel instead from upstream, right?  Or a
>> way to specify the channel using the symbol from the name field in
>> <channel>, right?
>
> Correct.  In fact as I mentioned above, I think having a way to select a
> package variant -- e.g. if you've introduced an existing package in your
> channel -- is a more generic way of describing it.  Selection of channel
> being only a type of metadata that can be used for selection of the
> variant.  More metadata could be added in the future as well.

Well, I think you are “overengineering“. :-) I am lacking of imagination
about a channel containing several packages with the exact same name,
version, output but applying a variation to ’arguments’ or ’inputs’.

About ’arguments’, you need a short key as metadata, and the simplest is
to put this key directly in the name or version or output.

About ’inputs’, it remembers me «parameterized package», e.g., see [1].

1: <https://yhetil.org/guix/87eeku8trb.fsf@gnu.org>.


>> Aside the syntax of the string – why not the one your are proposing –
>> and so adapt ’package-name->name+version’ (or similar), the function
>> ’find-best-packages-by-name’ requires some improvements.


Rethinking about the feature request, what I would do:

 1. channel with variants where each variant contains a ’properties’ (it
 is an association list describing whatever you want and kept as-is)

 2. write a manifest.scm where I would filter by name and ’properties’;
 probably using ’fold-packages’.

Well, #2 means write by hand my own version ’find-best-packages-by-name’.

Maybe the API could have a function returning a list of packages from
its arguments name, version, output *and* a ’properties’ specification.


>> What I am missing is the mapping from channel name to package.  Well,
>> Guix does not track this information at pull time.  And I miss how to
>> implement such mapping.

Well, this mapping could be included as another ’properties’.  When
building the cache (~/.config/guix/current/lib/package.cache) with
’generate-package-cache’, the cache already contains the module name…

--8<---------------cut here---------------start------------->8---
  (define expand-cache
    (match-lambda*
      (((module symbol variable) (result . seen))
       (let ((package (variable-ref variable)))
         (if (or (vhash-assq package seen)
                 (hidden-package? package))
             (cons result seen)
             (cons (cons `#(,(package-name package)
                            ,(package-version package)
                            ,(module-name module)
                            ,symbol
                            ,(package-outputs package)
                            ,(->bool (supported-package? package))
                            ,(->bool (package-superseded package))
                            ,@(let ((loc (package-location package)))
                                (if loc
                                    `(,(location-file loc)
                                      ,(location-line loc)
                                      ,(location-column loc))
                                    '(#f #f #f))))
                         result)
                   (vhash-consq package #t seen)))))))
--8<---------------cut here---------------end--------------->8---

…but then it is not exploited by ’specifications->manifest’ and co, IIUC.


All in all, maybe it is time to resume: :-)

    https://issues.guix.gnu.org/39258

Cheers,
simon

Re: Questions regarding substitutes with debug output

[Olivier Dion via]

On Fri, 29 Apr 2022, zimoun <zimon.toutoune@gmail.com> wrote:
> Hi Olivier,
>
> On Thu, 28 Apr 2022 at 10:11, Olivier Dion via <help-guix@gnu.org> wrote:
>> On Thu, 28 Apr 2022, zimoun <zimon.toutoune@gmail.com> wrote:
>
>> In my case, I prefer to avoid using package object directly.  As
>> mentioned in GWL' manual, the version of Guix running GWL and the
>> version of Guix used by GWL (the inferior) might not be the same.  Thus,
>> it is not okay for reproducibility in the long term.  In my case, I use
>> `guix time-machine --channels` as the inferior.
>
> You wrote:
>
>          I need to specify the package programmatically as a string in
>          Guile.  More specifically in the process packages field of Guix
>          Workflow Language.
>
> so I do not understand why it would be an issue to deal with the package
> instead of the specification string.  Because your need seems «to
> specify the package programmatically». :-)

Because importing the package using use-modules would yield the package
from the instance of Guix driving the workflow instead of using the
package of the inferior!  GWL will lookup for package in the inferior
context when the package is a specification.

Even if I were to import locally defined packages, their dependencies
would come from the Guix's main channel!

>>> Last, back to the feature you would like – be able to write:
>>>
>>>     (specifications->manifest (list "foo"))
>>>
>>> and select "foo" from your channel instead from upstream, right?  Or a
>>> way to specify the channel using the symbol from the name field in
>>> <channel>, right?
>>
>> Correct.  In fact as I mentioned above, I think having a way to select a
>> package variant -- e.g. if you've introduced an existing package in your
>> channel -- is a more generic way of describing it.  Selection of channel
>> being only a type of metadata that can be used for selection of the
>> variant.  More metadata could be added in the future as well.
>
> Well, I think you are “overengineering“. :-) I am lacking of imagination
> about a channel containing several packages with the exact same name,
> version, output but applying a variation to ’arguments’ or ’inputs’.

I have this idea because I've worked on Elbe in the past.  It's a root
filesystem generator for embedded system based on Debian.  Some users of
Elbe have different constraints on what can be installed depending on
the license, memory footprint, etc.  So being able to cherry-pick
packages based on some properties is an useful feature here.  Of course,
you could just define packages locally and ask Guix so use them as long
as they have unique name.  So you would prefix all of your packages with
something like "my/".

So now imagine you have a specification -- or something like a DSL --
for describing your full RFS.  You generate it and see that its total
size is too much for your SD card using default packages.  But you have
a channel with different variants of some packages.  You then apply a
preference filter like 'diet=1' instead of changing the specification or
the module imports.  And now you see that your RFS fits in your SD card.

But yes, I would also describe it to be over-engineered if the intent is
to only have channel preference when there's name collision.  But then
again, we never know what are the future use case of Guix.

>>> Aside the syntax of the string – why not the one your are proposing –
>>> and so adapt ’package-name->name+version’ (or similar), the function
>>> ’find-best-packages-by-name’ requires some improvements.
>
>
> Rethinking about the feature request, what I would do:
>
>  1. channel with variants where each variant contains a ’properties’ (it
>  is an association list describing whatever you want and kept as-is)

Yes so something along:

  (package
   ...
   (properties '((foo . bar)
                 (buz . fuz))))

>  2. write a manifest.scm where I would filter by name and ’properties’;
>  probably using ’fold-packages’.
>
> Well, #2 means write by hand my own version ’find-best-packages-by-name’.
>
> Maybe the API could have a function returning a list of packages from
> its arguments name, version, output *and* a ’properties’
> specification.

I concur.

>>> What I am missing is the mapping from channel name to package.  Well,
>>> Guix does not track this information at pull time.  And I miss how to
>>> implement such mapping.

Really this feature is almost not necessary with package's properties:

  (package
    ...
    (properties '((channel . "my-project"))))

of course this could be confusing because the actual channel and the
properties named 'channel' could have different name.  So I would instead do

  (package
    ...
    (properties '((from-my-project? . #t))))

and then

  "guile;from-my-project=#t"

or something similar.

> Well, this mapping could be included as another ’properties’.  When
> building the cache (~/.config/guix/current/lib/package.cache) with
> ’generate-package-cache’, the cache already contains the module name…
>
> --8<---------------cut here---------------start------------->8---
>   (define expand-cache
>     (match-lambda*
>       (((module symbol variable) (result . seen))
>        (let ((package (variable-ref variable)))
>          (if (or (vhash-assq package seen)
>                  (hidden-package? package))
>              (cons result seen)
>              (cons (cons `#(,(package-name package)
>                             ,(package-version package)
>                             ,(module-name module)
>                             ,symbol
>                             ,(package-outputs package)
>                             ,(->bool (supported-package? package))
>                             ,(->bool (package-superseded package))
>                             ,@(let ((loc (package-location package)))
>                                 (if loc
>                                     `(,(location-file loc)
>                                       ,(location-line loc)
>                                       ,(location-column loc))
>                                     '(#f #f #f))))
>                          result)
>                    (vhash-consq package #t seen)))))))
> --8<---------------cut here---------------end--------------->8---
>
> …but then it is not exploited by ’specifications->manifest’ and co,
> IIUC.

Interesting.  I'll look into that.

Regards,
old

-- 
Olivier Dion
oldiob.dev

Re: Questions regarding substitutes with debug output

[Ricardo Wurmus]

Olivier Dion <olivier.dion@polymtl.ca> writes:

> On Fri, 29 Apr 2022, zimoun <zimon.toutoune@gmail.com> wrote:
>> Hi Olivier,
>>
>> On Thu, 28 Apr 2022 at 10:11, Olivier Dion via <help-guix@gnu.org> wrote:
>>> On Thu, 28 Apr 2022, zimoun <zimon.toutoune@gmail.com> wrote:
>>
>>> In my case, I prefer to avoid using package object directly.  As
>>> mentioned in GWL' manual, the version of Guix running GWL and the
>>> version of Guix used by GWL (the inferior) might not be the same.  Thus,
>>> it is not okay for reproducibility in the long term.  In my case, I use
>>> `guix time-machine --channels` as the inferior.
>>
>> You wrote:
>>
>>          I need to specify the package programmatically as a string in
>>          Guile.  More specifically in the process packages field of Guix
>>          Workflow Language.
>>
>> so I do not understand why it would be an issue to deal with the package
>> instead of the specification string.  Because your need seems «to
>> specify the package programmatically». :-)
>
> Because importing the package using use-modules would yield the package
> from the instance of Guix driving the workflow instead of using the
> package of the inferior!  GWL will lookup for package in the inferior
> context when the package is a specification.
>
> Even if I were to import locally defined packages, their dependencies
> would come from the Guix's main channel!

What would you think about WITH-CHANNELS that would accept a list of
channels and evaluate any package reference in its body within an
inferior of those channels?

E.g.

(with-channels %my-channels
  (package
    (name "yay")
    …
    (inputs
      (list foo bar baz)) ;these are all looked up in %my-channels
    …
    (license gpl3+)))

-- 
Ricardo

Re: Questions regarding substitutes with debug output

[zimoun]

Hi,

On Fri, 29 Apr 2022 at 10:47, Olivier Dion via <help-guix@gnu.org> wrote:

> Because importing the package using use-modules would yield the package
> from the instance of Guix driving the workflow instead of using the
> package of the inferior!  GWL will lookup for package in the inferior
> context when the package is a specification.
>
> Even if I were to import locally defined packages, their dependencies
> would come from the Guix's main channel!

Hum, ok.  Well, I am not enough familiar with recent GWL.  However, I am
still missing a point since using “string” is just a key to refer to
package object.  Anyway.


> So now imagine you have a specification -- or something like a DSL --
> for describing your full RFS.  You generate it and see that its total
> size is too much for your SD card using default packages.  But you have
> a channel with different variants of some packages.  You then apply a
> preference filter like 'diet=1' instead of changing the specification or
> the module imports.  And now you see that your RFS fits in your SD card.

What you are describing is already possible, somehow.  The specification
(manifest) is done by a DSL.  This manifest.scm file can contain complex
filters.  Well, for instance, let install only the packages which use
the ’r-build-system’ with:

    guix package -m manifest.scm -p /tmp/all-r

where manifest.scm reads,

--8<---------------cut here---------------start------------->8---
(use-modules (guix packages)
             (gnu packages)
             (guix build-system r))

(packages->manifest
 (fold-packages (lambda (package result)
                  (if (eq? (package-build-system package) r-build-system)
                      (cons package result)
                      result))
                '()))
--8<---------------cut here---------------end--------------->8---


> But yes, I would also describe it to be over-engineered if the intent is
> to only have channel preference when there's name collision.  But then
> again, we never know what are the future use case of Guix.

I think it is a better design to have a rich Scheme DSL to easily select
(or transform) the packages you want than something like:

--8<---------------cut here---------------start------------->8---
(specifications->manifest
  (list
    "guile@3.0.8:debug;channel=my-channel;some-metadata=foo"))
--8<---------------cut here---------------end--------------->8---


And I agree that some “selectors” as channel are currently missing. :-)


> Really this feature is almost not necessary with package's properties:
>
>   (package
>     ...
>     (properties '((channel . "my-project"))))

Instead of the string "my-project", it appears to me better to have the
symbols of the channel name defined by the file
~/.config/guix/channels.scm, e.g., 

--8<---------------cut here---------------start------------->8---
(cons*
 (channel
  (name 'nonguix)
  (url "https://gitlab.com/nonguix/nonguix.git")
  (branch "master"))
 (channel               ; .guix.channel depends on channel:
  (name 'bimsb-nonfree) ; github.com/BIMSBbioinfo/guix-bimsb.git
  (url "https://github.com/BIMSBbioinfo/guix-bimsb-nonfree.git")
  (branch "master"))
 ;; (channel
 ;;  (name 'bimsb)
 ;;  (url "https://github.com/BIMSBbioinfo/guix-bimsb.git")
 ;;  (branch "master"))
 (channel
  (name 'past)
  (url "https://gitlab.inria.fr/guix-hpc/guix-past.git")
  (branch "master"))
  (channel
  (name 'genenet)
  (url "https://gitlab.com/genenetwork/guix-bioinformatics.git")
  (branch "master"))
 %default-channels)
--8<---------------cut here---------------end--------------->8---



Thanks for the discussion. :-)  Cool, a new feature to implement. ;-)


Cheers,
simon

Re: Questions regarding substitutes with debug output

[Olivier Dion via]

On Fri, 29 Apr 2022, Ricardo Wurmus <rekado@elephly.net> wrote:
>> Even if I were to import locally defined packages, their dependencies
>> would come from the Guix's main channel!
>
> What would you think about WITH-CHANNELS that would accept a list of
> channels and evaluate any package reference in its body within an
> inferior of those channels?
>
> E.g.
>
> (with-channels %my-channels
>   (package
>     (name "yay")
>     …
>     (inputs
>       (list foo bar baz)) ;these are all looked up in %my-channels
>     …
>     (license gpl3+)))

What about dependencies?  Says I have the following graph:

 G <- Guix's channel
 C <- My channel

 P@V <- My package at version V from C
 D@U <- Dependency of P at version U from G

The `guix time-machine` allows me to pin the commits of G and C and thus
pinning V and U.  Would that work the with WITH-CHANNELS ?  

-- 
Olivier Dion
oldiob.dev

Re: Questions regarding substitutes with debug output

[Ricardo Wurmus]

Olivier Dion <olivier.dion@polymtl.ca> writes:

> On Fri, 29 Apr 2022, Ricardo Wurmus <rekado@elephly.net> wrote:
>>> Even if I were to import locally defined packages, their dependencies
>>> would come from the Guix's main channel!
>>
>> What would you think about WITH-CHANNELS that would accept a list of
>> channels and evaluate any package reference in its body within an
>> inferior of those channels?
>>
>> E.g.
>>
>> (with-channels %my-channels
>>   (package
>>     (name "yay")
>>     …
>>     (inputs
>>       (list foo bar baz)) ;these are all looked up in %my-channels
>>     …
>>     (license gpl3+)))
>
> What about dependencies?  Says I have the following graph:
>
>  G <- Guix's channel
>  C <- My channel
>
>  P@V <- My package at version V from C
>  D@U <- Dependency of P at version U from G
>
> The `guix time-machine` allows me to pin the commits of G and C and thus
> pinning V and U.  Would that work the with WITH-CHANNELS ?  

Exactly the same way.  The first argument to WITH-CHANNELS would be
(cons C G) and thus P and D would be picked from the set of C and G.  C
and G can be specified with exact commits or branches, just like the
--channels option to “guix time-machine”.

We already have all the tools for WITH-CHANNELS — that’s (guix
inferior).  We just don’t have a convenient syntax to do this without a
lot of boilerplate.

-- 
Ricardo

Re: Questions regarding substitutes with debug output

[Olivier Dion via]

On Fri, 29 Apr 2022, Ricardo Wurmus <rekado@elephly.net> wrote:
> Exactly the same way.  The first argument to WITH-CHANNELS would be
> (cons C G) and thus P and D would be picked from the set of C and G.  C
> and G can be specified with exact commits or branches, just like the
> --channels option to “guix time-machine”.
>
> We already have all the tools for WITH-CHANNELS — that’s (guix
> inferior).  We just don’t have a convenient syntax to do this without a
> lot of boilerplate.

Right.  It would be great I think.  However, time-machine does the job
for now, at least for my case.  But it would definitely allow for better
channel granularity.  To continue!

-- 
Olivier Dion
oldiob.dev

Re: Questions regarding substitutes with debug output

[zimoun]

Hi,

On Fri, 29 Apr 2022 at 22:08, Ricardo Wurmus <rekado@elephly.net> wrote:

>>> (with-channels %my-channels
>>>   (package
>>>     (name "yay")
>>>     …
>>>     (inputs
>>>       (list foo bar baz)) ;these are all looked up in %my-channels

[...]

> We already have all the tools for WITH-CHANNELS — that’s (guix
> inferior).  We just don’t have a convenient syntax to do this without a
> lot of boilerplate.

Indeed.  Where %my-channels would refer to a channels.scm file, right?

However, as Olivier mentioned it, Guix commonly uses package name and
version to refer to concrete defined packages; where the channel origin
is not in the picture.

Therefore, we generally add a string to the package name or version if a
channel creates a variant for this very same package name and version.
Otherwise it becomes hard to find back such variant.

This hypothetical ’with-channel’ would ease to create new variants.

What seems still missing is a way to select these variants from the
channel.

I am proposing to somehow add a field to ’properties’.  WDYT?


Cheers,
simon

Re: Questions regarding substitutes with debug output

[Ricardo Wurmus]

zimoun <zimon.toutoune@gmail.com> writes:

> Hi,
>
> On Fri, 29 Apr 2022 at 22:08, Ricardo Wurmus <rekado@elephly.net> wrote:
>
>>>> (with-channels %my-channels
>>>>   (package
>>>>     (name "yay")
>>>>     …
>>>>     (inputs
>>>>       (list foo bar baz)) ;these are all looked up in %my-channels
>
> [...]
>
>> We already have all the tools for WITH-CHANNELS — that’s (guix
>> inferior).  We just don’t have a convenient syntax to do this without a
>> lot of boilerplate.
>
> Indeed.  Where %my-channels would refer to a channels.scm file, right?

A list of channels rather than a file.

> What seems still missing is a way to select these variants from the
> channel.
>
> I am proposing to somehow add a field to ’properties’.  WDYT?

Can we not infer this when adding package modules to the load path?
Doing this once per channel seems better than having channel maintainers
add this to every package.

-- 
Ricardo

Re: Questions regarding substitutes with debug output

[zimoun]

Hi,

On Mon, 09 May 2022 at 15:33, Ricardo Wurmus <rekado@elephly.net> wrote:

>> Indeed.  Where %my-channels would refer to a channels.scm file, right?
>
> A list of channels rather than a file.

Do yo mean

        (define %my-channels
          (read-this-channel-file "path/to/one-channels-file.scm"))

        (with-channels %my-channels
          (package …))

?


>> What seems still missing is a way to select these variants from the
>> channel.
>>
>> I am proposing to somehow add a field to ’properties’.  WDYT?
>
> Can we not infer this when adding package modules to the load path?

It depends on you walk the packages list.  I agree that it should be
done automatically at “guix pull” time (or at inferior time).

> Doing this once per channel seems better than having channel maintainers
> add this to every package.

I agree.


Cheers,
simon