Store channel specification in profile

Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 937 bytes --]

Hi,

After the long discussions we had on profile, guix environment,
manifest, multi-dimensional rollbacks, etc. I suggest the following fix:
let's store the "channel specifications" in the profile / environment,
similar to how we store the manifest in etc/manifest.

What I call "channel specification" is essentially the output of `guix
describe -f channels`.  Unless some official term already exists, I
propose to document this term in the manual.

I propose to store the channel spec under
"etc/channel-specification.scm".
Or shall we opt for the shorter "etc/channl-spec"?

One question arises though: channel specifications only make sense for
profiles generated with manifests.  What about ad-hoc profiles?  Shall
we still store the spec in this case?  I propose we don't, this could
also be a hacky way to check whether a profile is ad-hoc or not.

Thoughts?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[Konrad Hinsen]

Hi Pierre,

> One question arises though: channel specifications only make sense for
> profiles generated with manifests.

Not even for those, if the manifest uses inferior-packages. I'd go for
per-package channel specifications. They could be optimized (more
compact, more efficiently usable) by first storing a list of all
distinct channels and then a (package . channels) list.

Cheers,
  Konrad

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 141 bytes --]


In this case, how would you intend to use guix time-machine to reproduce
these profiles?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[Konrad Hinsen]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> In this case, how would you intend to use guix time-machine to reproduce
> these profiles?

"guix time-machine" and inferiors are different ways to access specific
Guix versions. "guix time-machine" simply runs a different Guix version.
You can then use it to access inferiors, if that versions supports them.
Mixing both is probably confusing for users, so I'd recommend not to do
it, but technically it is possible.

Assuming that we end up storing channel specifications for each package
in a manifest, we could then create a manifest file automatically that
re-generates the same profile using inferiors. No need for "guix
time-machine".

Cheers,
  Konrad.

Re: Store channel specification in profile

[zimoun]

Hi

Let add some related features. :-)

Some days ago, Konrad proposed [1] a new subcommad (yes another one!
;-) to deal with channels. Mimicking "git remote" management, if I
summarize correctly.

 guix channel init <directory-name>
 guix channel add <name> URL
 guix channel remove <name>
 guix channel describe <name>
 guix channel list

Superseding "guix describe". (or "guix describe" will be an alias to
"guix channel describe").


[1] https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00285.html


What do you think?


All the best,
simon

Re: Store channel specification in profile

[Ludovic Courtès]

Hi!

Konrad Hinsen <konrad.hinsen@fastmail.net> skribis:

>> One question arises though: channel specifications only make sense for
>> profiles generated with manifests.
>
> Not even for those, if the manifest uses inferior-packages. I'd go for
> per-package channel specifications. They could be optimized (more
> compact, more efficiently usable) by first storing a list of all
> distinct channels and then a (package . channels) list.

This is exactly what’s currently implemented if you look at
~/.guix-profile/manifest, under ‘provenance’.

Like zimoun writes, it would be nice to have some sort of a “describe”
command for a regular profile.  Actually maybe “guix describe -p”?

Actually ‘guix describe -p ~/.guix-profile’ works but doesn’t display
anything useful.  We could fix that by recognizing the kind of profile,
somehow.

Thoughts?

Ludo’.

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 707 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> This is exactly what’s currently implemented if you look at
> ~/.guix-profile/manifest, under ‘provenance’.

But does "provenance" tell about the channels?

> Like zimoun writes, it would be nice to have some sort of a “describe”
> command for a regular profile.  Actually maybe “guix describe -p”?
>
> Actually ‘guix describe -p ~/.guix-profile’ works but doesn’t display
> anything useful.  We could fix that by recognizing the kind of profile,
> somehow.

Seems like a good idea.  How do we define "anything useful" though?
The provenance of packages?  How would we format it?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[zimoun]

Hi Pierre,

On Mon, 9 Dec 2019 at 18:22, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
> Ludovic Courtès <ludo@gnu.org> writes:
>
> > This is exactly what’s currently implemented if you look at
> > ~/.guix-profile/manifest, under ‘provenance’.
>
> But does "provenance" tell about the channels?

Yes, I think so.

For example,  ~/.guix-profile/manifest looks like that:

--8<---------------cut here---------------start------------->8---
("diffoscope"
      "131"
      "out"
      "/gnu/store/h8zr4rxhvpikv9p07kdjkf2dsrja35wm-diffoscope-131"
      (propagated-inputs ())
      (search-paths ())
      (properties
        (provenance
          (repository
            (version 0)
            (url "https://git.savannah.gnu.org/git/guix.git")
            (branch "master")
            (commit
              "b5d4d5b9bcf267fddd02fcc14b88eac0bebf979f")))))
--8<---------------cut here---------------end--------------->8---



> > Like zimoun writes, it would be nice to have some sort of a “describe”
> > command for a regular profile.  Actually maybe “guix describe -p”?
> >
> > Actually ‘guix describe -p ~/.guix-profile’ works but doesn’t display
> > anything useful.  We could fix that by recognizing the kind of profile,
> > somehow.
>
> Seems like a good idea.  How do we define "anything useful" though?
> The provenance of packages?  How would we format it?

As I explained elsewhere, the file <your-profile>/manifest already
contains almost all the information we need (at least I think we need
;-)). But its format is not complaint with the other format (channels,
manifest). And you answered: it is plumbing! :-)
My point is: this plumbing manifest file should be more
"format-friendly" -- still being plumbing -- and easily compliant with
the --channel or --manifest option, IMHO.


Cheers,
simon

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 246 bytes --]

I've seen the "provenance" field mentioned a couple of times before, but
I can't see any "provenance" in my $PROFILE/manifest file.  Am I missing
something?

I install profiles with manifests.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[zimoun]

Hi Pierre,

On Thu, 12 Dec 2019 at 23:35, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
> I've seen the "provenance" field mentioned a couple of times before, but
> I can't see any "provenance" in my $PROFILE/manifest file.  Am I missing
> something?
>
> I install profiles with manifests.

You have right. It is a bug.


The manifest contains:

--8<---------------cut here---------------start------------->8---
(specifications->manifest
  '("emacs"
    "git"
    "guile"))
--8<---------------cut here---------------end--------------->8---


Let create 2 profiles: one using the --manifest option and the other
one by plain --install.

--8<---------------cut here---------------start------------->8---
guix package -m /tmp/manif.scm -p /tmp/kikoo-manif
guix package -i emacs git guile -p /tmp/kikoo-nomanif
--8<---------------cut here---------------end--------------->8---


And here we are...

--8<---------------cut here---------------start------------->8---
grep provenance /tmp/kikoo-manif/manifest

--8<---------------cut here---------------end--------------->8---

versus

--8<---------------cut here---------------start------------->8---
grep provenance /tmp/kikoo-nomanif/manifest
        (provenance
        (provenance
        (provenance
--8<---------------cut here---------------end--------------->8---


All the best,
simon


--8<---------------cut here---------------start------------->8---
guix describe
Generation 60   Dec 13 2019 12:52:55    (current)
  guix eb8aad6
    repository URL: https://git.savannah.gnu.org/git/guix.git
    branch: master
    commit: eb8aad6a23442cf7b23e0df88b89b4cd030dfbf5
--8<---------------cut here---------------end--------------->8---

Re: Store channel specification in profile

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Thu, 12 Dec 2019 at 23:35, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>>
>> I've seen the "provenance" field mentioned a couple of times before, but
>> I can't see any "provenance" in my $PROFILE/manifest file.  Am I missing
>> something?
>>
>> I install profiles with manifests.
>
> You have right. It is a bug.

D’oh!  Could you file it?

Thanks,
Ludo’.

Re: Store channel specification in profile

[zimoun]

Hi!

On Thu, 19 Dec 2019 at 17:12, Ludovic Courtès <ludo@gnu.org> wrote:

> > You have right. It is a bug.
>
> D’oh!  Could you file it?

Done in #38673.

https://debbugs.gnu.org/cgi/bugreport.cgi?bug=38673


Cheers,
simon

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1378 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

>> > Like zimoun writes, it would be nice to have some sort of a “describe”
>> > command for a regular profile.  Actually maybe “guix describe -p”?
>> >
>> > Actually ‘guix describe -p ~/.guix-profile’ works but doesn’t display
>> > anything useful.  We could fix that by recognizing the kind of profile,
>> > somehow.
>>
>> Seems like a good idea.  How do we define "anything useful" though?
>> The provenance of packages?  How would we format it?
>
> As I explained elsewhere, the file <your-profile>/manifest already
> contains almost all the information we need (at least I think we need
> ;-)). But its format is not complaint with the other format (channels,
> manifest). And you answered: it is plumbing! :-)
> My point is: this plumbing manifest file should be more
> "format-friendly" -- still being plumbing -- and easily compliant with
> the --channel or --manifest option, IMHO.

I agree.  But changing the format might break compatibility with existing
functions that rely on it.  In which places are manifests used?

Should we not change the format, should we add a command line option to
convert manifest to "manifest specifications" (those that work with
`-manifest')?  How do we include the provenance in manifest specifications?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[zimoun]

Hi Pierre,


On Mon, 6 Jan 2020 at 21:07, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
> zimoun <zimon.toutoune@gmail.com> writes:
>
> >> > Like zimoun writes, it would be nice to have some sort of a “describe”
> >> > command for a regular profile.  Actually maybe “guix describe -p”?
> >> >
> >> > Actually ‘guix describe -p ~/.guix-profile’ works but doesn’t display
> >> > anything useful.  We could fix that by recognizing the kind of profile,
> >> > somehow.
> >>
> >> Seems like a good idea.  How do we define "anything useful" though?
> >> The provenance of packages?  How would we format it?


You asked 3 questions. :-)

1.  How do we define "anything useful" though?
2. The provenance of packages?
3. How would we format it?

And the answer of these 3 questions is the file <your-profile>/manifest.


> > As I explained elsewhere, the file <your-profile>/manifest already
> > contains almost all the information we need (at least I think we need
> > ;-)). But its format is not complaint with the other format (channels,
> > manifest). And you answered: it is plumbing! :-)
> > My point is: this plumbing manifest file should be more
> > "format-friendly" -- still being plumbing -- and easily compliant with
> > the --channel or --manifest option, IMHO.
>
> I agree.  But changing the format might break compatibility with existing
> functions that rely on it.  In which places are manifests used?

IMHO, the improvement of ‘guix describe -p ~/.guix-profile’ or even a
kind of profile manager (e.g., a new "guix profile" subcommand doing
pieces of "guix package" and/or maybe "guix environment") should use
this file <your-profile>/manifest. Therefore, there 3 directions, the
2 as you said:

 a. keep the file <your-profile>/manifest as it is today; and add
options to convert it to regular manifests and channels.
 b. improve the format of this file to obtain consistent manifests.

and c. improve what the --manifest option accepts.


At the user level, the option a. breaks nothing. This file
<your-profile>/manifest is not known  -- I even do not know if it is
documented -- so the option b. might not break compatibility. About
functions that rely on it, I have not checked yet. And the option c.
needs to be discussed.


From my perspective, the option b. is a better path because it remove
layers and complexity.


> Should we not change the format, should we add a command line option to
> convert manifest to "manifest specifications" (those that work with
> `-manifest')?  How do we include the provenance in manifest specifications?

The provenance of what?

Basically an entry looks like:

--8<---------------cut here---------------start------------->8---
     ("diffoscope"
      "131"
      "out"
      "/gnu/store/h8zr4rxhvpikv9p07kdjkf2dsrja35wm-diffoscope-131"
      (propagated-inputs ())
      (search-paths ())
      (properties
        (provenance
          (repository
            (version 0)
            (url "https://git.savannah.gnu.org/git/guix.git")
            (branch "master")
            (commit
              "b5d4d5b9bcf267fddd02fcc14b88eac0bebf979f")))))
--8<---------------cut here---------------end--------------->8---



Cheers,
simon

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1973 bytes --]


>  a. keep the file <your-profile>/manifest as it is today; and add
> options to convert it to regular manifests and channels.
>  b. improve the format of this file to obtain consistent manifests.
>
> and c. improve what the --manifest option accepts.

Agreed.

> From my perspective, the option b. is a better path because it remove
> layers and complexity.

I thought you meant "do all 3 options".  How would option b. alone be enough?

>> Should we not change the format, should we add a command line option to
>> convert manifest to "manifest specifications" (those that work with
>> `-manifest')?  How do we include the provenance in manifest specifications?
>
> The provenance of what?

How do we include the "provenance" field from manifest into "manifest
specifications"?


> Basically an entry looks like:
>
> --8<---------------cut here---------------start------------->8---
>      ("diffoscope"
>       "131"
>       "out"
>       "/gnu/store/h8zr4rxhvpikv9p07kdjkf2dsrja35wm-diffoscope-131"
>       (propagated-inputs ())
>       (search-paths ())
>       (properties
>         (provenance
>           (repository
>             (version 0)
>             (url "https://git.savannah.gnu.org/git/guix.git")
>             (branch "master")
>             (commit
>               "b5d4d5b9bcf267fddd02fcc14b88eac0bebf979f")))))
> --8<---------------cut here---------------end--------------->8---

The above is not very human friendly.  Each entry should be something
like plist with default values, so that the user does not have to write
"out" every time.

What about this:

("diffoscope"
 :version "131"
 :output "doc" ; "out" is the default.
 :provenance (repository ...))

Questions:

- Do manifests really need the store path?  
- Same question about propagated-inputs.  Aren't they already encoded in
  the package definition?  Why repeating them here?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[zimoun]

Hi Pierre,

On Wed, 8 Jan 2020 at 16:17, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> >  a. keep the file <your-profile>/manifest as it is today; and add
> > options to convert it to regular manifests and channels.
> >  b. improve the format of this file to obtain consistent manifests.
> >
> > and c. improve what the --manifest option accepts.
>
> Agreed.
>
> > From my perspective, the option b. is a better path because it remove
> > layers and complexity.
>
> I thought you meant "do all 3 options".  How would option b. alone be enough?

I agree and I am on this wavelength. Maybe we had some noise in
previous emails. :-)


> >> Should we not change the format, should we add a command line option to
> >> convert manifest to "manifest specifications" (those that work with
> >> `-manifest')?  How do we include the provenance in manifest specifications?
> >
> > The provenance of what?
>
> How do we include the "provenance" field from manifest into "manifest
> specifications"?

I read "channel" instead of "provenance".



> > Basically an entry looks like:
> >
> > --8<---------------cut here---------------start------------->8---
> >      ("diffoscope"
> >       "131"
> >       "out"
> >       "/gnu/store/h8zr4rxhvpikv9p07kdjkf2dsrja35wm-diffoscope-131"
> >       (propagated-inputs ())
> >       (search-paths ())
> >       (properties
> >         (provenance
> >           (repository
> >             (version 0)
> >             (url "https://git.savannah.gnu.org/git/guix.git")
> >             (branch "master")
> >             (commit
> >               "b5d4d5b9bcf267fddd02fcc14b88eac0bebf979f")))))
> > --8<---------------cut here---------------end--------------->8---
>
> The above is not very human friendly.  Each entry should be something
> like plist with default values, so that the user does not have to write
> "out" every time.
>
> What about this:
>
> ("diffoscope"
>  :version "131"
>  :output "doc" ; "out" is the default.
>  :provenance (repository ...))
>
> Questions:
>
> - Do manifests really need the store path?

I do not see why neither.

Maybe in this scenario: you build a profile with --manifest=foo.scm,
then you send me this foo.scm manifest file and <profile>/manifest.
With these, I instantiate a profile with --manifest=foo.scm and I can
"challenge" against what you have by comparing my hashes to your
hashes.

> - Same question about propagated-inputs.  Aren't they already encoded in
>   the package definition?  Why repeating them here?

I agree.


All the best,
simon

Re: Store channel specification in profile

[Ludovic Courtès]

Hi,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

>> Basically an entry looks like:
>>
>> --8<---------------cut here---------------start------------->8---
>>      ("diffoscope"
>>       "131"
>>       "out"
>>       "/gnu/store/h8zr4rxhvpikv9p07kdjkf2dsrja35wm-diffoscope-131"
>>       (propagated-inputs ())
>>       (search-paths ())
>>       (properties
>>         (provenance
>>           (repository
>>             (version 0)
>>             (url "https://git.savannah.gnu.org/git/guix.git")
>>             (branch "master")
>>             (commit
>>               "b5d4d5b9bcf267fddd02fcc14b88eac0bebf979f")))))
>> --8<---------------cut here---------------end--------------->8---
>
> The above is not very human friendly.  Each entry should be something
> like plist with default values, so that the user does not have to write
> "out" every time.

Note that it was never designed to be human-friendly.  :-)  The Scheme
code passed to ‘--manifest’ is friendlier.

> Questions:
>
> - Do manifests really need the store path?  
> - Same question about propagated-inputs.  Aren't they already encoded in
>   the package definition?  Why repeating them here?

This ‘manifest’ file exists mostly for one purpose: to allow incremental
operations on a profile with ‘guix upgrade’, ‘guix install’, and so on.
If ‘--manifest’ were the only way to build a profile, this ‘manifest’
file would (almost) not be needed.  (Actually it’s also needed for
‘--list-installed’.)

Incremental operations need all this info so they can pull in the right
store items, set the right search paths, etc.  IOW, they need to work
around the fact that some of the <package> objects for their profile are
not available.

HTH!

Ludo’.

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1327 bytes --]

Hi,

>> Questions:
>>
>> - Do manifests really need the store path?  
>> - Same question about propagated-inputs.  Aren't they already encoded in
>>   the package definition?  Why repeating them here?
>
> This ‘manifest’ file exists mostly for one purpose: to allow incremental
> operations on a profile with ‘guix upgrade’, ‘guix install’, and so on.
> If ‘--manifest’ were the only way to build a profile, this ‘manifest’
> file would (almost) not be needed.  (Actually it’s also needed for
> ‘--list-installed’.)

Makes sense.

So what's the take-away of this thread?

1. Simon suggested to add options to convert the manifest to the
user-friendly specification file (i.e. something compatible with the
--manifest option).

What about this instead: systematically generate this "specification"
file in every profile?  This way no need for extra command line options,
the work is already done for the user.  "To reproduce a profile" would
boil down to passing around this specification file.

2. On December 2, Simon mentioned Ludo's suggestion (from
https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00285.html)
that we added a "guix channel" subcommand for channel management.

Shall we open 2 bugs for these?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[zimoun]

Hi Ludo,

On Sun, 12 Jan 2020 at 00:48, Ludovic Courtès <ludo@gnu.org> wrote:

> Note that it was never designed to be human-friendly.  :-)  The Scheme
> code passed to ‘--manifest’ is friendlier.

Why do not change a bit both <profile>/manifest and code accepted by
'--manifest' to have something consistent and human friendly?

(Even if successive transactions will produce an unavoidable mess to
<profile>/manifest.)

Well, I do not see why there is 2 different syntax/semantic; one for
<profile>/manifest and another one for the code accepted by
'--manifest'.
Except, it is some pieces of work to modify the actual code. :-)


Cheers,
simon

Re: Store channel specification in profile

[zimoun]

Hi Pierre,

On Mon, 13 Jan 2020 at 15:02, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> So what's the take-away of this thread?
>
> 1. Simon suggested to add options to convert the manifest to the
> user-friendly specification file (i.e. something compatible with the
> --manifest option).
>
> What about this instead: systematically generate this "specification"
> file in every profile?  This way no need for extra command line options,
> the work is already done for the user.  "To reproduce a profile" would
> boil down to passing around this specification file.

Sounds good to me. ;-)


> 2. On December 2, Simon mentioned Ludo's suggestion (from
> https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00285.html)
> that we added a "guix channel" subcommand for channel management.

You mean Konrad's suggestions. :-)


> Shall we open 2 bugs for these?

The UI is not clear to me. And before adding some new CLI, we need to
agree (reach consensus) on what it is expected. Because then, it will
be hard to change/remove this CLI.

For example, Konrad also suggested "guix profile" doing part of "guix package".
And this same name "guix profile" showed up again in the discussion
about "guix environment".
Maybe "guix channel" could supersede "guix describe".
etc.


All the best,
simon

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 244 bytes --]

If I understand correctly, it's because of the manifest files need
information like the store path and the propagated inputs, which are too
inconvenient for a user-facing "specification file."

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[zimoun]

On Mon, 13 Jan 2020 at 15:59, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
> If I understand correctly, it's because of the manifest files need
> information like the store path and the propagated inputs, which are too
> inconvenient for a user-facing "specification file."

Hum? I am not convinced yet. :-)

For example, the record <package> contains the field "outputs" but
some packages do not use it; e.g., "xmag".
Same for "native-inputs", etc.

To me, the aim is to have something compliant between
<profile>/manifest and --manifest. And compliant does not mean that
<profile>/manifest is the entry point for the user specifications.
What I find a bit odd is: today, --manifest accepts a DSL and Guix
outputs to <profile>/manifest another DSL. Both are restricted tiny
DSL. And from all the recent discussions about manifests and so on, I
find appealing to extend the DSL of --manifest and use a subset to
write <profile>/manifest. I do not know.


Cheers,
simon

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 803 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

> To me, the aim is to have something compliant between
> <profile>/manifest and --manifest. And compliant does not mean that
> <profile>/manifest is the entry point for the user specifications.
> What I find a bit odd is: today, --manifest accepts a DSL and Guix
> outputs to <profile>/manifest another DSL. Both are restricted tiny
> DSL. And from all the recent discussions about manifests and so on, I
> find appealing to extend the DSL of --manifest and use a subset to
> write <profile>/manifest. I do not know.

As I see it, <profile>/manifest or even a subset of it is not convenient
for users to write.  If we want to expose a convenient format, I can't
see a way around using a new file.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 2244 bytes --]

We've made some progress on the topic during the Guix Days.

First, let's recall some of the use cases:

1. Allow the user to reproduce a profile easily.
2. Keep a profile specification under version control with least effort.
3. Eaysily interface with inferiors (e.g. build a profile with pkg1 and pkg2 from
   two different Guix versions.)

   
Ideas that are not that good on second thought:

- Create a profile specification file from CLI options,
  like --export PROFILE or --convert MANIFEST.

  The problem is that those are extra steps that the user would have to run
  manually.  We can save those extra steps by dumping the specification file
  automatically into the profile.

The Plan©:

On every profile installation, we generate a "specifications.scm" file alongside
the internal "manifest".

Problems:

- There may be too many provenances, we need to add a CLI flag to
  ignore provenance.
  
- Too many provenances can be unusable (too slow to install, this happens when
  using imperative style).  Raise a warning, option to use latest Guix only.
  In all cases, we need an option to override all the provenance with a channel
  specification as per =guix describe -f channels=.

Proposed format for "specifications.scm": we can reuse
`specifications->manifest`.  Each entry is either or string, in which case it
acts as before, or a list, with the following self-explanatory elements:

--8<---------------cut here---------------start------------->8---
(specifications->manifest
 '(("my-package"
    #:outputs '("out")
    #:version "2.3"
    #:channel (channel
               (name 'guix)
               (branch "master")
               (url "https://git.savannah.gnu.org/git/guix.git")
               (commit
                "704719edade1368f798c9301f3a8197a0df5c930")))
   ("my-package2")
   "old-style-package"))
--8<---------------cut here---------------end--------------->8---

   
A somewhat unrelated propostion: To avoid further confusion between the internal
"manifest" and the user-facing "--manifest", we could rename the internal
manifest to $profile/internal-maifest.

Thoughts?
Green light for a patch?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[zimoun]

Hi Pierre,


On Thu, 30 Jan 2020 at 20:24, Pierre Neidhardt <mail@ambrevar.xyz> wrote:

> We've made some progress on the topic during the Guix Days.

Nice that you reported that fast. :-)


> Green light for a patch?

Sounds like a good plan.


All the best,
simon

Re: Store channel specification in profile

[Konrad Hinsen]

Hi Pierre,

> We've made some progress on the topic during the Guix Days.

Thanks for the summary!

> The Plan©:
>
> On every profile installation, we generate a "specifications.scm" file alongside
> the internal "manifest".

Sounds good.

> Problems:
>
> - There may be too many provenances, we need to add a CLI flag to
>   ignore provenance.

Too many in what sense?

I'd just print a warning if the specification refers to more than, say,
five inferiors. Users can then clean up their profile or live with the
mess. Cleaning up just means updating the profile. That would re-create
specifications.scm, in a better way than a special CLI flag because
specifications.scm is always the same as the current profile.

> Proposed format for "specifications.scm": we can reuse
> `specifications->manifest`.  Each entry is either or string, in which case it
> acts as before, or a list, with the following self-explanatory elements:

Looks good.

> A somewhat unrelated propostion: To avoid further confusion between the internal
> "manifest" and the user-facing "--manifest", we could rename the internal
> manifest to $profile/internal-maifest.

Fine, but not so critical. I doubt there are many users who ever look at
that file.

Cheers,
  Konrad.

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 858 bytes --]

Konrad Hinsen <konrad.hinsen@fastmail.net> writes:

>> - There may be too many provenances, we need to add a CLI flag to
>>   ignore provenance.
>
> Too many in what sense?
>
> I'd just print a warning if the specification refers to more than, say,
> five inferiors. Users can then clean up their profile or live with the
> mess.
>
> Cleaning up just means updating the profile.

You mean with

guix upgrade -p /path/to/profile?

> That would re-create
> specifications.scm, in a better way than a special CLI flag because
> specifications.scm is always the same as the current profile.

OK, but what about a system where you don't have the profile, only the
specification file?

I believe we still need a flag to override the channels, basically add
-C,--channels to `guix package`.

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1489 bytes --]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

> Konrad Hinsen <konrad.hinsen@fastmail.net> writes:
>
>>> - There may be too many provenances, we need to add a CLI flag to
>>>   ignore provenance.
>>
>> Too many in what sense?
>>
>> I'd just print a warning if the specification refers to more than, say,
>> five inferiors. Users can then clean up their profile or live with the
>> mess.
>>
>> Cleaning up just means updating the profile.
>
> You mean with
>
> guix upgrade -p /path/to/profile?
>
>> That would re-create
>> specifications.scm, in a better way than a special CLI flag because
>> specifications.scm is always the same as the current profile.
>
> OK, but what about a system where you don't have the profile, only the
> specification file?
>
> I believe we still need a flag to override the channels, basically add
> -C,--channels to `guix package`.

After a discussion with Zimoun, -C would break the guarantee that =guix
package= is using the current Guix, plus would overlap with =guix
time-machine=.

Instead, I suggest we use a --use-default-channels boolean flag and then we
could leverage guix time-machine to force a specific channel
specification on a manifest:

--8<---------------cut here---------------start------------->8---
guix time-machine --channels=foo.scm -- package -m specs.scm --use-default-channels
--8<---------------cut here---------------end--------------->8---

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[Konrad Hinsen]

Pierre Neidhardt <mail@ambrevar.xyz> writes:

>>> Cleaning up just means updating the profile.
>>
>> You mean with
>>
>> guix upgrade -p /path/to/profile?

Right.

>> OK, but what about a system where you don't have the profile, only the
>> specification file?

OK, that's at the other end, for the user of a specification file. That
must indeed be handled by some option to "guix package".

> Instead, I suggest we use a --use-default-channels boolean flag and then we
> could leverage guix time-machine to force a specific channel
> specification on a manifest:
>
> --8<---------------cut here---------------start------------->8---
> guix time-machine --channels=foo.scm -- package -m specs.scm --use-default-channels
> --8<---------------cut here---------------end--------------->8---

LGTM!

Cheers,
  Konrad.

Re: Store channel specification in profile

[Ludovic Courtès]

Hello!

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Ideas that are not that good on second thought:
>
> - Create a profile specification file from CLI options,
>   like --export PROFILE or --convert MANIFEST.
>
>   The problem is that those are extra steps that the user would have to run
>   manually.  We can save those extra steps by dumping the specification file
>   automatically into the profile.
>
> The Plan©:
>
> On every profile installation, we generate a "specifications.scm" file alongside
> the internal "manifest".

One thing to keep in mind, though, is that if the ‘specifications.scm’
is part of the profile, it must be future-proof.  That is, the APIs it
uses must essentially be guaranteed to remain forever.  That’s a very
strong constraint.

In contrast, versioned data formats like the famous ‘manifest’ file
don’t have this problem at all, but they’re less directly usable from
the CLI.

> Problems:
>
> - There may be too many provenances, we need to add a CLI flag to
>   ignore provenance.

Like Konrad wrote, just write if there are too many of them.

> Proposed format for "specifications.scm": we can reuse
> `specifications->manifest`.  Each entry is either or string, in which case it
> acts as before, or a list, with the following self-explanatory elements:
>
> (specifications->manifest
>  '(("my-package"
>     #:outputs '("out")
>     #:version "2.3"
>     #:channel (channel
>                (name 'guix)
>                (branch "master")
>                (url "https://git.savannah.gnu.org/git/guix.git")
>                (commit
>                 "704719edade1368f798c9301f3a8197a0df5c930")))
>    ("my-package2")
>    "old-style-package"))

As a rule of thumb, I think ‘specifications->manifest’ must remain what
it is: it should take a specification as is currently defined and return
a manifest.  I think that if we need a new concept, we should not
overload an existing one.

We also need to distinguish between APIs, which should use first-class
objects (<channel>, etc.), and data formats, which are plain sexps.
(The above example is a mixture of both and in fact looks very similar
to what’s already in the ‘manifest’ file.)  But then again, that means
relying on a larger chunk of the API.

So, all in all, I think I’d rather see it implemented as ‘guix package
--export’ or similar.  The generated Scheme file could use all the parts
of the current API (and we could adjust the generator as we change the
API).  So it could generate something similar to the example in the
manual (info "(guix) Inferiors"):

--8<---------------cut here---------------start------------->8---
(use-modules (guix inferior) (guix channels)
             (srfi srfi-1))   ;for 'first'

(define channels
  ;; This is the old revision from which we want to
  ;; extract guile-json.
  (list (channel
         (name 'guix)
         (url "https://git.savannah.gnu.org/git/guix.git")
         (commit
          "65956ad3526ba09e1f7a40722c96c6ef7c0936fe"))))

(define inferior
  ;; An inferior representing the above revision.
  (inferior-for-channels channels))

;; Now create a manifest with the current "guile" package
;; and the old "guile-json" package.
(packages->manifest
 (list (first (lookup-inferior-packages inferior "guile-json"))
       (specification->package "guile")))
--8<---------------cut here---------------end--------------->8---

There could also be an option to generate a “symbolic” manifest: one
that would install packages of the same name, but not resorting to
inferiors.  The result would not be a faithful translation of what’s
actually in the store, but an approximation thereof that could be useful
for people transitioning.  In fact it would be easier to start by
prototyping this before going further.

As far as faithfulness is concerned, we should also keep in mind that we
don’t always have all the information needed to reconstruct what’s in a
profile.  For example, we lack information about the package
transformation options that were used (if any), and we lack information
about arbitrary user code that might have been used to generate manifest
entries.

Thus, as we design this, I think we must keep in mind that the result is
necessarily an approximation.

> A somewhat unrelated propostion: To avoid further confusion between the internal
> "manifest" and the user-facing "--manifest", we could rename the internal
> manifest to $profile/internal-maifest.

Renaming ‘manifest’ is not really an option because, as we discussed,
it’s a contract over time: today’s Guix can interact with a profile
created 5 years ago just fine.

Thanks for sharing your thoughts!

Ludo’.

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 4505 bytes --]

Hi!

Ludovic Courtès <ludo@gnu.org> writes:

> Hello!
>
>> The Plan©:
>>
>> On every profile installation, we generate a "specifications.scm" file alongside
>> the internal "manifest".
>
> One thing to keep in mind, though, is that if the ‘specifications.scm’
> is part of the profile, it must be future-proof.  That is, the APIs it
> uses must essentially be guaranteed to remain forever.  That’s a very
> strong constraint.

I think that's OK.  In the example format I suggested, we use keys, which
make it easy to extend the format later on, if need be.

> In contrast, versioned data formats like the famous ‘manifest’ file
> don’t have this problem at all, but they’re less directly usable from
> the CLI.

We can version the specifications.scm as well.  Can you explain what the
problem is, I think I'm missing the point.

>> Proposed format for "specifications.scm": we can reuse
>> `specifications->manifest`.  Each entry is either or string, in which case it
>> acts as before, or a list, with the following self-explanatory elements:
>>
>> (specifications->manifest
>>  '(("my-package"
>>     #:outputs '("out")
>>     #:version "2.3"
>>     #:channel (channel
>>                (name 'guix)
>>                (branch "master")
>>                (url "https://git.savannah.gnu.org/git/guix.git")
>>                (commit
>>                 "704719edade1368f798c9301f3a8197a0df5c930")))
>>    ("my-package2")
>>    "old-style-package"))
>
> As a rule of thumb, I think ‘specifications->manifest’ must remain what
> it is: it should take a specification as is currently defined and return
> a manifest.  I think that if we need a new concept, we should not
> overload an existing one.

In my understanding it's not so much of a new concept as an extension of
the existing one.  But defining a new function is not a problem at all
either.

What about

  specifications->manifest*

?

> We also need to distinguish between APIs, which should use first-class
> objects (<channel>, etc.), and data formats, which are plain sexps.
> (The above example is a mixture of both and in fact looks very similar
> to what’s already in the ‘manifest’ file.)  But then again, that means
> relying on a larger chunk of the API.

I'm not sure I understood what you meant here.  Which APIs?

> So, all in all, I think I’d rather see it implemented as ‘guix package
> --export’ or similar.

I think I didn't understand why you'd prefer to have a command instead
of systematically generating the file inside the profile.  Or maybe we
could generate the file somewhere else if that's the problem?

In my opinion, the `--export` approach would be cumbersome because it
means that for users who want to save the specifications file, they'd
need to systematically call it on every profile-modifying command, e.g.

  guix package -m manifest.scm && guix package --export

> There could also be an option to generate a “symbolic” manifest: one
> that would install packages of the same name, but not resorting to
> inferiors.

I think this can be implemented with the --use-default-channels option I
suggested before.

> As far as faithfulness is concerned, we should also keep in mind that we
> don’t always have all the information needed to reconstruct what’s in a
> profile.  For example, we lack information about the package
> transformation options that were used (if any),

Like `--with-input'?
I didn't think of this.  Can't we extend the format then to include all
transformation options, with future-proof provisions?

Example:

--8<---------------cut here---------------start------------->8---
(specifications->manifest
 '(("my-package"
    #:outputs '("out")
    #:version "2.3"
    #:transformation '((input PACKAGE REPLACEMENT)
                       (source SOURCE))
    #:channel (channel
               (name 'guix)
               (branch "master")
               (url "https://git.savannah.gnu.org/git/guix.git")
               (commit
                "704719edade1368f798c9301f3a8197a0df5c930")))
--8<---------------cut here---------------end--------------->8---

> and we lack information about arbitrary user code that might have been
> used to generate manifest entries.

Is this a problem?  Even if we only store generated manifest entries,
this should be enough to reproduce the profile, no?

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[Ludovic Courtès]

Howdy!

Turns out I just watched your talk (since I couldn’t be there at the
time!).  I was surprised you didn’t explicitly mention ‘guix search’ and
the shortcomings you’d like to address (well, not directly), but I liked
the perspectives!

  https://video.fosdem.org/2020/AW1.125/gnuguixpackagemanager.webm

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

>> One thing to keep in mind, though, is that if the ‘specifications.scm’
>> is part of the profile, it must be future-proof.  That is, the APIs it
>> uses must essentially be guaranteed to remain forever.  That’s a very
>> strong constraint.
>
> I think that's OK.  In the example format I suggested, we use keys, which
> make it easy to extend the format later on, if need be.

I feel there a Common Lisp vs. Scheme cultural clash here.  ;-)

The way I see it, a public procedure should take clearly defined,
structured objects, and of only a single type (that’s a pattern mostly
followed throughout Guix).

So, if I were to implement the proposed ‘specifications->manifest*’, it
would take a record to describe the package/version/channel.  But then
that record wouldn’t be different from a <manifest-entry> referencing an
<inferior-package>.  IOW: the API already exists, mostly.

I would not use sexps (like the one you propose) as substitutes for
records in an API.  I know that’s the traditional Lisp way to do it, but
it’s IMO too clumsy, error-prone (unless you use a pattern matcher, but
even then it’s not enough), inefficient, etc.

I would, however, use sexps as a serialization format.  Compared to an
API, an object serialized to an sexp has the advantage that we can write
code to handle changes in the serialization format, so it’s future-proof
if we get it right.  But then: we’re back to ‘manifest’.  :-)

I hope this is a bit clearer, but I realize it’s tricky to discuss such
things!

>> In contrast, versioned data formats like the famous ‘manifest’ file
>> don’t have this problem at all, but they’re less directly usable from
>> the CLI.
>
> We can version the specifications.scm as well.  Can you explain what the
> problem is, I think I'm missing the point.

‘manifest’ looks like this:

  (manifest
    (version 3)
    …)

We have an explicit ‘read-manifest’ procedure that can handle version 3,
but also prior versions, and this is all transparent.

You cannot do that with code.  Code is just evaluated, and if it’s
incompatible, if fails in some unspecified way.

>> We also need to distinguish between APIs, which should use first-class
>> objects (<channel>, etc.), and data formats, which are plain sexps.
>> (The above example is a mixture of both and in fact looks very similar
>> to what’s already in the ‘manifest’ file.)  But then again, that means
>> relying on a larger chunk of the API.
>
> I'm not sure I understood what you meant here.  Which APIs?

‘specification->package’ is part of the API.

>> So, all in all, I think I’d rather see it implemented as ‘guix package
>> --export’ or similar.
>
> I think I didn't understand why you'd prefer to have a command instead
> of systematically generating the file inside the profile.  Or maybe we
> could generate the file somewhere else if that's the problem?
>
> In my opinion, the `--export` approach would be cumbersome because it
> means that for users who want to save the specifications file, they'd
> need to systematically call it on every profile-modifying command, e.g.
>
>   guix package -m manifest.scm && guix package --export

I agree that ‘--export’ is less convenient.  Note that ‘guix system
reconfigure’ does exactly what you have in mind: it stores a
‘channels.scm’ and a ‘config.scm’ file in the system (in addition to
serialized & versioned metadata in the ‘provenance’ file) because that’s
so convenient:

  guix time-machine -C /run/current-system/channels.scm --
    system build --save-provenance -C /run/current-system/configuration.scm

But in this case it’s OK: ‘channels.scm’ uses a tiny teeny subset of the
API, and ‘configuration.scm’ is evaluated in the right context where the
APIs it expects are available.

Does that make sense?

We cannot do the same thing with profiles because of the possibly
multiple provenances.

>> As far as faithfulness is concerned, we should also keep in mind that we
>> don’t always have all the information needed to reconstruct what’s in a
>> profile.  For example, we lack information about the package
>> transformation options that were used (if any),
>
> Like `--with-input'?
> I didn't think of this.  Can't we extend the format then to include all
> transformation options, with future-proof provisions?

We could store package transformations as manifest entry properties.

However, that’ll be an approximation: the exact implementation of
‘--with-input’, for instance, can vary over time.

>> and we lack information about arbitrary user code that might have been
>> used to generate manifest entries.
>
> Is this a problem?

All I’m saying is that we can only approximate all these things.
Because of that, it may make more sense to not over-engineer the thing
and focus on making a rough approximation.

After all, the goal of the functionality we’re discussing is to allow
users to move towards the declarative ‘manifest.scm’ style, right?

Thanks,
Ludo’.

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 3470 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> Turns out I just watched your talk (since I couldn’t be there at the
> time!).  I was surprised you didn’t explicitly mention ‘guix search’ and
> the shortcomings you’d like to address (well, not directly), but I liked
> the perspectives!

I guess I took "guix search" for granted and assumed everyone knew what
I was talking about :p

> I would, however, use sexps as a serialization format.  Compared to an
> API, an object serialized to an sexp has the advantage that we can write
> code to handle changes in the serialization format, so it’s future-proof
> if we get it right.  But then: we’re back to ‘manifest’.  :-)
>
> I hope this is a bit clearer, but I realize it’s tricky to discuss such
> things!

Indeed, because we were on the same page all along: my "keyed code snippet"
was about the serialization format, not the data structure!

So agreed on all points here.

> ‘manifest’ looks like this:
>
>   (manifest
>     (version 3)
>     …)
>
> We have an explicit ‘read-manifest’ procedure that can handle version 3,
> but also prior versions, and this is all transparent.
>
> You cannot do that with code.  Code is just evaluated, and if it’s
> incompatible, if fails in some unspecified way.

Same thing, what I had in mind was to store the version number in the
_serialized_ specifications.scm, as for the manifest.

This way I believe we can support multiple version for specifications.scm.
Am I missing something?

> I agree that ‘--export’ is less convenient.  Note that ‘guix system
> reconfigure’ does exactly what you have in mind: it stores a
> ‘channels.scm’ and a ‘config.scm’ file in the system (in addition to
> serialized & versioned metadata in the ‘provenance’ file) because that’s
> so convenient:
>
>   guix time-machine -C /run/current-system/channels.scm --
>     system build --save-provenance -C /run/current-system/configuration.scm

Nice example, thanks for sharing.

> But in this case it’s OK: ‘channels.scm’ uses a tiny teeny subset of the
> API, and ‘configuration.scm’ is evaluated in the right context where the
> APIs it expects are available.
>
> Does that make sense?
>
> We cannot do the same thing with profiles because of the possibly
> multiple provenances.

With the serialization I proposed, the provenance is serialize
per-package.  Do you think it would still be a problem?

> We could store package transformations as manifest entry properties.
>
> However, that’ll be an approximation: the exact implementation of
> ‘--with-input’, for instance, can vary over time.

Hmmm, even if we have the provenance?  If so, we could re-use a given
version of Guix to apply the transformation.  Maybe too
sophisticated for what it's worth.

> All I’m saying is that we can only approximate all these things.
> Because of that, it may make more sense to not over-engineer the thing
> and focus on making a rough approximation.

Absolutely.

> After all, the goal of the functionality we’re discussing is to allow
> users to move towards the declarative ‘manifest.scm’ style, right?

Yes, so I'll try to sum up what I want to achieve in one sentence:
"automate the textual serialization of profile specifications to
simplify their backup/deployment/reproduction".

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[Ludovic Courtès]

Howdy!

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> I would, however, use sexps as a serialization format.  Compared to an
>> API, an object serialized to an sexp has the advantage that we can write
>> code to handle changes in the serialization format, so it’s future-proof
>> if we get it right.  But then: we’re back to ‘manifest’.  :-)
>>
>> I hope this is a bit clearer, but I realize it’s tricky to discuss such
>> things!
>
> Indeed, because we were on the same page all along: my "keyed code snippet"
> was about the serialization format, not the data structure!

Good.

>> ‘manifest’ looks like this:
>>
>>   (manifest
>>     (version 3)
>>     …)
>>
>> We have an explicit ‘read-manifest’ procedure that can handle version 3,
>> but also prior versions, and this is all transparent.
>>
>> You cannot do that with code.  Code is just evaluated, and if it’s
>> incompatible, if fails in some unspecified way.
>
> Same thing, what I had in mind was to store the version number in the
> _serialized_ specifications.scm, as for the manifest.
>
> This way I believe we can support multiple version for specifications.scm.
> Am I missing something?

Then I don’t get how ‘specifications.scm’ would differ from ‘manifest’.
In both cases we’re talking about a serialization format containing
per-package provenance info, right?

>> We could store package transformations as manifest entry properties.
>>
>> However, that’ll be an approximation: the exact implementation of
>> ‘--with-input’, for instance, can vary over time.
>
> Hmmm, even if we have the provenance?

Yes.

>> After all, the goal of the functionality we’re discussing is to allow
>> users to move towards the declarative ‘manifest.scm’ style, right?
>
> Yes, so I'll try to sum up what I want to achieve in one sentence:
> "automate the textual serialization of profile specifications to
> simplify their backup/deployment/reproduction".

OK.

To me, serialization is already automated: it’s the ‘manifest’ file.

What we lack is the ability to obtain a “manifest.scm” kind of file
(code) that users can pick as a starting point in their migration
towards declarative deployment.

Does that make sense?  Am I paraphrasing you correctly?  :-)

Thanks,
Ludo’.

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 3576 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

>> Same thing, what I had in mind was to store the version number in the
>> _serialized_ specifications.scm, as for the manifest.
>>
>> This way I believe we can support multiple version for specifications.scm.
>> Am I missing something?
>
> Then I don’t get how ‘specifications.scm’ would differ from ‘manifest’.
> In both cases we’re talking about a serialization format containing
> per-package provenance info, right?

The main differences:
- specifications.scm is a good fit to the --manifest argument.
- specifications.scm is human-editable because it does not contain
fields like the store path.

>> Yes, so I'll try to sum up what I want to achieve in one sentence:
>> "automate the textual serialization of profile specifications to
>> simplify their backup/deployment/reproduction".
>
> OK.
>
> To me, serialization is already automated: it’s the ‘manifest’ file.
>
> What we lack is the ability to obtain a “manifest.scm” kind of file
> (code) that users can pick as a starting point in their migration
> towards declarative deployment.

You mean a script like the following?

--8<---------------cut here---------------start------------->8---
;; Run with:
;;     guile -s FILE ~/.guix-profile

(use-modules (guix profiles)
             (gnu packages)             ; fold-packages
             (guix packages)            ; package structure
             (ice-9 match)
             (ice-9 pretty-print))


(define (packages-by-name name)
  (fold-packages (lambda (package list)
                   (if (string=? (package-name package) name)
                       (cons package list)
                       list))
                 '()))

(define (guix-manifest where)
  (sort (map (lambda (entry)
               (let* ((name (manifest-entry-name entry))
                      (out (manifest-entry-output entry))
                      (version (manifest-entry-version entry))
                      (default-version (match (packages-by-name name)
                                         ((first-name . rest)
                                          (package-version
                                           first-name))
                                         (else #f))))
                 (string-append name
                                (if (and default-version
                                         (not (string= version default-version)))
                                    (format #f "@~a" version)
                                    "")
                                (if (string= out "out")
                                    ""
                                    (format #f ":~a" out)))))
             (manifest-entries (profile-manifest where)))
        string<?))

;; Thanks to Ivan Vilata-i-Balaguer for this:
(define (guix-commit)
  (let ((guix-manifest (profile-manifest (string-append (getenv "HOME") "/.config/guix/current"))))
    (match (assq 'source (manifest-entry-properties (car (manifest-entries guix-manifest))))
      (('source ('repository ('version 0) _ _
                             ('commit commit) _ ...))
       commit)
      (_ #f))))

(match (command-line)
  ((_ where)
   (format #t ";; commit: ~a\n" (guix-commit))
   (pretty-print
    `(specifications->manifest
      ',(guix-manifest where))))
  (_ (error "Please provide the path to a Guix profile.")))
--8<---------------cut here---------------end--------------->8---

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[Ludovic Courtès]

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> What we lack is the ability to obtain a “manifest.scm” kind of file
>> (code) that users can pick as a starting point in their migration
>> towards declarative deployment.
>
> You mean a script like the following?
>
> ;; Run with:
> ;;     guile -s FILE ~/.guix-profile
>
> (use-modules (guix profiles)
>              (gnu packages)             ; fold-packages
>              (guix packages)            ; package structure
>              (ice-9 match)
>              (ice-9 pretty-print))
>
>
> (define (packages-by-name name)
>   (fold-packages (lambda (package list)
>                    (if (string=? (package-name package) name)
>                        (cons package list)
>                        list))
>                  '()))
>
> (define (guix-manifest where)
>   (sort (map (lambda (entry)
>                (let* ((name (manifest-entry-name entry))
>                       (out (manifest-entry-output entry))
>                       (version (manifest-entry-version entry))
>                       (default-version (match (packages-by-name name)
>                                          ((first-name . rest)
>                                           (package-version
>                                            first-name))
>                                          (else #f))))
>                  (string-append name
>                                 (if (and default-version
>                                          (not (string= version default-version)))
>                                     (format #f "@~a" version)
>                                     "")
>                                 (if (string= out "out")
>                                     ""
>                                     (format #f ":~a" out)))))
>              (manifest-entries (profile-manifest where)))
>         string<?))

Yes!  That alone would already be a useful tool to migrate from the
“imperative” style to the declarative style.  I’m sure it’d be good
enough 90% of the time.

(Note: we should remove the ‘sort’ call here as the order of packages in
the manifest is significant when there are file collisions.)

How does that sound?  Would you like to integrate something like this,
maybe ‘guix package --export’ or something?

> ;; Thanks to Ivan Vilata-i-Balaguer for this:
> (define (guix-commit)
>   (let ((guix-manifest (profile-manifest (string-append (getenv "HOME") "/.config/guix/current"))))
>     (match (assq 'source (manifest-entry-properties (car (manifest-entries guix-manifest))))
>       (('source ('repository ('version 0) _ _
>                              ('commit commit) _ ...))
>        commit)
>       (_ #f))))
>
> (match (command-line)
>   ((_ where)
>    (format #t ";; commit: ~a\n" (guix-commit))
>    (pretty-print
>     `(specifications->manifest
>       ',(guix-manifest where))))

Emitting channel info as comments like this, and/or a warning when
there’s a conflict (packages coming from different commits of the same
channel) could be done as a second step.

Thanks,
Ludo’.

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1295 bytes --]

Ludovic Courtès <ludo@gnu.org> writes:

> (Note: we should remove the ‘sort’ call here as the order of packages in
> the manifest is significant when there are file collisions.)

But the export is not an "internal manifest", it should not need to know
about package order, right?

I sorted the export so that it produces a reproducible output, which is
more version-control friendly.

> How does that sound?  Would you like to integrate something like this,
> maybe ‘guix package --export’ or something?

While it would be good to have it, what I was suggesting is that we
automatically generate this export on every profile generation, just
like we do with system profiles and

--8<---------------cut here---------------start------------->8---
/run/current-system/configuration.scm
/run/current-system/channels.scm
--8<---------------cut here---------------end--------------->8---

I didn't understand from your previous message which downside you find
to embedding the specifications.scm file.  It would save efforts to the
user and be more systematic, in the sense that it gives a guarantee to
the user that a specifications.scm corresponds to a profile.  This last
point, I believe, is crucial.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[zimoun]

On Tue, 25 Feb 2020 at 11:32, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
> Ludovic Courtès <ludo@gnu.org> writes:

> > (Note: we should remove the ‘sort’ call here as the order of packages in
> > the manifest is significant when there are file collisions.)
>
> But the export is not an "internal manifest", it should not need to know
> about package order, right?
>
> I sorted the export so that it produces a reproducible output, which is
> more version-control friendly.

IMHO, the sort would not be too expensive; even on all the packages
that Guix already includes now. ;-)
So, it does not matter too much, I guess.

However, the manifest returned is not necessary functional because the
pkg1@v1 and pkg2@v2 are not necessary provided by the same commit.
Therefore we could imagine "options" to the '--export' command, as
'--export=full' or '--export=light' etc. And even '--export=sorted'
and maybe combined as '--export=light,sorted'.


Cheers,
simon

Re: Store channel specification in profile

[zimoun]

Hi Ludo,

On Mon, 24 Feb 2020 at 17:16, Ludovic Courtès <ludo@gnu.org> wrote:

> > ;; Thanks to Ivan Vilata-i-Balaguer for this:
> > (define (guix-commit)
> >   (let ((guix-manifest (profile-manifest (string-append (getenv "HOME") "/.config/guix/current"))))
> >     (match (assq 'source (manifest-entry-properties (car (manifest-entries guix-manifest))))
> >       (('source ('repository ('version 0) _ _
> >                              ('commit commit) _ ...))
> >        commit)
> >       (_ #f))))
> >
> > (match (command-line)
> >   ((_ where)
> >    (format #t ";; commit: ~a\n" (guix-commit))
> >    (pretty-print
> >     `(specifications->manifest
> >       ',(guix-manifest where))))
>
> Emitting channel info as comments like this, and/or a warning when
> there’s a conflict (packages coming from different commits of the same
> channel) could be done as a second step.

The snippet uses the last commit of "guix pull", right?
Or the commit of last installed package with "guix package -i"?

Even if it can be done as a second step, the commits of each installed
packages are IMHO the key point of the exporter. Because otherwise, I
am doing "guix package -I | cut" and I am almost done.

Cheers,
simon

Re: Store channel specification in profile

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 1572 bytes --]

zimoun <zimon.toutoune@gmail.com> writes:

>> I sorted the export so that it produces a reproducible output, which is
>> more version-control friendly.
>
> IMHO, the sort would not be too expensive; even on all the packages
> that Guix already includes now. ;-)
> So, it does not matter too much, I guess.

I think we are talking about different things :)

What I meant is that if you keep the exported specs under version
control, you will want the packages to be in the same order every time.
Other wise, you could very well have

A
B
C

then on next export

A
C
B

which is the same, but will warrant another commit in your repository
(for no good reason).
Ideally, we would like to avoid useless  commits.

> However, the manifest returned is not necessary functional because the
> pkg1@v1 and pkg2@v2 are not necessary provided by the same commit.
> Therefore we could imagine "options" to the '--export' command, as
> '--export=full' or '--export=light' etc. And even '--export=sorted'
> and maybe combined as '--export=light,sorted'.

As you mentioned in the other email, I think a more convenient solution
to this is to use the format that I suggested which will include the
provenance.

This will save the use from having to deal with too many CLI parameters.

Which I'm still against.  I find that including the specs within the
profile is a much better idea: less effort, plus it ties the spec to a
profile, thus giving a relationship guarantee to the user.

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Store channel specification in profile

[zimoun]

Hi Pierre,

On Wed, 4 Mar 2020 at 09:09, Pierre Neidhardt <mail@ambrevar.xyz> wrote:
>
> zimoun <zimon.toutoune@gmail.com> writes:
>
> >> I sorted the export so that it produces a reproducible output, which is
> >> more version-control friendly.
> >
> > IMHO, the sort would not be too expensive; even on all the packages
> > that Guix already includes now. ;-)
> > So, it does not matter too much, I guess.
>
> I think we are talking about different things :)

We are talking about the same thing. :-)
I am just saying that even if a profile is full of all the packages
that exists on Earth for Guix, then sorting will not be too
costly---if the 'sort' function in Guile is the state-of-art of
sorting, say merge-sort or quick-sort or tim-sort.


> What I meant is that if you keep the exported specs under version
> control, you will want the packages to be in the same order every time.

I agree.


> Other wise, you could very well have
>
> A
> B
> C
>
> then on next export
>
> A
> C
> B
>
> which is the same, but will warrant another commit in your repository
> (for no good reason).
> Ideally, we would like to avoid useless  commits.
>
> > However, the manifest returned is not necessary functional because the
> > pkg1@v1 and pkg2@v2 are not necessary provided by the same commit.
> > Therefore we could imagine "options" to the '--export' command, as
> > '--export=full' or '--export=light' etc. And even '--export=sorted'
> > and maybe combined as '--export=light,sorted'.
>
> As you mentioned in the other email, I think a more convenient solution
> to this is to use the format that I suggested which will include the
> provenance.

I do not remember exactly the format but I do remember our discussion
at the Guix Days.:-)
And I agree. :-)

> This will save the use from having to deal with too many CLI parameters.

Well, I was suggesting options for the CLI '--export' which was the
Ludo's suggestion.
These options are almost nothing to do once the exporter is done
(roughly your script sent elsewhere in this thread).
It is complementary to this very format, IMHO.


> Which I'm still against.  I find that including the specs within the
> profile is a much better idea: less effort, plus it ties the spec to a
> profile, thus giving a relationship guarantee to the user.

Well, I do not understand the '--export' suggestion as a replacement
of the format specification but I do understand it as an easy move
that improves the current situation. :-)

Cheers,
simon