Using --manfistest with <profile>/manifest files

Using --manfistest with <profile>/manifest files

[elaexuotee]

[-- Attachment #1.1: Type: text/plain, Size: 1148 bytes --]

In an attempt to tar up the *build* environment for a package to share with a
colleague, I encountered this:

    [env]$ guix pack -m $GUIX_ENVIRONMENT/manifest
    (manifest ...): Wrong number of arguments

From playing around a bit, my guess is that the `<profile>/manifest' files are
just human-readable serializations of <manifest> objects and don't deserialize
back:

    $ guix environment -m ~/.guix-profile/manifest
    (manifest ...): Wrong number of arguments

Asking around on #guix, user leoprikler suggested a workaround by putting this
in a file:

    (call-with-input-file
      "/path/to/manifest"
      (@@ (guix profiles) read-manifest))

and feeding that file to --manifest. This works as intended.

First, am I missing something? Is there a better/preferred way to make use of
the `manifest' files in profiles?

If not, does it seem reasonable to make the `manifest' files automatically
deserialize into the objects they describe? I think this could be a fun, simple
project for a Guix newbie like me, but before I start figuring out how to do
this, I just want to sanity check the idea.

Cheers!

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Using --manfistest with <profile>/manifest files

[Ludovic Courtès]

Hi,

elaexuotee@wilsonb.com skribis:

> In an attempt to tar up the *build* environment for a package to share with a
> colleague, I encountered this:
>
>     [env]$ guix pack -m $GUIX_ENVIRONMENT/manifest
>     (manifest ...): Wrong number of arguments
>
> From playing around a bit, my guess is that the `<profile>/manifest' files are
> just human-readable serializations of <manifest> objects and don't deserialize
> back:
>
>     $ guix environment -m ~/.guix-profile/manifest
>     (manifest ...): Wrong number of arguments
>
> Asking around on #guix, user leoprikler suggested a workaround by putting this
> in a file:
>
>     (call-with-input-file
>       "/path/to/manifest"
>       (@@ (guix profiles) read-manifest))
>
> and feeding that file to --manifest. This works as intended.
>
> First, am I missing something? Is there a better/preferred way to make use of
> the `manifest' files in profiles?

You’re not missing anything: it’s a longstanding source of confusion
that these ‘manifest’ files are not like the ‘manifest.scm’ files.
These ‘manifest’ files are meant for internal consumption.

I think Pierre Neidhardt started looking at ways to address it from
different angles, but that hasn’t materialized yet.  One thing we
discussed was to add an ‘--export-manifest’ (or similar) option that
would export a ‘manifest.scm’ file for a profile, where the
‘manifest.scm’ file could be an approximation of what the profile
contains, in some cases.

> If not, does it seem reasonable to make the `manifest' files automatically
> deserialize into the objects they describe? I think this could be a fun, simple
> project for a Guix newbie like me, but before I start figuring out how to do
> this, I just want to sanity check the idea.

Yup!

Thanks,
Ludo’.

Re: Using --manfistest with <profile>/manifest files

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]

Hi,

Indeed, that was discussed at length but we still need to reach a
consensus :)

I personally disagree with the --export-manifest suggestion (which may
be one of the reasons why this is stalling :p).

Links to the related discussions (lots to read in there!):

- Store channel specification in profile:
  https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00464.html

  In particular, from there:
  https://lists.gnu.org/archive/html/guix-devel/2020-02/msg00047.html

While I'm at it, Ludo, if you want to continue the discussion, we've
left it on Feb 25th with this question from me to you:

> I didn't understand from your previous message which downside you find
> to embedding the specifications.scm file.  It would save efforts to the
> user and be more systematic, in the sense that it gives a guarantee to
> the user that a specifications.scm corresponds to a profile.  This last
> point, I believe, is crucial.

Cheers!

--
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Using --manfistest with <profile>/manifest files

[Ludovic Courtès]

Hi,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Indeed, that was discussed at length but we still need to reach a
> consensus :)
>
> I personally disagree with the --export-manifest suggestion (which may
> be one of the reasons why this is stalling :p).

Ah, OK.

> Links to the related discussions (lots to read in there!):
>
> - Store channel specification in profile:
>   https://lists.gnu.org/archive/html/guix-devel/2019-11/msg00464.html
>
>   In particular, from there:
>   https://lists.gnu.org/archive/html/guix-devel/2020-02/msg00047.html
>
> While I'm at it, Ludo, if you want to continue the discussion, we've
> left it on Feb 25th with this question from me to you:
>
>> I didn't understand from your previous message which downside you find
>> to embedding the specifications.scm file.  It would save efforts to the
>> user and be more systematic, in the sense that it gives a guarantee to
>> the user that a specifications.scm corresponds to a profile.  This last
>> point, I believe, is crucial.

I think there were several issues we discussed:

  1. We can only approximate that actual profile content; storing
     an approximate ‘manifest.scm’ along with the profile would IMO be
     deceptive.

  2. It’s easy to maintain compatibility over a data format, but it’s
     much harder to maintain compatibility for code.

I think we discussed these issues the best we could in the megathread,
so I’m personally in favor of moving forward in a pragmatic way.

Thanks,
Ludo’.

Re: Using --manfistest with <profile>/manifest files

[Pierre Neidhardt]

[-- Attachment #1: Type: text/plain, Size: 809 bytes --]

Hi,

Ludovic Courtès <ludo@gnu.org> writes:

> I think there were several issues we discussed:

Allow me to reiterate, at the risk of bikeshedding... :)

>   1. We can only approximate that actual profile content; storing
>      an approximate ‘manifest.scm’ along with the profile would IMO be
>      deceptive.

Why is an approximate file more of a problem than a command producing
an approximate result?  We could include a warning comment at the top of
the file to make it explicit that it's approximate.

>   2. It’s easy to maintain compatibility over a data format, but it’s
>      much harder to maintain compatibility for code.

I'm confused: my suggestion is to add a new data format, so isn't it
better then?

Cheers!

-- 
Pierre Neidhardt
https://ambrevar.xyz/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

Re: Using --manfistest with <profile>/manifest files

[Ludovic Courtès]

Hi Pierre,

Pierre Neidhardt <mail@ambrevar.xyz> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:
>
>> I think there were several issues we discussed:
>
> Allow me to reiterate, at the risk of bikeshedding... :)
>
>>   1. We can only approximate that actual profile content; storing
>>      an approximate ‘manifest.scm’ along with the profile would IMO be
>>      deceptive.
>
> Why is an approximate file more of a problem than a command producing
> an approximate result?  We could include a warning comment at the top of
> the file to make it explicit that it's approximate.
>
>>   2. It’s easy to maintain compatibility over a data format, but it’s
>>      much harder to maintain compatibility for code.
>
> I'm confused: my suggestion is to add a new data format, so isn't it
> better then?

We seem to be talking past each other.  IMO, exporting a file is easily
done and better than nothing.  I think we should be pragmatic.

If you want, maybe you can prototype something that we could discuss?

Thanks,
Ludo’.

Re: Using --manfistest with <profile>/manifest files

[zimoun]

Hi,

On Sun, 14 Jun 2020 at 17:24, Ludovic Courtès <ludo@gnu.org> wrote:

> I think there were several issues we discussed:
>
>   1. We can only approximate that actual profile content; storing
>      an approximate ‘manifest.scm’ along with the profile would IMO be
>      deceptive.
>
>   2. It’s easy to maintain compatibility over a data format, but it’s
>      much harder to maintain compatibility for code.
>
> I think we discussed these issues the best we could in the megathread,
> so I’m personally in favor of moving forward in a pragmatic way.

By pragmatic way, you mean:

 - let the format of <profile>/manifest as it is,
 - write '--export-manifest' as an approximation

right?

Well, I personally changed my workflow and now I always use manifest
files.  And the situation that I described in the manifest about the
"Working Scientific" doing install, pull, install, pull, remove,
install, etc. is rooted in bad practises, so it should be avoided.

Therefore, I agree that '--export-manifest' is the right approach, as an
helping tool; too bad for some corner cases. :-)


All the best,
simon

Re: Using --manfistest with <profile>/manifest files

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> On Sun, 14 Jun 2020 at 17:24, Ludovic Courtès <ludo@gnu.org> wrote:
>
>> I think there were several issues we discussed:
>>
>>   1. We can only approximate that actual profile content; storing
>>      an approximate ‘manifest.scm’ along with the profile would IMO be
>>      deceptive.
>>
>>   2. It’s easy to maintain compatibility over a data format, but it’s
>>      much harder to maintain compatibility for code.
>>
>> I think we discussed these issues the best we could in the megathread,
>> so I’m personally in favor of moving forward in a pragmatic way.
>
> By pragmatic way, you mean:
>
>  - let the format of <profile>/manifest as it is,
>  - write '--export-manifest' as an approximation
>
> right?

For example, yes.

> Well, I personally changed my workflow and now I always use manifest
> files.  And the situation that I described in the manifest about the
> "Working Scientific" doing install, pull, install, pull, remove,
> install, etc. is rooted in bad practises, so it should be avoided.
>
> Therefore, I agree that '--export-manifest' is the right approach, as an
> helping tool; too bad for some corner cases. :-)

Yeah, I think our goal is just to provide a tool to migrate from the
“imperative” way to the declarative way.  Once people have gotten
started with manifests, they no longer need that migration tool.

Thanks,
Ludo’.

Re: Using --manfistest with <profile>/manifest files

[elaexuotee]

[-- Attachment #1.1: Type: text/plain, Size: 1438 bytes --]

> 1. We can only approximate that actual profile content; storing
>    an approximate ‘manifest.scm’ along with the profile would IMO be
>    deceptive.

Is this a technical barrier or a pragmatic one?

If it is the former, then I don't quite grok why. I explain my reasoning in
great detail in a previous reply, but the gist is this: generating profiles is
deterministic, no? That is, given the inputs of channel revision, command line
invocation, and any referred-to external files, then `guix time-machine' is
enough to recreate a profile, no?

If the problem is of pragmatics, then at the very least I would be interested
in hearing a delineation of the challenges. I think this could be helpful for
the discussion though.

> Yeah, I think our goal is just to provide a tool to migrate from the
> “imperative” way to the declarative way.  Once people have gotten
> started with manifests, they no longer need that migration tool.

Would you mind commenting on the use case that I started this thread with?
Specifically, I was trying to `guix pack' a `guix environment'. The equivalent
is straightforward for purely --ad-hoc environments but not otherwise.

Personally, I have already encountered several instances where this would have
been useful. I also think it would be just plain cool to have the ability to
pack up, containerize, and share arbitrary profiles with non-guix users.

Thoughts?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Using --manfistest with <profile>/manifest files

[zimoun]

Dear,

On Tue, 16 Jun 2020 at 20:33, elaexuotee@wilsonb.com wrote:
>> 1. We can only approximate that actual profile content; storing
>>    an approximate ‘manifest.scm’ along with the profile would IMO be
>>    deceptive.
>
> Is this a technical barrier or a pragmatic one?

[...]

> If the problem is of pragmatics, then at the very least I would be interested
> in hearing a delineation of the challenges. I think this could be helpful for
> the discussion though.

It is a pragmatic one. As any good Zen says: "Now is better than never.
Although never is often better than *right* now."

Going from imperative/sequential "install, pull, remove" way to the
declarative manifest.scm way, in the general case, needs to change the
format of <profile>/manifest (or add another file). Which means
transition plan, etc.. Otherwise, on the technical level, all the
material is there.

So it is some work and it is not clear that it will pay off.


>> Yeah, I think our goal is just to provide a tool to migrate from the
>> “imperative” way to the declarative way.  Once people have gotten
>> started with manifests, they no longer need that migration tool.
>
> Would you mind commenting on the use case that I started this thread with?
> Specifically, I was trying to `guix pack' a `guix environment'. The equivalent
> is straightforward for purely --ad-hoc environments but not otherwise.

I do not see why it is straightforward for some cases.


> Personally, I have already encountered several instances where this would have
> been useful. I also think it would be just plain cool to have the ability to
> pack up, containerize, and share arbitrary profiles with non-guix users.

Well, I have re-read your initial message and maybe miscommunication
here. :-)

Because your use case -- pack an existing profile for sharing -- is not
really related to transform <profile>/manifest to a valid manifest.scm,
if I understand correctly.  And I agree with you that it should be
possible to pack an existing profile (created by any mean).

Does "pack --profile-name" fit your needs?

If not, yes packing an existing profile could be a feature to "guix
pack" -- doing transparently something similar to the Leo's suggestion
-- because it is an internal consumption of these <profile>/manifest
files.


All the best,
simon

Re: Using --manfistest with <profile>/manifest files

[elaexuotee]

[-- Attachment #1.1: Type: text/plain, Size: 6069 bytes --]

Thank you for the direct reply, zimoun.

> [The problem] is a pragmatic one. As any good Zen says: "Now is better than
> never. Although never is often better than *right* now."

Okay. If that is the case, then I very much empathize with the problem.

> Going from imperative/sequential "install, pull, remove" way to the
> declarative manifest.scm way, in the general case, needs to change the
> format of <profile>/manifest (or add another file). Which means
> transition plan, etc.. Otherwise, on the technical level, all the
> material is there.

That said. This makes me wonder that we may be thinking of different things
altogether.

The discussion seems to have congealed around smoothing the transition to
declarative profile management for users. However, the proposal I have in mind
is *first class profiles*. I am thinking of a file that can be fed to the
--manifest (or some potential future equivalent) option of various guix
commands. This hypothetical file would let users operate directly on profiles
as needed.

My current specific use case for this is packing the *development environments*
produced by `guix environment'.

> I do not see why it is straightforward for some cases.

    guix environment --ad-hoc <package>

can be indirectly packed via

    guix pack <package>

unless I am missing something. Otherwise, users are at an impasse.

> Because your use case -- pack an existing profile for sharing -- is not
> really related to transform <profile>/manifest to a valid manifest.scm,
> if I understand correctly.  And I agree with you that it should be
> possible to pack an existing profile (created by any mean).
> 
> Does "pack --profile-name" fit your needs?

I am not quite sure this works. From the manual:

> ‘--localstatedir’
> ‘--profile-name=NAME’
>      Include the “local state directory”, ‘/var/guix’, in the resulting
>      pack, and notably the ‘/var/guix/profiles/per-user/root/NAME’
>      profile—by default NAME is ‘guix-profile’, which corresponds to
>      ‘~root/.guix-profile’.
> 
>      ‘/var/guix’ contains the store database (*note The Store::) as well
>      as garbage-collector roots (*note Invoking guix gc::).  Providing
>      it in the pack means that the store is “complete” and manageable by
>      Guix; not providing it pack means that the store is “dead”: items
>      cannot be added to it or removed from it after extraction of the
>      pack.

I do not see how to use this with the transitory /gnu/store/<hash>-profile
produced by a `guix environment' invocation. Also, my intention is simply to
provide the profile's environment, not include the local state directory.

Put more simply, I want to be able to produce a tarball/container capable of
reproducing `guix environment --container <package>'. I think this would be
very useful.

Am I just failing to grok something fundamental? Thoughts?


More generally, I think first class profiles could be both a powerful feature
and an important future-proofing against extra maintenance burden. Profiles are
a central concept to guix usage. They form the atomic unit with which users
interact. Wanting to tarball a profile is just one use case, but future guix
commands (guix merge, anyone?) or future --manifest options (guix archive,
anyone?) seem likely to directly benefit from an existing infrastructure that
supports store profiles being created, recreated and munged.




zimoun <zimon.toutoune@gmail.com> wrote:
> Dear,
> 
> On Tue, 16 Jun 2020 at 20:33, elaexuotee@wilsonb.com wrote:
> >> 1. We can only approximate that actual profile content; storing
> >>    an approximate ‘manifest.scm’ along with the profile would IMO be
> >>    deceptive.
> >
> > Is this a technical barrier or a pragmatic one?
> 
> [...]
> 
> > If the problem is of pragmatics, then at the very least I would be interested
> > in hearing a delineation of the challenges. I think this could be helpful for
> > the discussion though.
> 
> It is a pragmatic one. As any good Zen says: "Now is better than never.
> Although never is often better than *right* now."
> 
> Going from imperative/sequential "install, pull, remove" way to the
> declarative manifest.scm way, in the general case, needs to change the
> format of <profile>/manifest (or add another file). Which means
> transition plan, etc.. Otherwise, on the technical level, all the
> material is there.
> 
> So it is some work and it is not clear that it will pay off.
> 
> 
> >> Yeah, I think our goal is just to provide a tool to migrate from the
> >> “imperative” way to the declarative way.  Once people have gotten
> >> started with manifests, they no longer need that migration tool.
> >
> > Would you mind commenting on the use case that I started this thread with?
> > Specifically, I was trying to `guix pack' a `guix environment'. The equivalent
> > is straightforward for purely --ad-hoc environments but not otherwise.
> 
> I do not see why it is straightforward for some cases.
> 
> 
> > Personally, I have already encountered several instances where this would have
> > been useful. I also think it would be just plain cool to have the ability to
> > pack up, containerize, and share arbitrary profiles with non-guix users.
> 
> Well, I have re-read your initial message and maybe miscommunication
> here. :-)
> 
> Because your use case -- pack an existing profile for sharing -- is not
> really related to transform <profile>/manifest to a valid manifest.scm,
> if I understand correctly.  And I agree with you that it should be
> possible to pack an existing profile (created by any mean).
> 
> Does "pack --profile-name" fit your needs?
> 
> If not, yes packing an existing profile could be a feature to "guix
> pack" -- doing transparently something similar to the Leo's suggestion
> -- because it is an internal consumption of these <profile>/manifest
> files.
> 
> 
> All the best,
> simon



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

how to "guix pack" a profile?

[zimoun]

Hi,

On Wed, 17 Jun 2020 at 09:45, elaexuotee@wilsonb.com wrote:

> The discussion seems to have congealed around smoothing the transition to
> declarative profile management for users. However, the proposal I have in mind
> is *first class profiles*. I am thinking of a file that can be fed to the
> --manifest (or some potential future equivalent) option of various guix
> commands. This hypothetical file would let users operate directly on profiles
> as needed.

[...]

> Put more simply, I want to be able to produce a tarball/container capable of
> reproducing `guix environment --container <package>'. I think this would be
> very useful.


Well, if I re-read correctly the emails and proposal, they are 2 points:

 1. Easily share a profile via "guix pack"
 2. A mean via recreating "manifest.scm" files

About the #2. it means changing a bit the format <profile>/manifest,
transition plan etc. and *if* someone comes with a prototype, it could
be happen, otherwise.  In the meantime, we need to work, so what could
happen is "--export-to-manifest" because it is doable and actionable.

The #1. is AFAIK new and it appears to me a good idea: add a feature to
"guix pack" and directly deal with profiles, i.e., use internally
<profile>/manifest, which appears to me doable and actionable.

Ludo, WDYT about "guix pack -p profile" to generate a (relocatable)
tarball or Docker image?  I mean if it is not already possible. :-)


> More generally, I think first class profiles could be both a powerful feature
> and an important future-proofing against extra maintenance burden. Profiles are
> a central concept to guix usage. They form the atomic unit with which users
> interact. Wanting to tarball a profile is just one use case, but future guix
> commands (guix merge, anyone?) or future --manifest options (guix archive,
> anyone?) seem likely to directly benefit from an existing infrastructure that
> supports store profiles being created, recreated and munged.

From my understanding, the way to go is the declarative via manifest.scm
and channel.scm.  Not profile.

For example, manifest.scm and channel.scm are easy to put under version
control system, they work trough all the Guix commands, etc..

You wrote elsewhere in this thread ``Naively, a profile is just a sum of
outputs'' which is correct, IMHO, but the correct level to interact with
and maintain this list is manifest.scm and channel.scm, again IMHO.


All the best,
simon

Re: how to "guix pack" a profile?

[elaexuotee]

[-- Attachment #1.1: Type: text/plain, Size: 2806 bytes --]

Zimoun, thank you for taking the time to closely read my long emails.

> Well, if I re-read correctly the emails and proposal, they are 2 points:
> 
>  1. Easily share a profile via "guix pack"
>  2. A mean via recreating "manifest.scm" files

Hrm. I agree that point 1 is the immediate goal I have and what prompted me to
create this issue. However, my proposal is more general and orthogonal to point
2.

> From my understanding, the way to go is the declarative via manifest.scm

Perhaps I am using terminology incorrectly. I completely agree that declarative
management of a `user profile' is the way to go. But before I go any further,
let me distinguish some terms:

By `profile', I mean any store output named like /gnu/store/<hash>-profile.
These outputs are created by `guix package' invocations as well as `guix
environment' ones.

By `user profile', I mean the (non-static) contents pointed to by
~/.guix-profile. These contents get updated on `guix package' invocations (by
updating the ~/.guix-profile symlink).

(Do the above two concepts have established nomenclature?)

Using this terminology, we both agree than managing your `user profile' via
"manifest.scm" (and "channels.scm") is the way to go. Those two files provide a
specification which guix *resolves* into a concrete `profile'.

The imminently practical issue that you, Ludo, and Pierre seem to be working on
is a way of easing the transition from imperative to declarative `user profile'
management.

In contrast, I am proposing making `profiles' first class. By this I mean a
file under /gnu/store/<hash>-profile that contains the code capable of
completely and deterministically rebuilding that exact profile, and that
profile *only*.

Just for the sake of argument, let's call this file "recipe.scm", and let's
choose a profile, /gnu/store/123abc-profile. Then the following should be true:

    $ guix package --manifest /gnu/store/123abc-profile/recipe.scm
    $ readlink -f ~/guix-profile
    /gnu/store/123abc-profile

I.e. that `guix package' invocation would have the effect of setting the
current user generation to that exact profile.

Of course, if we had such a "recipe.scm" then the following would Just Work:

    # Containerize a profile.
    $ guix pack --manifest /gnu/store/123abc-profile/recipe.scm

    # Share profiles with your Guix friends.
    $ guix environment --manifest ~/downloads/friends-awesome-profile.scm

And it plays well with potential future features:

    # Copy whole profiles between Guix stores
    $ guix archive --manifest /gnu/store/123abc-profile/recipe.scm

Hopefully my generalized idea is clearer now.

So, what do you say? Are "first class profiles" a coherent and interesting
concept? What about practicality?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: how to "guix pack" a profile?

[zimoun]

Dear,

On Thu, 18 Jun 2020 at 18:20, elaexuotee@wilsonb.com wrote:

> By `profile', I mean any store output named like /gnu/store/<hash>-profile.
> These outputs are created by `guix package' invocations as well as `guix
> environment' ones.
>
> By `user profile', I mean the (non-static) contents pointed to by
> ~/.guix-profile. These contents get updated on `guix package' invocations (by
> updating the ~/.guix-profile symlink).

Well, from my understanding, any profile point to /gnu/store/<hash>-profile.

--8<---------------cut here---------------start------------->8---
$ readlink -f ~/.guix-profile
/gnu/store/192k3kck2w3g2ms43dry41la9lwrqf37-profile

$ guix package -i hello -p /tmp/hello
$ readlink -f /tmp/hello
/gnu/store/p196x4m6ix33mgsbfacvqaw60129pndw-profile
--8<---------------cut here---------------end--------------->8---


> Using this terminology, we both agree than managing your `user profile' via
> "manifest.scm" (and "channels.scm") is the way to go. Those two files provide a
> specification which guix *resolves* into a concrete `profile'.
>
> The imminently practical issue that you, Ludo, and Pierre seem to be working on
> is a way of easing the transition from imperative to declarative `user profile'
> management.
>
> In contrast, I am proposing making `profiles' first class. By this I mean a
> file under /gnu/store/<hash>-profile that contains the code capable of
> completely and deterministically rebuilding that exact profile, and that
> profile *only*.

For which use case?

Well, let say there is a profile which could be a mess and I only see
two usages currently uncovered:

 1. create an environment based on this profile; either to temporarily
   extend it (which is AFAIK already more or less covered), either to
   isolate it,
 2. pack it for sharing with your colleague.


> Just for the sake of argument, let's call this file "recipe.scm", and let's
> choose a profile, /gnu/store/123abc-profile. Then the following should be true:
>
>     $ guix package --manifest /gnu/store/123abc-profile/recipe.scm
>     $ readlink -f ~/guix-profile
>     /gnu/store/123abc-profile
>
> I.e. that `guix package' invocation would have the effect of setting the
> current user generation to that exact profile.
>
> Of course, if we had such a "recipe.scm" then the following would Just Work:
>
>     # Containerize a profile.
>     $ guix pack --manifest /gnu/store/123abc-profile/recipe.scm
>
>     # Share profiles with your Guix friends.
>     $ guix environment --manifest ~/downloads/friends-awesome-profile.scm
>
> And it plays well with potential future features:
>
>     # Copy whole profiles between Guix stores
>     $ guix archive --manifest /gnu/store/123abc-profile/recipe.scm
>
> Hopefully my generalized idea is clearer now.

What you describe here is exactly what Pierre and other have proposed.
And the work-to-do is to prototype the format of what you called
"recipe.scm", which means equivalently in the previous emails change the
format of <profile>/manifest.


All the best,
simon

Re: how to "guix pack" a profile?

[elaexuotee]

[-- Attachment #1.1: Type: text/plain, Size: 3864 bytes --]

Ludo, Pierre, would you mind commenting? Starting to doubt my sanity here.

> Well, from my understanding, any profile point to /gnu/store/<hash>-profile.

Exactly. This is something we both know. I feel like we are not communicating
well with each other.

You emphasize that declarative package management with "manifest.scm" is the
way to go. I agree but wanted to point out that

    $ guix package -m manifest.scm

produces a different /gnu/store/<hash>-profile, depending on when it's run (or
more precisely, depending on which channel commits guix resolves for the
invocation).

You know that, of course, but the point I try to make is that "manifest.scm"
and "channels.scm" are *not enough* to uniquely determine a specific
/gnu/store/<hash>-profile. We need to separate out the concepts of "declarative
profile management" and "deterministic profile regeneration."

What Pierre (and others?) initially proposed---what I am re-proposing---is that
we put a blob of Guile into the profiles that is capable of uniquely and
completely generating the profile where it resides.

> For which use case?

The examples I gave in my previous email...
I am not sure what's getting miscommunicated here.

> Well, let say there is a profile which could be a mess and I only see
> two usages currently uncovered:
> 
>  1. create an environment based on this profile; either to temporarily
>    extend it (which is AFAIK already more or less covered), either to
>    isolate it,
>  2. pack it for sharing with your colleague.

Sure, about the only new feature now would be to guix pack an existing profile.

However, it's not too hard to think of potential uses or features users may
want in the future:

1. Composable profiles,
2. Sharing "light" profiles buy sending only the "recipe.scm" instead of an
   entire container.
3. guix archive --manifest
4. guix profile --manifest-from-recipe <profile>/recipe.scm

The last one there is intended to be the tool for "migrate from imperative to
declarative" user profile management, starting from a given profile.

> What you describe here is exactly what Pierre and other have proposed.
> And the work-to-do is to prototype the format of what you called
> "recipe.scm", which means equivalently in the previous emails change the
> format of <profile>/manifest.

I agree. However, in previous emails I have tried to make a rebuttal to Ludo's
argument than the best we can do is *approximate* a manifest.scm.

See https://lists.gnu.org/archive/html/guix-devel/2020-02/msg00098.html:

>>> (Ludo)
>>> As far as faithfulness is concerned, we should also keep in mind that we
>>> don’t always have all the information needed to reconstruct what’s in a
>>> profile.  For example, we lack information about the package
>>> transformation options that were used (if any),
>>
>> (Pierre)
>> Like `--with-input'?
>> I didn't think of this.  Can't we extend the format then to include all
>> transformation options, with future-proof provisions?
>
> (Ludo)
> We could store package transformations as manifest entry properties.
>
> However, that’ll be an approximation: the exact implementation of
> ‘--with-input’, for instance, can vary over time.

However, with `time-machine' and a given `guix environment' or `guix profile'
invocation, we are able to deterministically resolve a
/gnu/store/<hash>-profile, no? Better yet, this is in a future-proof way, no?
If that is so, then why not canonify this profile recipe in guile code instead
of what is needed now: guile + bash? What am I missing here?


Just to be clear, I would be more than thrilled with a --from-profile option to
`guix pack'. However, I am trying to make a case that "first class profiles"
is both feasible and may pay back more in maintenance cost than it consumes.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: how to "guix pack" a profile?

[zimoun]

Dear,

On Fri, 19 Jun 2020 at 11:52, elaexuotee@wilsonb.com wrote:

> What Pierre (and others?) initially proposed---what I am
> re-proposing---is that we put a blob of Guile into the profiles that
> is capable of uniquely and completely generating the profile where it
> resides.

Others?  For example me.  With Pierre, we spent some time at Guix Days
to propose a new format.

> 1. Composable profiles,

This is already possible.

> 2. Sharing "light" profiles buy sending only the "recipe.scm" instead
> of an entire container.

I am not sure to get what is a "light" profile but from my understanding
what you want here already exist and it is manifest.scm+channel.scm.

> 3. guix archive --manifest
> 4. guix profile --manifest-from-recipe <profile>/recipe.scm
>
> The last one there is intended to be the tool for "migrate from
> imperative to declarative" user profile management, starting from a
> given profile.

See below.

>> What you describe here is exactly what Pierre and other have
>> proposed.  And the work-to-do is to prototype the format of what you
>> called "recipe.scm", which means equivalently in the previous emails
>> change the format of <profile>/manifest.
>
> I agree. However, in previous emails I have tried to make a rebuttal
> to Ludo's argument than the best we can do is *approximate* a
> manifest.scm.
>
> See https://lists.gnu.org/archive/html/guix-devel/2020-02/msg00098.html:

I have the impression that the discussion is going nowhere.

Make what you called "recipe.scm" or modify "<profile/manifest" to
become an acceptable "--manifest" is doable.  Note that the idea is not
new either.  As I said, all the technical material is here.  But Ludo
and others (me) are not convinced that it will pay off because the
general case is hard.  Well, to be concrete, there are 2 possible next
actions:

 a) prototype the new format and implement it
 b) make an approximation exposed with "--export"

Frankly, I doubt that a) happens because no one will do the tough work.
Therefore, b) is pragmatic.


>>>> (Ludo)
>>>> As far as faithfulness is concerned, we should also keep in mind
>>>> that we don’t always have all the information needed to reconstruct
>>>> what’s in a profile.  For example, we lack information about the
>>>> package transformation options that were used (if any),
>>>
>>> (Pierre)
>>> Like `--with-input'?  I didn't think of this.  Can't we extend the
>>> format then to include all transformation options, with future-proof
>>> provisions?
>>
>> (Ludo)
>> We couldstore package transformations as manifest entry properties.
>>
>> However, that’ll be an approximation: the exact implementation of
>> ‘--with-input’, for instance, can vary over time.

This snippet of quotation shows that it is not "easy" for the general
case.  And the transformation using "--load-path" has not been evoked.
For example:

--8<---------------cut here---------------start------------->8---
(define-module (foo)
 #:use-module (guix packages)
 #:use-module (guix profiles) ;; imagine it is used
 #:use-module (gnu packages base))

(define (crazy-transformation pkg)
  "Could do really complicated things"
  (package
    (inherit pkg)
    (name (string-append "ex-" (package-name pkg)))))

(define bonjour
  (package
    (inherit hello)
    (name "bonjour")))

(define-public crazy-bonjour
  (crazy-transformation bonjour))
--8<---------------cut here---------------end--------------->8---

then "guix install -L /path/to/foo ex-bonjour".


> However, with `time-machine' and a given `guix environment' or `guix
> profile' invocation, we are able to deterministically resolve a
> /gnu/store/<hash>-profile, no? Better yet, this is in a future-proof
> way, no?  If that is so, then why not canonify this profile recipe in
> guile code instead of what is needed now: guile + bash? What am I
> missing here?

You are missing the difficulty to make it concretely, I guess. :-)


> Just to be clear, I would be more than thrilled with a --from-profile
> option to `guix pack'. However, I am trying to make a case that "first
> class profiles" is both feasible and may pay back more in maintenance
> cost than it consumes.

Thank you for the ideas.  Especially the two ones:

 1. create an environment from a profile
 2. pack a profile

Well, I do not know if they will happen but it should be cool to have.


I am going to move on since we already raised all the points, I guess. :-)


All the best,
simon

Re: how to "guix pack" a profile?

[elaexuotee]

[-- Attachment #1.1: Type: text/plain, Size: 1856 bytes --]

> > 1. Composable profiles,
> 
> This is already possible.

Would you kindly point me to the documentation? I am failing to find it.

By "composable profiles" what I have in mind is the sum of two existing
profiles and extending them as necessary

    $ guix environment --with-profile saved-profiles/profile-A \
                       --with-profile saved-profiles/profile-B \
                       <packageA>

> This snippet of quotation shows that it is not "easy" for the general
> case.  And the transformation using "--load-path" has not been evoked.
> For example:
> 
> --8<---------------cut here---------------start------------->8---
> (define-module (foo)
>  #:use-module (guix packages)
>  #:use-module (guix profiles) ;; imagine it is used
>  #:use-module (gnu packages base))
> 
> (define (crazy-transformation pkg)
>   "Could do really complicated things"
>   (package
>     (inherit pkg)
>     (name (string-append "ex-" (package-name pkg)))))
> 
> (define bonjour
>   (package
>     (inherit hello)
>     (name "bonjour")))
> 
> (define-public crazy-bonjour
>   (crazy-transformation bonjour))
> --8<---------------cut here---------------end--------------->8---
> 
> then "guix install -L /path/to/foo ex-bonjour".

Thank you for the concrete example.

I guess it wasn't clear in my previous emails, but I proposed *simply
copying /path/to/foo* into the generated profile and letting "manifest.scm"
refer to these copies. Is this obviously broken.

> Thank you for the ideas.  Especially the two ones:
> 
>  1. create an environment from a profile
>  2. pack a profile
> 
> Well, I do not know if they will happen but it should be cool to have.

I agree. If I'm to make any points, I should just submit some concrete patches.

Again, thank you for your time and patience.

Cheers,

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: how to "guix pack" a profile?

[zimoun]

On Fri, 19 Jun 2020 at 14:34, <elaexuotee@wilsonb.com> wrote:
> > > 1. Composable profiles,
> >
> > This is already possible.
>
> Would you kindly point me to the documentation? I am failing to find it.

  guix install hello -p foo
  guix install emacs -p bar
  eval $(guix package -p foo -p bar --search-paths=exact)

So it composes.  However maybe not what you had in mind. ;-)

> By "composable profiles" what I have in mind is the sum of two existing
> profiles and extending them as necessary
>
>     $ guix environment --with-profile saved-profiles/profile-A \
>                        --with-profile saved-profiles/profile-B \
>                        <packageA>

Yes, because "guix environment" cannot support "profile".  As said,
IMHO you raised 2 interesting use cases:

 1. create an environment based on a profile
 2. pack an existing profile

Obviously "a profile" means the composition of profiles if any.


> I agree. If I'm to make any points, I should just submit some concrete patches.

Patches are always welcome. :-)


Cheers,
simon

ps:
Well, maybe we are falling in the "take it to private email" thread pattern. ;-)
https://joeyh.name/blog/entry/thread_patterns/

Re: how to "guix pack" a profile?

[zimoun]

On Fri, 19 Jun 2020 at 11:52, elaexuotee@wilsonb.com wrote:

> You emphasize that declarative package management with "manifest.scm" is the
> way to go. I agree but wanted to point out that
>
>     $ guix package -m manifest.scm
>
> produces a different /gnu/store/<hash>-profile, depending on when it's run (or
> more precisely, depending on which channel commits guix resolves for the
> invocation).

Yes, but AFAIU the correct is:

  guix describe -f channels > channel.scm
  guix package -m manifest.scm

then later or elsewhere, reproduce with:

  guix time-machine -C channel.scm -- package -m manifest.scm


> You know that, of course, but the point I try to make is that "manifest.scm"
> and "channels.scm" are *not enough* to uniquely determine a specific
> /gnu/store/<hash>-profile. We need to separate out the concepts of "declarative
> profile management" and "deterministic profile regeneration."

Do you have examples?

I am not sure we are talking about the same channels.scm file.  I guess,
you have in mind ~/.config/guix/channels.scm but it is not the one of
interest for reproducing.  You need the real one, i.e., "guix describe
-f channels".

AFAIU, manifest.scm+$(guix describe -f channel) is enough to have
"deterministic profile (re)generation".  If not, it means there is a
"bug" in Inferiors or I miss something.


All the best,
simon

Re: how to "guix pack" a profile?

[elaexuotee]

[-- Attachment #1.1: Type: text/plain, Size: 690 bytes --]

> I am not sure we are talking about the same channels.scm file.  I guess,
> you have in mind ~/.config/guix/channels.scm but it is not the one of
> interest for reproducing.  You need the real one, i.e., "guix describe
> -f channels".
> 
> AFAIU, manifest.scm+$(guix describe -f channel) is enough to have
> "deterministic profile (re)generation".  If not, it means there is a
> "bug" in Inferiors or I miss something.

Bingo! This is exactly what I was missing. Thank you for sifting this out.

Given the stated "goal of ... [allowing] users to move towards declarative
'manifest.scm' style..." I completely mis-interpreted the sense in which you
were using "channels.scm".

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: how to "guix pack" a profile?

[Ludovic Courtès]

Hi,

elaexuotee@wilsonb.com skribis:

> What Pierre (and others?) initially proposed---what I am re-proposing---is that
> we put a blob of Guile into the profiles that is capable of uniquely and
> completely generating the profile where it resides.

What you propose is what ‘guix system reconfigure’ does: it produces
/run/current-system/{channels,configuration}.scm and you can reuse them
right away.

The different with ‘guix package’ is that ‘guix system’ supports nothing
but a declarative mode, so it’s easy to do the above.

Conversely, profiles can be managed “imperatively”.  Thus, in the
general case, it’s not possible to write a ‘manifest.scm’ file
corresponding to a profile.

I hope this clarifies the situation.

Ludo’.

Re: how to "guix pack" a profile?

[Ludovic Courtès]

Hi,

zimoun <zimon.toutoune@gmail.com> skribis:

> Ludo, WDYT about "guix pack -p profile" to generate a (relocatable)
> tarball or Docker image?  I mean if it is not already possible. :-)

That could work, but honestly I’d rather encourage -m.  I’m not
convinced there’s a need for what you suggest here.

Ludo’.

Re: Using --manfistest with <profile>/manifest files

[George Clemmer]

Ludovic Courtès <ludo@gnu.org> writes:

> elaexuotee@wilsonb.com skribis:

>> First, am I missing something? Is there a better/preferred way to make use of
>> the `manifest' files in profiles?

> You’re not missing anything: it’s a longstanding source of confusion
> that these ‘manifest’ files are not like the ‘manifest.scm’ files.
> These ‘manifest’ files are meant for internal consumption.

This hurt my head for a while a few years ago until I realized that
'manifest.scm' is the guix "order" and ‘.guix-profile/manifest’ is the
guix "packing list".

But actually a guix' 'manifest' packing list goes well beyond what we
normally find in a packing list by containing detailed info about how
the specific products were made, down to the specific design for the
specific version shipped.

Thought of this way it is easy to understand why a receiver of a
'manifest' can only estimate the set of 'manifest.scm' that might
produce it. A simple-minded example: did the manifest.scm specify the
version of the package shipped or is this an artifact of a) when
'manifest.scm' was processed or b) of the requirements of the other
packages that were received?

In any event, once I saw it this way it no longer troubled me that guix
doesn't have a pushbutton way to "reverse" 'manifest' into
'manifest.scm'.

ISTM we set ourselves up for confused users and a lot of explaining by
labeling two very different things with same name :-0

Yes, only 'manifest.scm' is in the doc, but '.guix-profile/manifest'
smacks a user in the face pretty quickly which leads to these messy
questions.

IMO we could dramatically simplify the situation, and simplify our
lives, by simply renaming the .guix-profile/manifest file ;-)

George

Re: Using --manfistest with <profile>/manifest files

[elaexuotee]

[-- Attachment #1.1: Type: text/plain, Size: 2728 bytes --]

This is a good point. The naming of `<package>/manifest' does invite confusion
when first encountering it.

That said, I am pretty sure there is a place for `<profile>/manifest.scm'.
Given the `--manifest' option to several commands, it makes profiles first
class. In particular, it would let users easily `guix pack' or `guix archive'
arbitrary profiles as needed. At the moment, it's not at all obvious how to
`guix pack' the equivalent of `guix environment --container <package>'.

FWIW, I never expected `<profile>/manifest' to encode "this is what the user
ordered," so much as "this is the recipe for (deterministically) reproducing
this exact profile." For the former we have `packages->manifest',
`specifications->manifest' etc. The latter is what I understand this discussion
to be about.

George Clemmer <myglc2@gmail.com> wrote:
> 
> Ludovic Courtès <ludo@gnu.org> writes:
> 
> > elaexuotee@wilsonb.com skribis:
> 
> >> First, am I missing something? Is there a better/preferred way to make use of
> >> the `manifest' files in profiles?
> 
> > You’re not missing anything: it’s a longstanding source of confusion
> > that these ‘manifest’ files are not like the ‘manifest.scm’ files.
> > These ‘manifest’ files are meant for internal consumption.
> 
> This hurt my head for a while a few years ago until I realized that
> 'manifest.scm' is the guix "order" and ‘.guix-profile/manifest’ is the
> guix "packing list".
> 
> But actually a guix' 'manifest' packing list goes well beyond what we
> normally find in a packing list by containing detailed info about how
> the specific products were made, down to the specific design for the
> specific version shipped.
> 
> Thought of this way it is easy to understand why a receiver of a
> 'manifest' can only estimate the set of 'manifest.scm' that might
> produce it. A simple-minded example: did the manifest.scm specify the
> version of the package shipped or is this an artifact of a) when
> 'manifest.scm' was processed or b) of the requirements of the other
> packages that were received?
> 
> In any event, once I saw it this way it no longer troubled me that guix
> doesn't have a pushbutton way to "reverse" 'manifest' into
> 'manifest.scm'.
> 
> ISTM we set ourselves up for confused users and a lot of explaining by
> labeling two very different things with same name :-0
> 
> Yes, only 'manifest.scm' is in the doc, but '.guix-profile/manifest'
> smacks a user in the face pretty quickly which leads to these messy
> questions.
> 
> IMO we could dramatically simplify the situation, and simplify our
> lives, by simply renaming the .guix-profile/manifest file ;-)
> 
> George



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Using --manfistest with <profile>/manifest files

[George Clemmer]

Hello!

elaexuotee@wilsonb.com writes:

> FWIW, I never expected `<profile>/manifest' to encode "this is what the user
> ordered," so much as "this is the recipe for (deterministically) reproducing
> this exact profile." For the former we have `packages->manifest',
> `specifications->manifest' etc. The latter is what I understand this discussion
> to be about.

I am sorry if it sounded like I was criticizing your level of guix
knowledge. Not my intent. You knowledge seems very high to me ;-)

I do admit to taking advantage of your public suffering in this guix
"trap" ...

>     $ guix environment -m ~/.guix-profile/manifest
>     (manifest ...): Wrong number of arguments

... and Ludo's acknowledgment of the "source of confusion that these
‘manifest’ files are not like the ‘manifest.scm’ files" that landed you
there to recycle a fix I proposed earlier:

https://lists.gnu.org/archive/html/guix-devel/2018-10/msg00011.html

Motivation: I see others fall into this trap periodically. When they
report their difficulty they get this "~/.guix-profile/manifest is not a
manifest" lecture.  This is a waste of time for all. It must be truly
annoying for someone in the trap to hear. And of course I fell into the
trap circa 2017 ;-)

FWIW, I support your request. In fact, I acctually suggest something
similar in the post referenced above.

HTH - George

Re: Using --manfistest with <profile>/manifest files

[zimoun]

Dear,

On Mon, 15 Jun 2020 at 22:51, George Clemmer <myglc2@gmail.com> wrote:

> ISTM we set ourselves up for confused users and a lot of explaining by
> labeling two very different things with same name :-0

I think there is a confusion here.  The file <profile>/manifest is an
internal detail implementation and the user should *not* be exposed to.

Well, there are a lot of internal files that the user is not aware.  And
sometimes, the names are more or less well-chosen.  But hey! it is
internal and naming is hard. :-)


> Yes, only 'manifest.scm' is in the doc, but '.guix-profile/manifest'
> smacks a user in the face pretty quickly which leads to these messy
> questions.

I am not convinced by "smacks in the face pretty quickly".  But I agree
that questions about this topic regularly come.  For example, to pick an
old one:

https://lists.gnu.org/archive/html/guix-devel/2018-10/msg00011.htlm


> IMO we could dramatically simplify the situation, and simplify our
> lives, by simply renaming the .guix-profile/manifest file ;-)

In contradiction with what I wrote above, I agree. :-)

<profile>/manifest should be renamed <profile>/specifications or
something like that.

And a comment could be inserted in this file saying: internal usage, do
not modify, etc..

WDYT?


All the best,
simon

Re: Using --manfistest with <profile>/manifest files

[George Clemmer]

Hi zimoun,

zimoun <zimon.toutoune@gmail.com> writes:

> In contradiction with what I wrote above, I agree. :-)
>
> <profile>/manifest should be renamed <profile>/specifications or
> something like that.
>
> And a comment could be inserted in this file saying: internal usage, do
> not modify, etc..
>
> WDYT?

Sure, that would work. But, on further thought, I would like to amend my
suggestion -- to change which ever is easier. I say that because ...

"manifest" occurs ~600 times in the ./guix directory. I am guessing its
use is deeply embedded with developers. If so, renaming it internally
seems like a bad idea. And if we write our internal manifest into the
profile and call it "specifications" it will only add the confusion.

OTOH, "manifest" occurs only ~50 times in guix.info and the "user API"
seems limited to the --manifest option and the functions:
specifications->manifest and packages->manifest. Furthermore it is a new
concept for new users. So I don't think users care what we call it.

Bottom line: change whichever is more convenient for developers.

HTH - George

Re: Using --manfistest with <profile>/manifest files

[zimoun]

Dear,

On Tue, 16 Jun 2020 at 10:03, George Clemmer <myglc2@gmail.com> wrote:

> "manifest" occurs ~600 times in the ./guix directory. I am guessing its
> use is deeply embedded with developers. If so, renaming it internally
> seems like a bad idea. And if we write our internal manifest into the
> profile and call it "specifications" it will only add the confusion.
>
> OTOH, "manifest" occurs only ~50 times in guix.info and the "user API"
> seems limited to the --manifest option and the functions:
> specifications->manifest and packages->manifest. Furthermore it is a new
> concept for new users. So I don't think users care what we call it.

Thank your for checking.
You convince me. :-)

All the best,
simon

Re: Using --manfistest with <profile>/manifest files

[Ludovic Courtès]

Hi,

George Clemmer <myglc2@gmail.com> skribis:

> Yes, only 'manifest.scm' is in the doc, but '.guix-profile/manifest'
> smacks a user in the face pretty quickly which leads to these messy
> questions.

That’s something I had not anticipated.  It’s an interesting lesson!

Ludo’.

Re: Using --manfistest with <profile>/manifest files

[zimoun]

Dear,

On Thu, 11 Jun 2020 at 10:40, elaexuotee@wilsonb.com wrote:
> In an attempt to tar up the *build* environment for a package to share with a
> colleague, I encountered this:
>
>     [env]$ guix pack -m $GUIX_ENVIRONMENT/manifest
>     (manifest ...): Wrong number of arguments
>
> From playing around a bit, my guess is that the `<profile>/manifest' files are
> just human-readable serializations of <manifest> objects and don't deserialize
> back:
>
>     $ guix environment -m ~/.guix-profile/manifest
>     (manifest ...): Wrong number of arguments

It is more "complicated" than that.  The detailed explanations are in
the mega thread. :-) In short and from my understanding, going from
"<profile>/manifest" to "-m manifest.scm" cannot be done in the general
case because two concepts -- imperative vs declarative -- are not well
aligned.  Pragmatically, it means that the result could be more than
often unpractical with too much inferiors.  Well, only an
"approximation" could be exported.

Sorry I am too lazy to search, but I think I remember that at the time
Pierre sent -- probably in the mega thread :-) -- a small script to
extract relevant information from <profile>/manifest; the preliminary
for '--export-manifest'. :-)

Otherwise, let open <profile>/manifest, write an Emacs macro and extract
the relevant information. ;-)


Well, because I hit the same problem some time ago -- well when I raised
or revived the discussion about this conversion -- I fixed for myself by
converting by hand some profiles and totally changed my workflow.

Now, I never use "guix install" but always "guix package -m".  For
temporary test, I extend the profile with "guix environment" which is on
purpose because it is a temporary profile; if it is worth, I add the
package to the manifest file.

(Sometimes, I run "guix install foo -p /tmp/foo" to test something
temporary, mainly when I try to figure out an issue that has been
reported.)

Each time, I run "guix package -m" or "guix update", then I run in the
same time "guix describe" and track the output.

Well, I have 2 kind of manifests:

 - the ones of my setup, living under ~/.config/guix/manifests and where
   I applied some recipe of [1]
 - the ones of projects I work on, living in the project folder

and each manifest is accompanied by channel.scm (guix describe -f).  And
I have almost no package in ~/.guix-profile but they live separated
profiles (Emacs, Python, Apps, etc.).


All the best,
simon


1:
https://guix.gnu.org/cookbook/en/html_node/Guix-Profiles-in-Practice.html#Guix-Profiles-in-Practice

Re: Using --manfistest with <profile>/manifest files

[elaexuotee]

[-- Attachment #1.1: Type: text/plain, Size: 3815 bytes --]

> It is more "complicated" than that.  The detailed explanations are in
> the mega thread. :-) In short and from my understanding, going from
> "<profile>/manifest" to "-m manifest.scm" cannot be done in the general
> case because two concepts -- imperative vs declarative -- are not well
> aligned.  Pragmatically, it means that the result could be more than
> often unpractical with too much inferiors.  Well, only an
> "approximation" could be exported.

I went ahead and read through the threads that Pierre shared in a different
reply. For posterity and to collect my own thoughts, let me see if I can
distill the discussion so far:


The goal is to enable a profile to generate itself by reifying it into some
collection of code and data. Given such a tool, two places of use are proposed:

1) Implicitly, upon profile generation, outputting new files under /etc, or
2) Explicitly via a command like `guix package --export'.

It turns out that `guix system reconfigure' does some (most?) of this by
generating `/gnu/store/<hash>-provenance' and `/gnu/store/<hash>-channels.scm'
along with the profile itself.

Naively, a profile is just a sum of outputs; however, there are subtleties:

a) Provenance and inferiors
   For reproducibility, channel and revision information must be stored.

This is alreading in profile/manifest, however, so we already have the
necessary infrastracture in this regard.

b) The family of `--with-*' options to `guix package' means that provenance
   data itself is insufficient. Worse, `--expression' means that the collection
   of outputs might be specified with *arbitrary code*.

That said, the sum of provenance and command-line invocation should be
sufficient, no? If so, an extreme proof-of-concept reification could simply be
a bash script, something akin to the following:

    #!/bin/env bash
    guix time-machine -C <profile>/channels.scm -- \
    	guix package -p <profile> [<other options> ...]

c) Profile reifications need to be forward-compatible, meaning that future
   revisions of guix should produce the same profile as current/older ones
   given a reification.

This, of course, shows why the bash script idea is untenable, but with a more
reasonable Guile implementation, storing version information in the way of
`<profile>/manifest' neatly solves this issue.


Am I missing any major points? Almost surely there are minor ones I am
overlooking. Here are some of my questions:

Theorem: Given provenance data (e.g. a `channels.scm' file) and a command line
         invocation, profiles generation is deterministic.

Note that by "command line invocation" I am including any external files its
options reference as well.

Is this true at least mostly? Ludo mentions something about the "possibility of
multiple provenances" which I fail to grok. What is going on here? Does this
introduce a source of non-determinism for users building profiles? I.e. given
the right bash script, can a user reliably reproduce a given profile?

If the answer to the final question above is no, then that seems like a much
larger problem. However, if the answer is yes, then I would naively expect
profile reification to be mostly a matter of collecting together all the
sources of input that define a profile. Does forward-compatibility make this
less straightforward than I am thinking? What else am I missing?

> Sorry I am too lazy to search, but I think I remember that at the time
> Pierre sent -- probably in the mega thread :-) -- a small script to
> extract relevant information from <profile>/manifest; the preliminary
> for '--export-manifest'. :-)

Perhaps you are thinking of Pierre's script here?
https://lists.gnu.org/archive/html/guix-devel/2020-02/msg00154.html


Cheers,

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 260 bytes --]

Re: Using --manfistest with <profile>/manifest files

[zimoun]

Dear,

On Mon, 15 Jun 2020 at 19:08, elaexuotee@wilsonb.com wrote:

> I went ahead and read through the threads that Pierre shared in a different
> reply. For posterity and to collect my own thoughts, let me see if I can
> distill the discussion so far:

[...]

> If the answer to the final question above is no, then that seems like a much
> larger problem. However, if the answer is yes, then I would naively expect
> profile reification to be mostly a matter of collecting together all the
> sources of input that define a profile. Does forward-compatibility make this
> less straightforward than I am thinking? What else am I missing?

Thank you for your inputs.

From my understanding, what you are proposing is a variant of what
Pierre proposes.  To be precise, at Guix Days, Pierre and I discussed to
change a bit the format of <profile>/manifest in order to unify the
current situation of "manifest.scm" (code evaluated) and
"<profile>/manifest" (flat data); as Pierre explained elsewhere in this
thread.  The change of internals will not happen, IMHO, because dealing
with the general case adds too much burden, and the use-case discussed
here -- recreate the exact same profile from another imperative profile
-- does not deserve so much attention, again IMHO.

From my point of view, it is a technical problem of internal
representation and then of UI.  And Ludo expressed that he is not in
favor for such internal change, because a lot of reasons he explained
elsewhere (argh!  I do not like what I am doing here: be imprecise
without citing exactly, anyway!).

The best, if I understand correctly, is simply to robustify the Pierre's
script and provide a clean '--export-manifest' from a profile which
should be an approximation.


>> Sorry I am too lazy to search, but I think I remember that at the time
>> Pierre sent -- probably in the mega thread :-) -- a small script to
>> extract relevant information from <profile>/manifest; the preliminary
>> for '--export-manifest'. :-)
>
> Perhaps you are thinking of Pierre's script here?
> https://lists.gnu.org/archive/html/guix-devel/2020-02/msg00154.html

Yes.


All the best,
simon